Treescope#
Treescope is an interactive HTML pretty-printer and N-dimensional array (“tensor”) visualizer, designed for machine learning and neural networks research in IPython notebooks. It’s a drop-in replacement for the standard IPython/Colab renderer, and adds support for:
Expanding and collapsing subtrees of rendered objects, to let you focus on the parts of your model that you care about,
Automatically embedding faceted visualizations of arbitrary-dimensional arrays and tensors directly into the output renderings, so you can quickly understand their shapes and the distribution of their values,
Color-coding parts of neural network models to emphasize shared structures,
Inserting “copy path” buttons that let you easily copy the path to any part of a rendered object,
Customizing the visualization strategy to support rendering your own data structures,
And more!
Treescope was originally developed as the pretty-printer for the Penzai neural network library, but it also supports rendering neural networks developed with other libraries, including Equinox, Flax NNX, and PyTorch. You can also use it with basic JAX and Numpy code.
With Treescope, instead of looking at this:
{'transformer': {'embedder.embeddings': Array([[-0.06861613, 0.01181955, 0.00733836, ..., 0.10034265,
0.05210693, 0.01150018],
[-0.00130964, -0.02908528, -0.0687367 , ..., 0.00270294,
-0.15112652, 0.14402005],
[ 0.03263972, 0.05464187, 0.00571925, ..., 0.01460573,
-0.05059952, 0.03807304],
...,
[-0.09474448, -0.01169327, 0.18301104, ..., 0.07539418,
-0.07683797, -0.02783352],
[-0.05568585, 0.00837327, 0.03356831, ..., 0.02185784,
0.04199443, 0.03226794],
[ 0.00023756, -0.09477348, -0.03787931, ..., 0.08830107,
-0.05957158, -0.04195255]], dtype=float32),
'block_0': {'pre_attention_norm': {'scale.weights': Array([1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1.], dtype=float32)},
'attention': {'query.weights': Array([[[ 0.00425512, 0.04423223, -0.07731628, ..., 0.1002233 ,
0.05203784, 0.06643772],
[-0.06178787, 0.04951855, -0.04079008, ..., -0.07338586,
0.02419353, 0.05889542],
[-0.04198842, -0.02659851, 0.10808309, ..., -0.00600227,
0.05246251, -0.08817653],
...,
[-0.03152804, -0.03930124, 0.04335218, ..., -0.10359181,
0.08420438, -0.01995611],
[-0.04870198, 0.07418844, 0.05810077, ..., -0.01861358,
-0.03782224, -0.00981956],
[ 0.10657898, 0.01568166, -0.01895318, ..., 0.01430033,
-0.09570113, -0.06903884]],
[[-0.025809 , -0.02712015, 0.01038032, ..., -0.05019396,
-0.0320798 , 0.03290451],
[ 0.08708496, 0.03964422, -0.08021147, ..., -0.03409857,
-0.05668447, -0.04848173],
[-0.04783362, 0.07073171, 0.03198322, ..., -0.03625632,
-0.10772 , 0.10282816],
...,
[-0.08468845, 0.03919826, -0.05250417, ..., -0.07438178,
0.09458494, -0.06394692],
[-0.10185295, 0.10737941, 0.10707574, ..., 0.09467778,
0.07761698, 0.03767436],
[-0.09037171, -0.05101899, 0.04059414, ..., -0.07484726,
-0.08460118, 0.05412047]],
[[ 0.04541534, 0.06503828, -0.04732819, ..., 0.03280171,
-0.00688228, -0.08610719],
[ 0.01273782, 0.04679164, -0.10776922, ..., -0.00843291,
-0.00138115, 0.00600488],
[ 0.07184575, -0.04019329, -0.0333933 , ..., 0.08710042,
0.04921516, -0.03265277],
...,
[ 0.00088171, 0.07229332, -0.00938513, ..., 0.00711978,
-0.00552056, 0.0769027 ],
[-0.05446356, 0.09569021, -0.09071126, ..., -0.08280504,
-0.04847672, 0.05099529],
[ 0.04198303, -0.02963323, 0.08103859, ..., -0.02164687,
-0.09514426, -0.01438455]],
...,
[[ 0.07336465, -0.04795771, 0.04007612, ..., -0.03697992,
0.04596545, 0.05015703],
[-0.01922942, -0.01693954, -0.02137136, ..., 0.00187217,
0.06122724, -0.09723517],
[ 0.00749154, 0.00263441, -0.04710845, ..., -0.0802056 ,
-0.02508158, 0.0033021 ],
...,
[ 0.04472937, 0.10616562, 0.00826446, ..., 0.01648008,
0.02515806, -0.01627337],
[ 0.00426475, 0.10275359, -0.02950539, ..., -0.10586847,
0.01523957, 0.07377515],
[-0.02854254, -0.06535842, -0.0363766 , ..., -0.01973454,
0.03138622, -0.08147995]],
[[-0.02418953, -0.02253676, -0.03196352, ..., -0.03505985,
0.09221888, -0.05973263],
[-0.0068432 , 0.05276265, -0.01458847, ..., -0.07371967,
0.05117131, -0.10285216],
[-0.03759643, -0.00053421, -0.0320171 , ..., 0.05825619,
-0.06150484, -0.00032701],
...,
[ 0.02530337, -0.02023746, 0.02158828, ..., 0.01514616,
0.05733294, 0.05211958],
[ 0.01878573, -0.06363847, 0.02239432, ..., -0.09784771,
0.02876685, 0.02651159],
[-0.06688988, -0.06454545, 0.01667278, ..., 0.07162189,
-0.08915381, 0.05585 ]],
[[-0.10498646, 0.05194503, 0.01682626, ..., -0.0804112 ,
0.00909207, 0.10391068],
[ 0.10105568, -0.02788559, 0.02396658, ..., -0.0345314 ,
-0.07004751, -0.07798508],
[ 0.00612345, -0.01679488, 0.08799656, ..., -0.04938953,
0.04337936, -0.03931122],
...,
[ 0.06886904, -0.05069381, -0.07608119, ..., -0.04655473,
-0.01851638, -0.04966667],
[-0.09835783, -0.08687965, 0.00372564, ..., 0.03642675,
-0.04896679, 0.0871337 ],
[-0.08821019, 0.00788428, -0.04496114, ..., 0.0112867 ,
-0.08532362, -0.06964879]]], dtype=float32),
'key.weights': Array([[[-0.01865319, 0.06913534, -0.036454 , ..., 0.10805877,
0.04868521, 0.05018144],
[-0.00312714, -0.10543523, 0.08125185, ..., 0.01414251,
-0.08474313, 0.00317001],
[-0.07085393, 0.04478639, -0.06728067, ..., -0.03341797,
0.02909409, 0.07326552],
...,
[ 0.07441433, 0.06590378, -0.0843019 , ..., -0.08667397,
0.04632611, -0.03923467],
[-0.05764252, 0.08886369, -0.07261553, ..., -0.06767029,
-0.00798463, 0.03002644],
[ 0.10521097, -0.04420717, -0.03766233, ..., -0.02754624,
-0.02313121, 0.04865493]],
[[-0.05781049, 0.05442218, 0.0631882 , ..., 0.07042073,
-0.09657256, 0.09194744],
[ 0.06347954, -0.03891489, 0.02408453, ..., 0.00690917,
0.01667187, 0.0993045 ],
[ 0.04382475, 0.00655664, 0.01995838, ..., 0.08413692,
-0.10403207, 0.01414181],
...,
[ 0.00387807, -0.00923936, -0.03314687, ..., -0.04494814,
-0.10269962, 0.03882897],
[ 0.01323107, -0.06116403, -0.03184152, ..., -0.04570743,
0.09287694, 0.01829919],
[-0.05301608, -0.06633244, 0.0899824 , ..., -0.0807281 ,
-0.03648717, 0.08300515]],
[[ 0.07898037, 0.02842779, 0.00286009, ..., -0.05747582,
-0.06717198, 0.04402707],
[-0.01703961, 0.02662789, -0.08440263, ..., -0.07947817,
0.03163961, 0.06644085],
[ 0.04617559, 0.02781422, 0.04360057, ..., -0.03410574,
-0.008882 , -0.04214715],
...,
[ 0.01615382, 0.07187683, 0.02740653, ..., -0.06949554,
0.09181245, -0.03981616],
[ 0.05989012, -0.10192609, -0.08399411, ..., 0.03657484,
0.03019887, 0.06880327],
[-0.03154685, 0.03371465, 0.02436408, ..., -0.00247078,
0.08914382, 0.03787157]],
...,
[[-0.06675363, 0.06390405, -0.04679091, ..., -0.05667975,
-0.07123782, -0.05104828],
[ 0.08407015, 0.00782969, -0.10678745, ..., 0.06258544,
-0.08356232, 0.00153087],
[-0.08932763, -0.06902669, -0.06483761, ..., 0.04801186,
-0.09582744, 0.05803366],
...,
[ 0.05741106, 0.04454479, 0.09445331, ..., -0.07480575,
-0.02068585, -0.04553172],
[ 0.07731251, -0.01280965, -0.02500965, ..., -0.08578791,
-0.05992881, 0.04611419],
[ 0.07967839, -0.01928633, 0.0984563 , ..., 0.07148853,
-0.10442412, 0.03923847]],
[[-0.04064782, -0.08173067, -0.03644141, ..., 0.10769321,
-0.02772915, -0.09922107],
[-0.0083439 , 0.03581999, -0.06778966, ..., -0.05737555,
0.03442697, -0.06005995],
[ 0.06461294, -0.03963728, -0.02639668, ..., -0.02908774,
0.03096126, -0.0526678 ],
...,
[ 0.03814262, 0.01074909, 0.10141044, ..., -0.0351332 ,
-0.10065917, 0.09221991],
[ 0.05076076, 0.08041846, -0.10727135, ..., 0.04690411,
0.02995022, 0.05057604],
[-0.09999597, 0.08160675, -0.04225888, ..., -0.09716489,
0.02527115, 0.00827426]],
[[ 0.07150082, 0.06617072, -0.09273553, ..., -0.0049749 ,
-0.03591781, 0.02089096],
[-0.05534978, 0.01970886, 0.02972857, ..., 0.09339951,
-0.0737289 , -0.0611032 ],
[ 0.05779025, -0.06219819, -0.00770769, ..., -0.01945481,
0.00809602, -0.02756529],
...,
[-0.08361438, 0.00607885, -0.02241925, ..., -0.03156706,
-0.03043206, 0.00182327],
[ 0.07754482, 0.07543445, 0.08595543, ..., 0.06339122,
-0.06007399, -0.00336789],
[-0.09229829, 0.06760266, 0.0046538 , ..., -0.07452484,
0.01681333, 0.03724615]]], dtype=float32),
'value.weights': Array([[[ 0.060807 , -0.09430943, -0.05298686, ..., 0.02425545,
-0.05830301, -0.04966006],
[-0.06123699, 0.03310462, -0.09598609, ..., -0.09503989,
-0.0069023 , 0.02083351],
[ 0.07594024, -0.05183586, 0.07630898, ..., -0.07724711,
-0.01686031, 0.0093435 ],
...,
[ 0.0794818 , 0.09729231, -0.04079202, ..., 0.10802427,
0.03060426, 0.0176866 ],
[-0.05194196, -0.07882418, 0.02130861, ..., 0.09413511,
-0.05461694, -0.09290484],
[ 0.03369289, -0.08238543, 0.09440154, ..., 0.0765 ,
-0.09488645, -0.01425976]],
[[ 0.00316593, -0.09941892, -0.05606189, ..., -0.05652989,
0.08087002, 0.09108754],
[ 0.02648245, 0.02866065, 0.04578783, ..., -0.03773772,
0.03249203, 0.04347774],
[ 0.08532349, 0.08334874, -0.07387519, ..., 0.1064342 ,
0.07922402, -0.02333717],
...,
[ 0.10641109, -0.0623649 , 0.01392759, ..., -0.03952655,
0.09982023, 0.09805328],
[-0.00348329, 0.07489028, 0.08029795, ..., -0.09941461,
0.02098259, -0.07160114],
[ 0.08834912, 0.04947118, 0.10658376, ..., -0.0012552 ,
0.10507638, -0.08887359]],
[[-0.06594443, -0.04612255, -0.05825204, ..., -0.02785384,
-0.04137862, 0.07970921],
[-0.10689492, 0.0617172 , 0.06004617, ..., 0.03540417,
0.04109154, -0.03544193],
[ 0.10590412, -0.03783987, 0.01348049, ..., -0.0003261 ,
-0.08130878, -0.06378525],
...,
[ 0.03408417, 0.10782541, -0.00179155, ..., -0.08150347,
-0.06353503, -0.03034916],
[-0.08402589, -0.03508411, -0.04628265, ..., 0.01353415,
0.07180281, -0.02800978],
[-0.07261176, -0.06540944, -0.05729154, ..., -0.10089094,
-0.06556655, -0.01446077]],
...,
[[-0.04428548, -0.02135912, -0.05055733, ..., -0.09586572,
0.05155012, -0.04292898],
[-0.00673906, -0.09602788, 0.10510464, ..., 0.06211354,
-0.04887062, 0.09935362],
[-0.0329257 , 0.01974251, -0.01476516, ..., 0.07165389,
0.08511521, -0.08617296],
...,
[-0.08197919, -0.04745102, 0.00840058, ..., 0.02407509,
0.03289411, -0.07807301],
[ 0.04669539, -0.00514854, -0.0614021 , ..., 0.03252104,
0.05726591, 0.03398725],
[-0.08349689, 0.01777422, -0.03976317, ..., 0.07841548,
0.06636639, 0.00677189]],
[[ 0.02278773, -0.10388034, -0.00592952, ..., -0.01410609,
-0.09243606, 0.05849129],
[ 0.02589211, -0.08700392, 0.0369664 , ..., -0.02261532,
0.0967467 , -0.06340934],
[-0.02886826, 0.0485855 , 0.00998048, ..., 0.01185872,
-0.06932623, -0.10165689],
...,
[-0.02325912, -0.03552042, -0.10567299, ..., -0.05976012,
-0.03988296, -0.01325061],
[ 0.0104291 , 0.02217052, 0.06899785, ..., -0.10649167,
0.02991861, -0.10069851],
[-0.07434025, 0.03407258, -0.08694967, ..., -0.01410681,
-0.0893549 , 0.02375319]],
[[-0.07728796, -0.05275607, -0.05248221, ..., -0.0441762 ,
-0.04211195, 0.08625981],
[ 0.08887386, 0.06074692, 0.07999159, ..., -0.02245275,
-0.05372197, -0.08777037],
[-0.06890883, -0.06106324, 0.00896867, ..., -0.08989609,
-0.10494957, 0.0475762 ],
...,
[ 0.0772803 , 0.02082236, 0.04188999, ..., -0.0494677 ,
0.02687135, 0.10709071],
[-0.05018415, -0.09475797, 0.02481595, ..., -0.05298875,
-0.007999 , 0.04869512],
[-0.00865544, 0.09138167, -0.08480624, ..., 0.02868857,
0.01925734, 0.04829025]]], dtype=float32),
'output.weights': Array([[[ 0.09401354, 0.08879067, -0.00749887, ..., 0.0446185 ,
0.02851673, -0.07824904],
[-0.08081626, -0.08861338, 0.01750418, ..., -0.01313209,
0.02107911, -0.07022807],
[-0.03021784, 0.07040884, 0.05938885, ..., -0.03577882,
0.08015499, -0.01967804],
...,
[-0.073653 , 0.08618104, -0.05159955, ..., -0.00265369,
0.02072402, 0.10017039],
[-0.06285188, -0.04204211, -0.08126129, ..., 0.00349998,
0.1062774 , 0.05508373],
[-0.01172229, -0.09435405, 0.08956411, ..., -0.06670979,
-0.04233567, -0.02290336]],
[[ 0.09296552, 0.01246194, -0.09352814, ..., 0.06678572,
-0.06727248, -0.0694672 ],
[-0.0333177 , -0.04117563, 0.08438136, ..., 0.0149338 ,
-0.09327415, 0.01307874],
[ 0.01292801, -0.10350697, -0.01494392, ..., 0.06559891,
-0.01405824, -0.01068351],
...,
[ 0.0260147 , -0.08936294, -0.01005892, ..., 0.09096835,
-0.01403906, -0.00203157],
[ 0.02514368, -0.03317725, 0.04040059, ..., -0.06218152,
0.08395182, -0.05452109],
[-0.01564821, 0.0027217 , 0.08753777, ..., -0.09599675,
-0.03121241, 0.04361546]],
[[ 0.06368348, 0.0310296 , 0.02017152, ..., 0.05826433,
-0.09853993, -0.03686295],
[ 0.05315803, -0.05024849, 0.09887596, ..., -0.00231347,
0.01697206, 0.07600673],
[-0.04498127, -0.04095687, -0.10207214, ..., 0.09364338,
-0.05895465, -0.04074146],
...,
[-0.04984749, 0.04748109, -0.08129679, ..., -0.07231521,
0.00061465, 0.05359847],
[-0.10135236, -0.0957322 , -0.09715395, ..., -0.09067242,
-0.10762231, -0.09037723],
[-0.05903541, -0.03407738, 0.0274054 , ..., 0.01954858,
0.02858998, 0.04357925]],
...,
[[-0.07095383, -0.06637046, 0.1036773 , ..., -0.05868794,
0.07308026, -0.09407523],
[-0.05049361, -0.08899544, 0.09702983, ..., -0.10757043,
0.04493983, 0.03091911],
[ 0.0716719 , -0.00550565, -0.02661446, ..., 0.09682707,
-0.06674633, -0.05302607],
...,
[-0.02571988, -0.03124176, -0.05107569, ..., 0.09342524,
-0.03745745, -0.08849649],
[-0.02073123, 0.03565202, -0.06957932, ..., -0.00807081,
-0.08874661, 0.00737942],
[-0.00777614, 0.00389737, -0.1067605 , ..., -0.07771519,
-0.09310092, 0.08795134]],
[[-0.05403958, -0.0405856 , 0.01326841, ..., -0.00056825,
0.02004534, 0.07658798],
[ 0.06388139, 0.03140186, 0.07632377, ..., -0.08147928,
0.01065127, 0.08336617],
[ 0.0960383 , -0.05015439, -0.07784024, ..., 0.06566341,
0.09980108, 0.0307615 ],
...,
[-0.03056243, 0.07713037, 0.04095891, ..., 0.04013277,
0.08032061, -0.10537685],
[-0.00736122, 0.037655 , -0.06516511, ..., 0.04819663,
0.04806258, -0.02294326],
[ 0.08172651, -0.0881365 , 0.0653775 , ..., -0.00101726,
0.09168245, -0.05389629]],
[[-0.06479838, 0.00279763, -0.07987571, ..., -0.03329019,
-0.10151406, -0.0492303 ],
[ 0.07176022, 0.00932187, 0.00807372, ..., -0.0092814 ,
-0.05965969, -0.00208268],
[ 0.07565007, -0.09398895, 0.09076703, ..., -0.06261633,
0.03615776, -0.05908582],
...,
[ 0.08098658, 0.10423039, -0.03698803, ..., 0.03100439,
-0.09684725, -0.0504377 ],
[ 0.10718264, 0.06186468, 0.07258634, ..., -0.10225921,
-0.04866642, 0.04867256],
[-0.03782955, -0.08317758, -0.05903487, ..., 0.03216153,
0.06421012, 0.09982981]]], dtype=float32)},
'pre_ffw_norm': {'scale.weights': Array([1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1.], dtype=float32)},
'mlp': {'gating_linear.weights': Array([[-0.01427065, 0.00768034, -0.00447652, ..., -0.02888646,
0.07477827, 0.02553298],
[-0.07304782, -0.0667632 , 0.08360735, ..., -0.07122341,
0.02256534, -0.04226577],
[ 0.06133342, 0.0607366 , -0.01841109, ..., 0.06090352,
0.01182023, 0.06592193],
...,
[-0.03730955, -0.00198054, 0.02179668, ..., -0.06639501,
0.01567852, 0.03140555],
[ 0.00883623, -0.0143505 , 0.00785788, ..., 0.08791708,
-0.06239847, -0.02368771],
[-0.07817437, 0.03628072, -0.08112461, ..., -0.02304703,
0.08114208, 0.08687703]], dtype=float32),
'value_linear.weights': Array([[ 0.05659883, 0.01184432, -0.03085446, ..., 0.03592804,
-0.04561374, -0.07121216],
[ 0.07315332, 0.00563379, 0.04626453, ..., -0.04133404,
-0.04886309, -0.04048896],
[-0.05447267, -0.02950477, -0.04920761, ..., 0.02709244,
-0.02989819, 0.05613648],
...,
[ 0.07063601, 0.08769214, 0.00667168, ..., -0.06170665,
0.02877055, -0.07862035],
[-0.01249285, -0.07843712, -0.02518565, ..., 0.0293569 ,
-0.08054282, 0.01777286],
[ 0.0858836 , 0.0809206 , -0.04532727, ..., 0.03730327,
-0.01978756, -0.01601239]], dtype=float32),
'out_linear.weights': Array([[-0.05554331, -0.00361445, -0.01185418, ..., 0.06537496,
-0.00964446, -0.05631434],
[-0.02659093, 0.06202033, -0.0558922 , ..., -0.01540292,
0.01934494, 0.05419621],
[-0.0734439 , 0.02422565, 0.06639498, ..., 0.02091116,
-0.07948491, -0.00483403],
...,
[-0.00213891, 0.07057896, 0.07161637, ..., 0.06169894,
-0.05635135, -0.00086799],
[ 0.08112863, -0.0763196 , 0.03214591, ..., -0.01398831,
-0.01355491, -0.05937452],
[ 0.01076087, 0.08023544, 0.02675697, ..., 0.03871306,
-0.08120635, -0.05828308]], dtype=float32)}},
'block_1': {'pre_attention_norm': {'scale.weights': Array([1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1.], dtype=float32)},
'attention': {'query.weights': Array([[[ 0.064591 , -0.01677975, -0.07710495, ..., -0.04174767,
-0.00280591, -0.06320172],
[-0.09474751, -0.06573597, -0.05665025, ..., 0.01516408,
-0.02419472, 0.07509851],
[ 0.08540037, -0.10095555, -0.0595229 , ..., -0.05730119,
0.00076022, -0.04378191],
...,
[ 0.01015751, 0.10317416, 0.03090285, ..., 0.0678891 ,
0.01386402, 0.0676039 ],
[ 0.01892709, -0.01364784, 0.10166001, ..., -0.07173996,
-0.07864155, 0.09062947],
[-0.07278992, -0.07867055, -0.04318926, ..., 0.08605392,
0.08998451, 0.00638322]],
[[-0.00847875, -0.05726882, 0.08117532, ..., 0.01061191,
0.08778311, -0.02158356],
[-0.07245938, -0.08205567, -0.0402342 , ..., 0.00677284,
-0.0309985 , 0.03834419],
[-0.05080154, 0.02414968, -0.00919492, ..., -0.09530405,
0.05614105, 0.08277342],
...,
[ 0.08703536, 0.06568801, -0.02505193, ..., -0.08667785,
0.02228424, -0.06554051],
[ 0.04347965, 0.05077214, 0.07340143, ..., 0.07907713,
-0.01680231, 0.08635102],
[ 0.00743282, 0.05118497, 0.10289448, ..., -0.0697723 ,
0.08376753, -0.07104471]],
[[-0.10031831, -0.03339048, -0.02638068, ..., -0.06891459,
-0.05982883, -0.09115028],
[-0.08095272, -0.04772896, 0.0193037 , ..., 0.04893951,
-0.05420673, -0.00111712],
[-0.00506714, 0.08201516, -0.09856782, ..., -0.09161914,
-0.01259084, 0.08019146],
...,
[ 0.01004021, -0.09733606, -0.02677877, ..., 0.00919778,
-0.08369441, 0.08512878],
[-0.10414741, 0.0009684 , 0.09066336, ..., 0.06565645,
0.10548479, -0.0046563 ],
[ 0.00377225, -0.08837761, -0.1058597 , ..., -0.02838931,
0.10295535, -0.0702142 ]],
...,
[[ 0.009243 , 0.07788858, 0.03654039, ..., -0.01188087,
-0.03219098, -0.05234186],
[ 0.10446676, 0.04300661, -0.01843061, ..., 0.02893178,
0.05801446, 0.04272596],
[-0.01148366, 0.10615592, 0.03066357, ..., 0.05694068,
-0.05563658, -0.00339275],
...,
[-0.02314295, 0.03195039, 0.06948233, ..., 0.02263499,
0.04183656, 0.05841859],
[-0.07919914, 0.07045661, 0.0539094 , ..., 0.0434325 ,
-0.09500886, 0.01382665],
[-0.02976153, 0.02970591, -0.0186311 , ..., -0.03510006,
0.02182516, 0.05691433]],
[[ 0.05088635, 0.02871534, 0.08107182, ..., -0.03326807,
-0.02692377, 0.0185505 ],
[ 0.0487797 , -0.00039308, -0.08446527, ..., -0.00894591,
0.00620274, -0.00353503],
[-0.08977246, -0.00194227, -0.01647071, ..., 0.03860474,
0.03161683, 0.07506654],
...,
[-0.05537314, 0.06971957, 0.09382106, ..., 0.07625574,
-0.05727628, -0.02839651],
[ 0.01562026, 0.06043811, -0.04819119, ..., 0.03573962,
0.10236704, 0.03479742],
[ 0.00643242, 0.09919035, 0.02669569, ..., -0.02914101,
-0.0148404 , 0.05039535]],
[[ 0.06387362, -0.08004174, 0.08181816, ..., 0.03199323,
0.07582064, -0.07288506],
[ 0.0147798 , 0.01366534, -0.09877007, ..., -0.0215441 ,
0.09106847, 0.09482133],
[-0.02300213, 0.07702169, 0.10160742, ..., 0.09229029,
0.03917531, 0.08303344],
...,
[-0.05782208, -0.10051931, 0.10533212, ..., -0.06473176,
0.10129427, 0.09361773],
[-0.07485095, 0.08962385, 0.1019529 , ..., -0.04759997,
-0.09579621, 0.10410648],
[ 0.02303987, -0.01180641, 0.1053993 , ..., -0.0050005 ,
0.02275289, 0.05945998]]], dtype=float32),
'key.weights': Array([[[ 0.07069385, -0.0364453 , 0.10082451, ..., -0.00584829,
0.0977016 , -0.07788109],
[ 0.03422184, 0.09737067, 0.01999147, ..., 0.09997212,
-0.0998462 , 0.0211001 ],
[-0.0086061 , -0.05200163, -0.04420033, ..., -0.02035815,
0.03576306, -0.07526116],
...,
[ 0.09162443, -0.02163118, 0.093601 , ..., 0.09886969,
0.02279129, 0.08366411],
[ 0.10161418, -0.03393785, 0.0527602 , ..., -0.00213228,
-0.10346111, 0.09820309],
[ 0.09332711, 0.04627658, 0.08552449, ..., -0.10093069,
0.0588756 , 0.0410888 ]],
[[-0.07596904, 0.0498836 , 0.0819181 , ..., -0.06117608,
-0.06979287, 0.01382884],
[ 0.06855186, -0.04704029, 0.02636176, ..., 0.01847981,
-0.07535328, 0.01496573],
[ 0.00868951, 0.04948907, 0.06541695, ..., -0.01604369,
0.01451303, -0.05225906],
...,
[ 0.02885889, -0.05728986, 0.00784404, ..., -0.03605498,
0.07913712, 0.07202805],
[-0.05219219, 0.02862315, 0.03189087, ..., -0.10298051,
-0.07388298, -0.0564696 ],
[-0.00284607, -0.07932403, 0.08258218, ..., -0.06411765,
-0.06978747, -0.07372983]],
[[ 0.00539944, -0.10778107, 0.06171488, ..., 0.06513135,
0.01158 , 0.04797586],
[-0.09882308, 0.04422748, -0.08229864, ..., -0.00956425,
-0.0878171 , 0.06273619],
[ 0.09627183, 0.02201649, 0.05376394, ..., -0.09670953,
0.01532998, 0.04216499],
...,
[-0.04989931, 0.06076089, 0.09748555, ..., 0.07065371,
0.01133698, -0.0539549 ],
[-0.05904179, 0.05533605, -0.08310002, ..., -0.09324181,
0.02341854, -0.00773988],
[-0.05772114, 0.09916707, 0.07470569, ..., 0.02015921,
0.05861954, -0.10137209]],
...,
[[-0.06004547, -0.04708599, 0.02233911, ..., -0.07420664,
0.04971145, -0.03305263],
[-0.043408 , -0.07258552, -0.1030238 , ..., 0.01743999,
-0.05156674, -0.02762463],
[-0.0180706 , -0.09721832, 0.05185607, ..., -0.0465593 ,
0.0674227 , -0.03207641],
...,
[-0.02342236, 0.10020962, -0.01075224, ..., 0.04002424,
0.02728157, -0.01236885],
[-0.04987591, 0.09220241, -0.08074676, ..., -0.08966745,
-0.06909257, 0.06904503],
[ 0.07796392, 0.08749779, 0.04448295, ..., 0.03293128,
0.09276617, 0.02828788]],
[[ 0.094055 , -0.07676723, 0.10631451, ..., -0.08227272,
0.10511992, -0.05880398],
[-0.01445369, -0.05866972, 0.03958615, ..., 0.10636727,
0.04532281, 0.0120501 ],
[-0.0201182 , 0.00083925, 0.04254059, ..., 0.05044096,
-0.02443647, 0.00411877],
...,
[ 0.04557828, -0.08642496, 0.09047309, ..., -0.06248933,
-0.05595896, -0.08524908],
[-0.0662745 , -0.0852035 , -0.03958757, ..., -0.021569 ,
0.0343737 , -0.0446856 ],
[ 0.08773065, -0.06021297, 0.10625928, ..., -0.09347787,
-0.04226399, 0.07368815]],
[[ 0.07553552, -0.09811597, 0.00363618, ..., 0.04909723,
-0.06490508, -0.07829609],
[-0.03694684, -0.07487226, 0.10214808, ..., 0.06468074,
-0.05132434, -0.02845709],
[-0.03843519, 0.00128465, -0.01972396, ..., -0.02094498,
-0.06567061, 0.03949491],
...,
[ 0.08461486, 0.06953789, 0.06822127, ..., 0.07504369,
-0.01460153, 0.0165874 ],
[ 0.0407515 , -0.04854522, 0.03443373, ..., -0.08050453,
0.00624003, -0.07303723],
[-0.06631617, 0.00513332, -0.1029715 , ..., -0.09478941,
0.09518476, -0.08669966]]], dtype=float32),
'value.weights': Array([[[-0.06892963, 0.06381965, 0.04100784, ..., -0.05432295,
0.09444501, -0.00047332],
[ 0.04316932, 0.03664133, -0.01312241, ..., 0.08500472,
0.07115448, 0.10674649],
[-0.04039342, -0.00814158, 0.06548486, ..., 0.08714817,
-0.09233794, 0.04414435],
...,
[ 0.00558196, 0.00248621, 0.0219665 , ..., -0.00690811,
-0.01992445, -0.04578757],
[-0.08743053, 0.01264935, -0.06390589, ..., -0.03677146,
-0.05752204, -0.01912977],
[ 0.07449441, -0.10395808, 0.00482112, ..., -0.0526996 ,
0.02258776, -0.01578965]],
[[ 0.05933274, -0.08939485, -0.07383916, ..., 0.01473148,
0.09873344, -0.04545279],
[ 0.08277413, -0.06933242, -0.07319418, ..., 0.10239407,
0.06027574, -0.07802668],
[ 0.00039881, -0.01144621, 0.04892715, ..., 0.05184494,
0.0167485 , 0.00567586],
...,
[ 0.06793617, 0.02809661, -0.100798 , ..., 0.04686052,
-0.04467713, -0.01255863],
[ 0.01943081, -0.07827487, -0.00577053, ..., -0.09246167,
-0.00394437, -0.04195549],
[ 0.01889467, -0.0014889 , -0.01140871, ..., -0.08906467,
0.01644872, -0.0968392 ]],
[[ 0.01319421, -0.04103876, -0.02159608, ..., -0.06137693,
0.02719433, 0.01434955],
[-0.0675934 , 0.00393253, -0.04756933, ..., -0.00982735,
0.05259004, 0.06454645],
[-0.07795468, 0.08871984, -0.0773887 , ..., -0.05044924,
-0.05325229, -0.06570461],
...,
[ 0.00564515, 0.03872922, -0.01773969, ..., -0.05172444,
0.09030177, -0.08755008],
[-0.06380107, -0.00680443, -0.09919105, ..., -0.07342027,
0.06670818, 0.07563125],
[ 0.10670199, 0.09631434, -0.08963297, ..., 0.00828007,
0.07855178, 0.06818653]],
...,
[[-0.05066232, 0.04650734, 0.06557865, ..., 0.07587546,
0.03220247, 0.02642876],
[ 0.02881527, -0.08682514, 0.08709648, ..., -0.1064458 ,
0.01637225, -0.03782901],
[ 0.08496778, 0.09373645, 0.09292874, ..., -0.04742059,
-0.00265204, -0.08616297],
...,
[-0.07696669, -0.03213374, -0.00694739, ..., -0.08021681,
-0.07355902, -0.08963663],
[ 0.02245179, -0.00196855, 0.08667149, ..., 0.03665903,
0.04475209, 0.06696471],
[-0.00133115, 0.07821368, 0.01856632, ..., 0.04245808,
-0.06520075, 0.05478816]],
[[ 0.05447102, -0.10076744, -0.06098287, ..., -0.06139247,
-0.08870057, 0.08607847],
[ 0.07156343, 0.05630037, -0.01302836, ..., 0.01587046,
0.08021609, -0.03095468],
[ 0.0875587 , -0.10477487, 0.02881274, ..., -0.10133794,
0.02375213, -0.03767164],
...,
[-0.05975227, 0.06562356, -0.07528532, ..., -0.06564901,
-0.08073674, -0.05488531],
[ 0.00609302, -0.08634433, -0.10301355, ..., -0.09443597,
-0.03075063, 0.02832172],
[ 0.04153905, -0.00138688, 0.09001771, ..., 0.03747939,
0.00812467, -0.01285273]],
[[ 0.04084687, -0.02800663, -0.03538796, ..., 0.096389 ,
0.03697461, 0.00752403],
[-0.06196353, 0.03833304, -0.07929427, ..., -0.07366513,
-0.01546032, 0.02867484],
[-0.06852151, -0.09201018, -0.08654097, ..., -0.01178382,
-0.06130392, 0.04688517],
...,
[ 0.00622331, -0.01561678, -0.09193242, ..., 0.03635554,
0.08315118, -0.10722423],
[ 0.02447986, -0.02350689, -0.03530132, ..., -0.09015568,
0.0878671 , -0.08605256],
[-0.09598854, -0.05969105, 0.0580779 , ..., 0.09371728,
-0.08426814, -0.07824343]]], dtype=float32),
'output.weights': Array([[[ 0.05005521, -0.03655383, 0.05706439, ..., -0.00585062,
0.06954311, -0.05378549],
[ 0.05785669, 0.08642896, -0.05745285, ..., -0.05508908,
-0.04649524, -0.0836008 ],
[-0.05748557, 0.10261241, -0.08311889, ..., 0.07954116,
-0.09533365, 0.00720908],
...,
[-0.01798382, 0.07425898, -0.03582267, ..., -0.06229248,
-0.0731422 , -0.08944339],
[-0.06343992, -0.0963666 , -0.0059003 , ..., -0.02319034,
-0.10144304, -0.01481513],
[-0.06779642, 0.00240124, -0.09169391, ..., 0.0548907 ,
-0.08313886, -0.04354327]],
[[ 0.07074098, 0.09880143, -0.04731229, ..., -0.06248331,
-0.10175002, -0.04029364],
[ 0.05484043, 0.02230863, -0.08860631, ..., 0.04482523,
-0.10193326, 0.03795643],
[ 0.06476945, -0.05578044, -0.00320493, ..., -0.00286447,
-0.04095777, -0.09610314],
...,
[ 0.00552856, -0.09156409, -0.00071993, ..., 0.05349737,
-0.03327457, 0.0015886 ],
[-0.09461803, -0.00853528, -0.07264923, ..., 0.02560474,
0.10209772, 0.043733 ],
[-0.01988839, 0.06230259, -0.01050207, ..., 0.05933429,
0.08879127, 0.02146669]],
[[ 0.02947275, 0.05426278, 0.09165373, ..., 0.03517116,
0.09118704, -0.01965298],
[-0.00124898, -0.088405 , -0.0125497 , ..., 0.0160192 ,
0.03311489, -0.04412285],
[-0.06527111, -0.01643096, -0.07391612, ..., 0.01833243,
-0.04328231, 0.01114904],
...,
[ 0.05249147, 0.01254866, -0.0866247 , ..., -0.08025067,
-0.02958933, 0.08726373],
[-0.03796407, 0.03313399, -0.10372426, ..., 0.05982438,
0.09710693, 0.02162049],
[-0.07144411, -0.02492611, 0.06746858, ..., -0.00177629,
-0.05913122, 0.08199269]],
...,
[[ 0.06848773, 0.03704365, -0.09323084, ..., 0.01980427,
-0.02525159, -0.01145746],
[ 0.06054099, -0.0175591 , -0.09604156, ..., -0.01792345,
-0.09182711, 0.03077063],
[-0.0677814 , 0.01555847, 0.07506187, ..., -0.08882115,
0.08110651, -0.04169905],
...,
[ 0.03442743, -0.06656025, -0.02111424, ..., -0.0752494 ,
0.06181706, 0.04773239],
[ 0.09839766, -0.09719969, -0.10586225, ..., 0.10043667,
-0.10379839, -0.04003626],
[ 0.020952 , -0.06967238, 0.05705729, ..., 0.01449951,
0.03126726, 0.09440856]],
[[ 0.08065792, 0.0081351 , 0.05018332, ..., -0.0612499 ,
-0.07892798, 0.06711752],
[ 0.05442358, 0.01085881, 0.09712762, ..., 0.01278428,
0.10465276, -0.00630551],
[-0.02418924, -0.09948824, -0.08158427, ..., -0.02347625,
0.04228335, 0.01517468],
...,
[-0.10129249, 0.08653667, 0.07592932, ..., 0.10805508,
-0.03188811, -0.01697031],
[ 0.07056703, -0.02304348, -0.04035055, ..., -0.07490159,
0.01745248, -0.09575342],
[ 0.0655045 , 0.05065468, -0.0213607 , ..., -0.01635403,
0.05224264, -0.08038735]],
[[-0.06589392, 0.04604081, -0.0165412 , ..., 0.04541707,
-0.09396742, -0.0765694 ],
[ 0.02641963, 0.00768885, 0.03864087, ..., -0.02726732,
0.00981337, -0.06781433],
[-0.07552292, 0.09671775, -0.04023203, ..., 0.00343324,
0.10715701, -0.02716612],
...,
[-0.09559601, -0.04596031, -0.02748412, ..., -0.07590351,
0.09454948, -0.01561807],
[-0.03530042, 0.0856252 , 0.08615933, ..., -0.06619008,
-0.00315509, -0.00265593],
[ 0.0708262 , 0.0503477 , 0.02667447, ..., -0.08644801,
-0.1063043 , 0.0582094 ]]], dtype=float32)},
'pre_ffw_norm': {'scale.weights': Array([1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1.], dtype=float32)},
'mlp': {'gating_linear.weights': Array([[-0.08089793, -0.03153229, -0.01074452, ..., 0.01780026,
-0.03564561, -0.01420574],
[ 0.06600957, -0.05007182, -0.0437927 , ..., -0.04816006,
-0.08081407, 0.00309908],
[-0.01698056, 0.06722485, 0.00809888, ..., -0.00986961,
-0.04282206, -0.07243507],
...,
[ 0.07501657, 0.05629339, 0.02092532, ..., -0.0355814 ,
0.0854158 , 0.0613432 ],
[ 0.0807176 , 0.0145484 , -0.01655134, ..., -0.08058129,
-0.08101629, 0.05117289],
[-0.064259 , -0.05753384, -0.00182603, ..., -0.06319878,
-0.0658315 , -0.08707886]], dtype=float32),
'value_linear.weights': Array([[ 0.03410852, 0.08793687, 0.07743533, ..., -0.00509429,
0.07919045, -0.03491409],
[ 0.04716926, 0.04530573, 0.03945417, ..., 0.02323627,
0.02890669, -0.04586953],
[-0.06177489, 0.00236147, 0.06020578, ..., 0.00684867,
-0.0095549 , -0.03924415],
...,
[ 0.06370585, -0.03736426, 0.02687962, ..., -0.03440447,
0.07662794, -0.00730329],
[ 0.0340391 , 0.02393284, -0.03865454, ..., 0.05148604,
0.05237096, 0.00176928],
[ 0.08652971, -0.08391929, -0.07002031, ..., -0.01378862,
0.03854173, -0.04307928]], dtype=float32),
'out_linear.weights': Array([[ 0.05202093, -0.01135721, -0.05179439, ..., -0.0700395 ,
0.01688986, -0.06391211],
[-0.05734182, 0.03840903, -0.02613646, ..., 0.05300335,
0.02998099, 0.07031251],
[ 0.02262483, -0.0871903 , 0.00347703, ..., -0.05300643,
0.04024944, 0.04427659],
...,
[ 0.06980776, 0.05373394, -0.03647418, ..., -0.02644333,
-0.07305254, -0.04660885],
[-0.01393046, -0.00921861, -0.00229557, ..., -0.00906077,
0.03612191, -0.02497595],
[-0.04453076, -0.02970805, -0.00290809, ..., 0.04809204,
0.04824762, 0.05788305]], dtype=float32)}},
'block_2': {'pre_attention_norm': {'scale.weights': Array([1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1.], dtype=float32)},
'attention': {'query.weights': Array([[[ 0.01755337, 0.03235588, 0.02417226, ..., 0.06062951,
0.05328447, 0.09850284],
[ 0.08647591, 0.06817192, -0.09126111, ..., -0.07069191,
0.01473027, 0.06057715],
[-0.09377875, 0.09156042, -0.04892204, ..., -0.0848496 ,
-0.05146926, 0.03945245],
...,
[-0.03602443, -0.10505116, -0.00131889, ..., -0.10437635,
0.01325453, -0.03520815],
[-0.06228605, 0.03242642, 0.04671137, ..., -0.02177409,
0.02338269, -0.0361791 ],
[ 0.08959603, -0.04914165, 0.09938632, ..., -0.02710926,
0.10024534, 0.10658701]],
[[ 0.00908868, -0.02214644, -0.00982281, ..., -0.00581629,
0.0985399 , 0.07815754],
[ 0.00496968, -0.06058161, 0.09214297, ..., 0.03494117,
-0.09270494, 0.07180815],
[-0.06532727, -0.00135312, -0.07768055, ..., 0.01538875,
0.01455869, 0.01117719],
...,
[-0.05607906, -0.0914299 , -0.01621182, ..., 0.00894183,
0.08299221, -0.06493672],
[ 0.02387344, -0.07909773, 0.04799945, ..., 0.10523728,
-0.01489292, 0.07780013],
[-0.04796832, 0.02781169, 0.06698445, ..., -0.03257596,
0.03782994, 0.01950166]],
[[ 0.04542089, 0.00290791, 0.08411852, ..., 0.09208691,
0.01250721, -0.03703425],
[-0.09001745, -0.00538057, 0.07532512, ..., -0.04876916,
-0.02885814, 0.08180851],
[ 0.0330164 , 0.01850554, 0.09566546, ..., 0.0476537 ,
-0.02270684, -0.08581506],
...,
[-0.02563308, -0.00745638, -0.01811052, ..., -0.02911007,
-0.029689 , 0.09874327],
[-0.10374328, 0.07001312, 0.06616198, ..., 0.04381561,
0.03436472, 0.07600383],
[ 0.08268725, -0.04912534, -0.08299144, ..., 0.01438656,
-0.01168226, 0.0054762 ]],
...,
[[-0.0011254 , 0.03387617, 0.02213103, ..., -0.09806823,
0.04686868, 0.04931837],
[-0.04096489, 0.08369348, -0.00264868, ..., 0.04710089,
-0.03203285, 0.04015829],
[-0.06039619, 0.07052769, -0.03045281, ..., 0.01596686,
-0.06037598, -0.06036556],
...,
[ 0.00914753, -0.08984248, 0.05557941, ..., -0.06512503,
-0.08466571, -0.04769234],
[-0.03241387, -0.0834079 , -0.06768012, ..., -0.02649236,
0.04416613, -0.0677983 ],
[ 0.01819887, 0.05456027, 0.00633764, ..., 0.10250608,
0.08515913, -0.0110355 ]],
[[-0.10773205, 0.0776008 , 0.0167469 , ..., -0.03274762,
-0.06320469, 0.07155073],
[ 0.09565 , 0.02503368, 0.0787857 , ..., 0.03807895,
-0.0288705 , 0.08907047],
[ 0.05832498, 0.0869164 , 0.05795952, ..., -0.07210424,
-0.00082167, -0.0352679 ],
...,
[-0.02736653, -0.01328702, -0.04021299, ..., 0.06598425,
0.00819782, -0.07488001],
[-0.07500075, -0.09322679, 0.04855791, ..., 0.08405361,
0.03612431, 0.0867609 ],
[-0.03747668, -0.07933833, -0.10130823, ..., -0.00101876,
0.01828982, 0.05340965]],
[[-0.06867139, 0.0478856 , 0.08576667, ..., -0.03440511,
-0.09709105, 0.01058352],
[ 0.06535612, 0.0727877 , -0.08467285, ..., -0.00082379,
0.0210804 , -0.03218236],
[ 0.09712334, 0.00353465, -0.08094877, ..., -0.03459868,
0.04990058, 0.04759036],
...,
[-0.02367419, -0.03644334, 0.00603268, ..., 0.1073398 ,
-0.03873128, -0.09760033],
[-0.04535456, -0.09480561, -0.10803235, ..., 0.09298503,
-0.07487813, -0.03293414],
[ 0.01703981, -0.05606184, -0.09018663, ..., 0.00484923,
0.04693455, -0.09086478]]], dtype=float32),
'key.weights': Array([[[-1.04799286e-01, -1.93883013e-02, 3.48221399e-02, ...,
8.58072657e-03, -3.98466699e-02, 3.18516642e-02],
[-7.31944889e-02, -5.67956567e-02, -3.63895819e-02, ...,
-4.38713878e-02, 5.05844541e-02, -1.06210448e-01],
[ 9.87350419e-02, -1.04384217e-02, -1.00430034e-01, ...,
-5.83904050e-02, 1.99529119e-02, 6.53781369e-02],
...,
[-3.09822951e-02, 4.42788936e-02, 4.72039171e-02, ...,
-2.85852570e-02, 8.04156438e-02, 6.58106282e-02],
[ 7.15315789e-02, -8.66559818e-02, -7.43940920e-02, ...,
4.85284403e-02, 1.02461994e-01, 9.54393446e-02],
[-8.89927074e-02, -7.62421116e-02, 7.10136294e-02, ...,
-4.06188406e-02, -2.41206158e-02, 6.62450045e-02]],
[[ 1.05353363e-01, 6.77424222e-02, -1.63710862e-02, ...,
6.95649683e-02, 6.51726425e-02, 1.10521214e-02],
[-7.47442022e-02, 9.85617340e-02, 4.94170897e-02, ...,
-4.61003520e-02, -2.98494361e-02, 1.57667287e-02],
[-3.12720612e-02, 1.72811579e-02, -5.42638404e-03, ...,
3.77200656e-02, -5.89501634e-02, -1.05217196e-01],
...,
[-1.03954613e-01, -4.94058095e-02, -7.80334473e-02, ...,
-1.03233881e-01, -6.64335676e-03, 1.06122829e-01],
[-2.09524389e-02, -3.03085372e-02, 7.31081069e-02, ...,
-9.71986800e-02, -4.66528311e-02, -5.86544871e-02],
[-1.03732087e-01, 8.86567980e-02, 8.68634433e-02, ...,
9.47011337e-02, 5.64000495e-02, -1.03176482e-01]],
[[-1.08030178e-01, -7.81500265e-02, 8.64618719e-02, ...,
5.65487370e-02, 7.67293125e-02, 9.09327567e-02],
[-3.37406658e-02, 7.85447881e-02, -3.00090164e-02, ...,
8.75190571e-02, 6.77076578e-02, 5.17221950e-02],
[ 7.03513324e-02, -3.71828638e-02, 1.02530316e-01, ...,
1.29910139e-02, -3.49869095e-02, -1.16269523e-02],
...,
[ 1.01607755e-01, -1.11362869e-02, -4.44073752e-02, ...,
-9.09046531e-02, -8.31474587e-02, -5.71715720e-02],
[-9.98519510e-02, -5.57775749e-03, -6.97672367e-02, ...,
1.01959951e-01, -6.07526787e-02, -4.28216346e-02],
[-4.03581373e-02, 3.12189944e-02, -7.40883797e-02, ...,
-3.84518169e-02, 7.06441700e-02, 8.18530843e-02]],
...,
[[-2.61182766e-02, 4.31365110e-02, -5.02565205e-02, ...,
9.76574942e-02, 1.10410748e-03, 7.82652944e-02],
[-5.81861213e-02, -2.14870069e-02, -1.00843117e-01, ...,
-1.05679028e-01, 1.07371129e-01, -4.23024483e-02],
[-1.01244614e-01, 7.02015311e-02, -3.54290791e-02, ...,
-2.86655258e-02, 5.60703315e-02, 8.24645609e-02],
...,
[-7.69558176e-02, -6.03331178e-02, -1.45218046e-02, ...,
7.58626834e-02, 7.25767091e-02, 5.52787781e-02],
[-8.57743844e-02, -4.33259532e-02, 7.25563755e-03, ...,
9.22037512e-02, -1.00563444e-01, -5.77028394e-02],
[-6.53174892e-02, -3.74783319e-03, -4.89590988e-02, ...,
2.57156212e-02, -3.89452428e-02, 3.49399610e-03]],
[[-5.11428192e-02, -4.25557196e-02, 9.34901536e-02, ...,
-9.64821801e-02, -1.01750404e-01, 8.50380063e-02],
[-2.04386488e-02, -8.36359560e-02, 6.10307492e-02, ...,
-3.68954726e-02, 3.95211317e-02, -9.32393894e-02],
[-5.67030534e-02, -4.74168472e-02, 1.76947820e-03, ...,
-2.85970018e-05, -8.68118778e-02, 1.01169012e-01],
...,
[-7.71000981e-02, -2.04974171e-02, -7.50842690e-02, ...,
6.83007091e-02, 4.80693392e-02, -1.01747975e-01],
[-5.73398508e-02, 4.58594738e-03, -1.05788000e-02, ...,
-2.57795770e-02, -1.02661505e-01, -5.88585138e-02],
[ 7.51079060e-03, -7.10615069e-02, -9.51858908e-02, ...,
5.90625368e-02, -2.61892267e-02, -1.50518045e-02]],
[[ 2.44452227e-02, -2.21901108e-02, -7.49696195e-02, ...,
-5.08509129e-02, 9.34802145e-02, -4.20399643e-02],
[-5.33301793e-02, -5.24845570e-02, 5.13417870e-02, ...,
5.52253015e-02, 8.51315409e-02, -2.74079293e-02],
[-4.04833667e-02, 6.05521128e-02, 1.91796310e-02, ...,
1.01392657e-01, 9.20326114e-02, -3.33425552e-02],
...,
[ 8.50186273e-02, 7.93596432e-02, 1.03750564e-01, ...,
-9.67104137e-02, -7.43158115e-03, -1.05579145e-01],
[ 8.55301693e-02, -3.68854329e-02, 1.80959906e-02, ...,
-7.18218321e-03, 4.46861163e-02, 1.05866715e-01],
[ 5.86891770e-02, 3.89873646e-02, -9.21328813e-02, ...,
-4.39911196e-03, 2.02374887e-02, -1.06101064e-02]]], dtype=float32),
'value.weights': Array([[[ 0.03254974, 0.03891866, 0.06365793, ..., -0.00384119,
0.08047034, 0.04323939],
[ 0.08477057, 0.04564758, 0.03586505, ..., -0.03751988,
0.06870259, 0.03309646],
[-0.04540897, -0.10820566, -0.04378864, ..., 0.09461617,
0.02789255, -0.00201568],
...,
[ 0.06512205, 0.0704379 , 0.01785741, ..., -0.02463931,
0.05232085, -0.08698598],
[-0.01506631, -0.01865988, -0.10435768, ..., -0.09215915,
0.09173861, 0.03160118],
[ 0.05708018, 0.04399507, -0.03674529, ..., 0.01503118,
-0.07015222, -0.01262408]],
[[-0.03395303, -0.07199326, -0.04814032, ..., -0.03194252,
0.02472587, 0.08955747],
[ 0.00468769, 0.0562463 , 0.0563602 , ..., 0.08694198,
0.001645 , -0.09477087],
[-0.07030328, -0.04059853, -0.07450964, ..., 0.1034606 ,
-0.07015144, -0.06483996],
...,
[ 0.1026734 , -0.09937306, -0.04187603, ..., -0.02035306,
-0.0857153 , -0.03446236],
[ 0.04551362, 0.02410627, -0.00320715, ..., 0.06892422,
-0.0360514 , -0.07498795],
[-0.0698919 , 0.05123468, -0.04365885, ..., -0.07072472,
-0.0741227 , -0.00420647]],
[[-0.09064403, 0.03899999, 0.07843636, ..., -0.073202 ,
0.04325849, 0.04108483],
[ 0.05985652, -0.00407778, 0.0493129 , ..., -0.03029943,
0.01219412, 0.04321717],
[ 0.01324645, -0.07711729, 0.07147224, ..., 0.06768513,
-0.04207323, 0.04438048],
...,
[ 0.09592224, 0.02049256, 0.01071087, ..., 0.07260355,
0.06706696, 0.09692541],
[-0.0141932 , -0.02448543, -0.02178124, ..., -0.06487317,
0.05483633, -0.10262699],
[-0.08033626, -0.08401665, -0.03233051, ..., -0.01850118,
-0.01262705, -0.02531056]],
...,
[[ 0.05416685, -0.10268357, 0.01316515, ..., 0.03409586,
0.01407212, -0.05821077],
[ 0.02797876, -0.01812616, -0.05893721, ..., 0.00345399,
-0.08286285, 0.05225266],
[ 0.0288162 , -0.02281904, 0.06725266, ..., -0.08667645,
0.09027345, -0.09334092],
...,
[-0.02703116, -0.07102305, -0.0191635 , ..., -0.03585148,
0.01895919, 0.04753746],
[-0.04104738, 0.08158077, -0.0741441 , ..., 0.05586649,
0.09078265, -0.05796272],
[-0.10766095, -0.08614171, 0.07042135, ..., -0.10649662,
-0.01831589, 0.05045265]],
[[-0.09268156, -0.047177 , -0.0090749 , ..., -0.08065082,
-0.02207048, -0.03322574],
[-0.04439914, -0.02679678, 0.0979638 , ..., 0.05409698,
0.03274372, -0.05326832],
[ 0.00958196, 0.0602538 , -0.1053107 , ..., -0.08554963,
-0.00168227, -0.07800725],
...,
[ 0.03830114, -0.02066252, -0.06822318, ..., -0.03218345,
-0.01589474, 0.02839669],
[ 0.02609964, 0.07964686, 0.10769816, ..., -0.03041552,
-0.07737886, 0.00393818],
[ 0.01710539, -0.07579537, -0.05425793, ..., -0.04391849,
-0.00867981, 0.09422779]],
[[ 0.06336933, -0.03486093, -0.09560946, ..., -0.10788988,
-0.0773163 , -0.04610673],
[ 0.08774244, -0.06206582, 0.0219076 , ..., 0.00480486,
0.02103093, -0.08648717],
[-0.05080495, -0.10245033, 0.01523498, ..., 0.04842987,
-0.06493558, 0.00811683],
...,
[-0.05129411, 0.03377654, -0.00337561, ..., -0.06918637,
-0.07440086, 0.10049123],
[-0.01078024, -0.1041621 , 0.08638878, ..., 0.05044583,
-0.05729177, 0.104091 ],
[ 0.08788137, 0.0781339 , -0.10009015, ..., 0.05297336,
-0.06803397, -0.02180937]]], dtype=float32),
'output.weights': Array([[[-0.02202173, 0.08784648, 0.07826325, ..., 0.06986426,
0.07121719, -0.0077413 ],
[ 0.06909923, 0.09689498, 0.07940133, ..., 0.01958833,
-0.05243688, -0.06777773],
[ 0.09834607, -0.05469471, -0.01452864, ..., 0.08172969,
0.10540383, -0.07309654],
...,
[-0.0714167 , -0.07182341, -0.08362138, ..., 0.00567261,
-0.01867454, -0.03947811],
[ 0.0932549 , 0.00470896, 0.03396108, ..., 0.01198772,
-0.08154556, 0.0963036 ],
[ 0.07223972, -0.0766269 , 0.09423969, ..., -0.04671888,
-0.0858539 , 0.02485425]],
[[ 0.00876392, 0.07329445, 0.06330424, ..., -0.10187911,
-0.02813116, 0.08387797],
[-0.04533572, 0.00169533, -0.08260845, ..., 0.08694193,
-0.00743829, -0.07884322],
[-0.00826084, 0.05548427, -0.01123614, ..., 0.01235783,
0.10193101, -0.09037171],
...,
[ 0.01133538, 0.05179461, -0.05804554, ..., 0.07091166,
-0.09907387, -0.0008585 ],
[-0.04461734, 0.05796 , 0.03529128, ..., 0.06669597,
0.06259491, 0.0227477 ],
[-0.06992643, -0.0610083 , -0.04353047, ..., 0.09480642,
-0.06496919, 0.01817128]],
[[-0.07439383, 0.00528895, -0.02382788, ..., -0.04739535,
0.08770195, 0.0387319 ],
[ 0.06508386, -0.06385019, -0.00049407, ..., -0.05692527,
0.04908425, -0.00347614],
[ 0.07541403, -0.09832018, -0.01733128, ..., 0.06635382,
0.07256084, 0.02082437],
...,
[-0.00813203, 0.02234342, 0.1027118 , ..., 0.02035461,
0.00977904, 0.08726434],
[ 0.03533157, 0.07969873, 0.02277622, ..., 0.01804391,
0.04764402, -0.0125913 ],
[-0.04348579, -0.05105494, -0.10319097, ..., -0.00807499,
0.06475409, 0.00966367]],
...,
[[ 0.00781684, 0.08162534, -0.05007542, ..., 0.04404751,
0.09915785, 0.07804108],
[ 0.07919165, 0.01238658, -0.09090848, ..., 0.08665051,
-0.09601748, -0.10463508],
[ 0.01065504, -0.04111895, 0.06126105, ..., 0.07625174,
0.06838095, 0.05057037],
...,
[-0.05522354, 0.05784017, -0.06251552, ..., -0.01370682,
0.08571231, 0.09738156],
[ 0.06503642, -0.0211686 , 0.03029834, ..., 0.05537543,
0.10678118, 0.08103368],
[-0.10748678, -0.01669345, 0.05406854, ..., 0.10324599,
-0.07595772, 0.018848 ]],
[[ 0.08963475, -0.10052731, -0.021976 , ..., 0.09902922,
-0.00456791, 0.02279073],
[-0.05015315, 0.0978189 , 0.04923356, ..., -0.01544488,
0.03624804, 0.04066845],
[ 0.0647496 , -0.01818065, 0.08447716, ..., -0.06844576,
0.08238943, 0.08242366],
...,
[ 0.07900905, 0.0035954 , 0.09477624, ..., 0.04498977,
0.10752451, -0.03877485],
[ 0.0993968 , -0.09277497, -0.04189321, ..., -0.05066317,
-0.05137232, -0.06843967],
[-0.00814855, 0.00356665, -0.00572028, ..., 0.06100737,
0.01595669, 0.06203717]],
[[-0.01856167, -0.01330535, -0.00830813, ..., -0.01797541,
0.06080639, -0.02954943],
[-0.03034018, -0.02959991, 0.06917449, ..., -0.06271242,
0.07751852, -0.07460658],
[-0.08356717, -0.06862704, -0.0455941 , ..., -0.04885914,
0.02677797, 0.07318948],
...,
[-0.0002249 , -0.03408726, 0.09271654, ..., 0.09780076,
0.01568964, 0.01536214],
[ 0.02032565, 0.03469621, 0.0621513 , ..., -0.02674509,
0.04927426, 0.06885585],
[ 0.0561905 , 0.07488463, 0.00078482, ..., 0.01648785,
0.03937668, -0.04872598]]], dtype=float32)},
'pre_ffw_norm': {'scale.weights': Array([1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1.], dtype=float32)},
'mlp': {'gating_linear.weights': Array([[-0.01803297, 0.05755907, 0.05968959, ..., -0.07061078,
0.07045958, 0.03866632],
[-0.05948503, -0.03466835, -0.00013415, ..., 0.05702896,
0.06336534, -0.07293832],
[ 0.02374669, 0.02955221, -0.01612595, ..., -0.08524664,
0.00044773, -0.04452675],
...,
[-0.08090188, 0.04032872, 0.05337119, ..., 0.02463778,
-0.00180614, -0.04641896],
[-0.0689914 , 0.03322573, 0.02467797, ..., -0.0339951 ,
0.06350949, -0.02436109],
[-0.04552671, -0.08178674, -0.08203694, ..., 0.03100756,
0.00657358, -0.02037962]], dtype=float32),
'value_linear.weights': Array([[ 0.01053509, 0.07884613, -0.04493046, ..., 0.01667755,
-0.0247543 , -0.0298241 ],
[ 0.08757681, -0.0495735 , -0.02636608, ..., -0.00449637,
-0.06230682, -0.07240245],
[ 0.08377437, -0.03736409, 0.02422913, ..., -0.03157149,
0.0175609 , 0.0153856 ],
...,
[-0.00683727, 0.06680509, -0.04357953, ..., 0.01223854,
0.02020861, -0.0065296 ],
[-0.07040545, 0.00384955, -0.08585298, ..., -0.00225104,
-0.07745263, -0.00832135],
[ 0.04039122, -0.01912668, 0.08073986, ..., 0.04492306,
0.08034148, -0.03679447]], dtype=float32),
'out_linear.weights': Array([[-0.02619517, 0.07782717, 0.01617185, ..., -0.02671573,
-0.04182712, 0.01244988],
[ 0.08505649, -0.06492419, 0.04077269, ..., -0.06970031,
0.02384903, 0.03031469],
[-0.08684533, -0.00082222, -0.08128816, ..., 0.01531134,
-0.07000969, -0.07052723],
...,
[ 0.07456744, -0.05855554, -0.03644583, ..., 0.05829181,
0.06609578, 0.00756928],
[-0.03333074, -0.02918522, 0.08248016, ..., 0.06888045,
-0.06240146, -0.03228385],
[ 0.00777064, 0.03531116, 0.02988477, ..., 0.02102215,
0.01572802, 0.03600862]], dtype=float32)}},
'block_3': {'pre_attention_norm': {'scale.weights': Array([1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1.], dtype=float32)},
'attention': {'query.weights': Array([[[-0.04123504, 0.03378831, -0.03510667, ..., 0.02644477,
-0.0763295 , -0.04975713],
[ 0.06082151, 0.08779277, -0.07450288, ..., 0.0739423 ,
0.04722273, -0.07827725],
[ 0.02010586, -0.02588501, -0.03104135, ..., -0.09720709,
0.04843158, 0.07511389],
...,
[ 0.06593524, 0.06113556, -0.07440116, ..., 0.06624093,
-0.08399729, -0.04252467],
[ 0.02563765, -0.01421661, 0.09284383, ..., 0.03508772,
-0.08124382, -0.00845343],
[-0.06338151, 0.00236571, 0.06051079, ..., -0.03029963,
0.06065236, 0.0043572 ]],
[[-0.06946914, 0.05547088, 0.00499942, ..., 0.0566646 ,
-0.08726556, 0.1032241 ],
[-0.08679371, 0.08291476, 0.09818558, ..., -0.04386235,
-0.01050393, -0.06985499],
[-0.07590269, 0.05141467, -0.06506138, ..., 0.06046294,
-0.07920025, 0.08682904],
...,
[ 0.06544986, 0.08928872, 0.00261139, ..., -0.01034406,
-0.03731289, 0.01470717],
[ 0.03989948, 0.05850596, -0.08142284, ..., 0.1008114 ,
0.10009425, -0.09670564],
[-0.06567704, 0.08566487, 0.07644528, ..., -0.00880684,
0.07561594, -0.04908304]],
[[-0.01349381, 0.00068739, -0.10199082, ..., -0.06996682,
-0.08604474, -0.0707162 ],
[ 0.06398238, -0.09697119, 0.04588461, ..., 0.02916742,
-0.0719827 , 0.02253266],
[ 0.00655052, 0.08357357, -0.09994455, ..., 0.09212606,
-0.10334166, -0.09103006],
...,
[-0.02264648, -0.05569854, 0.01081449, ..., 0.09475745,
-0.05329996, -0.02641798],
[-0.0823271 , -0.08441264, 0.02749966, ..., 0.02483985,
-0.05631797, 0.01599452],
[-0.0473053 , -0.00621401, -0.00025776, ..., 0.09635755,
0.08066226, -0.04562306]],
...,
[[-0.10032657, -0.0522834 , 0.08695091, ..., -0.02303558,
0.06091771, -0.09945451],
[-0.01754929, -0.03036958, 0.00869491, ..., 0.03238868,
0.0673479 , 0.08038534],
[-0.0310543 , -0.01289867, 0.04531252, ..., 0.02399598,
-0.01267634, -0.07001932],
...,
[-0.07815568, -0.03053308, -0.07653414, ..., 0.09446023,
0.06472921, 0.06382572],
[-0.09000713, -0.06240271, -0.07098919, ..., 0.096144 ,
-0.08820523, -0.03216595],
[ 0.00819893, -0.00917509, -0.08370069, ..., -0.02839672,
0.10686955, -0.09492986]],
[[ 0.0859224 , 0.03258226, -0.07639918, ..., 0.00415988,
0.05079021, -0.09730235],
[-0.04113723, 0.09029356, 0.05930146, ..., 0.04372634,
-0.04825749, 0.10365543],
[ 0.08066843, 0.06602658, -0.08234643, ..., 0.09652905,
-0.0583735 , -0.03225597],
...,
[ 0.01870755, -0.01545405, -0.09924788, ..., -0.09828056,
-0.05405024, 0.08136092],
[-0.03040336, -0.08143014, 0.09516073, ..., 0.09094019,
0.03781378, 0.00030602],
[-0.08629733, -0.07508576, -0.03690908, ..., 0.09506015,
0.04468044, -0.03581036]],
[[-0.10362916, 0.00981045, 0.04616098, ..., -0.08367049,
0.0297504 , 0.05284297],
[ 0.06193236, -0.04369237, 0.041861 , ..., -0.1002138 ,
-0.01879928, 0.06823322],
[-0.08302189, 0.09516791, -0.02699028, ..., -0.0554563 ,
-0.03539181, -0.000561 ],
...,
[-0.04467334, 0.06377462, -0.10682544, ..., 0.07048043,
0.01146789, -0.07411865],
[-0.01604743, 0.02529023, -0.07976419, ..., 0.0074362 ,
0.05916118, 0.08060488],
[-0.01349564, -0.02411633, 0.07555126, ..., 0.01311622,
-0.07080785, 0.00129644]]], dtype=float32),
'key.weights': Array([[[ 0.0543819 , -0.03014341, -0.08131606, ..., 0.04287986,
-0.05074582, 0.05035769],
[-0.01419557, 0.00477428, -0.06892812, ..., -0.09675831,
0.08340666, 0.041712 ],
[ 0.03656576, -0.02452038, -0.02729671, ..., -0.01148805,
0.01071846, -0.07943424],
...,
[-0.07153532, 0.08338165, -0.05201191, ..., 0.01633149,
0.03957564, 0.04441143],
[ 0.0602966 , -0.09252841, -0.07575018, ..., -0.0039133 ,
0.00849434, 0.06683197],
[-0.09898733, -0.10108348, 0.08609672, ..., 0.02151481,
0.0675569 , -0.0389017 ]],
[[-0.01454785, -0.01158862, -0.08654774, ..., -0.01741385,
0.09398724, 0.09692048],
[ 0.01955235, 0.01273692, -0.02573255, ..., -0.08161654,
0.08440273, 0.09576728],
[ 0.01346245, 0.00795562, -0.09888058, ..., -0.09737018,
-0.05472075, 0.08048161],
...,
[ 0.04146263, 0.00564246, -0.02273681, ..., -0.06787284,
-0.082763 , -0.0052261 ],
[ 0.1076443 , 0.04521374, 0.02287871, ..., -0.03944123,
0.10295868, -0.09109121],
[-0.07972264, -0.0565644 , -0.05397377, ..., 0.05698564,
0.09432336, 0.09410796]],
[[-0.06046575, -0.01177987, 0.0277849 , ..., 0.0484809 ,
0.08232922, -0.07782238],
[-0.00890172, -0.0365289 , 0.03738725, ..., 0.0679352 ,
-0.00174829, -0.04727275],
[ 0.10448906, -0.06152518, 0.01767351, ..., -0.03475439,
0.0924405 , -0.01471453],
...,
[-0.00578506, -0.0250926 , -0.04100286, ..., -0.08780054,
0.05158915, 0.04901601],
[-0.07273779, -0.0265119 , 0.09412801, ..., -0.02229516,
0.03694911, 0.06149008],
[ 0.07087281, -0.09077325, -0.08281229, ..., 0.01079114,
-0.02953394, 0.00588259]],
...,
[[ 0.00025095, -0.09490909, -0.05351157, ..., 0.05227682,
0.10662281, -0.0515701 ],
[-0.10236459, -0.02390387, -0.09765868, ..., -0.0139286 ,
0.09207695, -0.00703419],
[ 0.04292446, 0.10502579, 0.10490712, ..., 0.08237214,
0.04629638, 0.00329069],
...,
[-0.05404854, 0.04226877, 0.01523405, ..., -0.06392842,
0.02119208, -0.06499525],
[ 0.04707719, 0.06268316, 0.00695692, ..., 0.09324659,
-0.08843749, -0.07887597],
[-0.09515218, -0.07652191, -0.09777224, ..., 0.0591665 ,
0.03744191, -0.0532599 ]],
[[-0.08026254, 0.03558967, -0.03529748, ..., -0.07335887,
-0.07025661, -0.02559076],
[-0.06010692, -0.09082521, 0.00597566, ..., 0.00735118,
0.05083734, 0.09982093],
[ 0.0348225 , 0.01310465, -0.0623013 , ..., -0.10547237,
0.0260288 , 0.03716833],
...,
[ 0.09503374, 0.00640872, 0.0901204 , ..., 0.07036085,
-0.10509914, -0.10244548],
[ 0.0182629 , -0.07611688, -0.05891966, ..., -0.00405504,
0.04175097, -0.00876526],
[-0.02469625, 0.04479741, 0.06389195, ..., 0.00425827,
-0.00756791, 0.07091618]],
[[-0.01199004, -0.07563724, -0.098108 , ..., 0.05404314,
0.08501336, -0.05924646],
[ 0.06697872, -0.00542282, 0.09113402, ..., -0.10269345,
-0.03936589, -0.00286739],
[ 0.06864741, 0.0418108 , 0.02685759, ..., 0.08494452,
0.05737235, -0.04858527],
...,
[ 0.00311454, 0.08728659, 0.0478977 , ..., -0.02047187,
0.09992923, 0.06345914],
[-0.03042207, -0.0131571 , -0.08456453, ..., -0.10458019,
-0.03285434, 0.08346169],
[ 0.03685137, 0.08960677, 0.05498161, ..., -0.04870547,
-0.01265557, -0.07854685]]], dtype=float32),
'value.weights': Array([[[-8.80052056e-03, 4.52095084e-02, -6.69654235e-02, ...,
4.83097807e-02, -5.53043559e-02, -9.12387595e-02],
[ 2.12478563e-02, -8.48218575e-02, 1.45853227e-02, ...,
7.16468692e-02, 3.55565771e-02, -2.73930617e-02],
[ 1.90673340e-02, 8.08892250e-02, -8.01817328e-02, ...,
3.72339413e-02, -9.66015756e-02, -6.10547774e-02],
...,
[ 7.66569749e-02, 2.11360753e-02, -1.60564929e-02, ...,
1.28932474e-02, 2.38113143e-02, 2.02686917e-02],
[-1.19449776e-02, 8.68523195e-02, 7.34931827e-02, ...,
7.64743164e-02, -4.92026620e-02, -9.00356472e-03],
[-2.67770644e-02, 3.02052218e-02, 1.04197823e-01, ...,
-7.14716241e-02, -8.17169100e-02, -6.14190300e-04]],
[[-1.98049452e-02, -8.46333131e-02, 4.59913583e-03, ...,
-8.86092037e-02, 8.54496956e-02, 8.22807252e-02],
[-8.03554282e-02, 6.97607547e-02, 1.01278342e-01, ...,
6.47625551e-02, -1.57850292e-02, 4.79380712e-02],
[ 9.29231644e-02, 8.21038485e-02, -3.31785865e-02, ...,
1.54021438e-02, 7.30178505e-03, 4.81003113e-02],
...,
[-2.69672815e-02, -6.04080409e-02, 5.54760136e-02, ...,
-2.82640569e-03, -3.99361514e-02, 6.28534704e-02],
[-8.45263348e-05, 1.75993629e-02, 7.98781887e-02, ...,
3.94128114e-02, 9.07731503e-02, -1.47140091e-02],
[-3.38595212e-02, -6.45041242e-02, 1.01184964e-01, ...,
9.23938379e-02, 9.44058746e-02, -8.35782215e-02]],
[[-8.44409317e-02, 5.89807481e-02, -1.04752906e-01, ...,
2.81123482e-02, -6.75336272e-02, -8.54845420e-02],
[ 1.97678581e-02, 2.22460665e-02, 6.06509633e-02, ...,
8.45247880e-02, 9.20265168e-02, -8.87683928e-02],
[ 6.25506490e-02, 2.38028225e-02, -3.96143310e-02, ...,
9.91790742e-02, 7.06546977e-02, -3.19645554e-02],
...,
[-6.17183670e-02, 3.93583253e-02, 4.55662757e-02, ...,
9.18723568e-02, 3.32082435e-02, -2.18332950e-02],
[ 3.88060771e-02, -7.14220181e-02, -8.28264132e-02, ...,
9.52897742e-02, 3.30309048e-02, 4.41720430e-03],
[ 7.98830613e-02, -6.41740933e-02, 9.31313708e-02, ...,
3.62505205e-02, 1.70220286e-02, 8.18808004e-02]],
...,
[[-8.61900970e-02, 5.74140809e-02, -7.28602782e-02, ...,
1.04825040e-02, -1.04653202e-01, 7.27335066e-02],
[-1.08027518e-01, 9.69373286e-02, 5.48332259e-02, ...,
5.89188561e-03, -6.50141239e-02, -1.41911581e-02],
[-7.03090578e-02, -6.46282434e-02, 1.03098638e-01, ...,
-9.91477668e-02, -2.01168284e-02, -1.39875207e-02],
...,
[-5.65977506e-02, -5.68299592e-02, 7.03673586e-02, ...,
1.82661805e-02, -2.31509246e-02, 1.35324728e-02],
[-4.86356281e-02, -2.38872971e-02, 8.20774436e-02, ...,
8.33477154e-02, 5.24764247e-02, -1.42868860e-02],
[ 5.10644615e-02, -8.59946162e-02, 4.43888418e-02, ...,
2.98033399e-03, 3.60232405e-02, 2.92082746e-02]],
[[ 8.55887607e-02, -2.03140657e-02, 1.21714305e-02, ...,
-2.83787549e-02, -1.06411405e-01, -5.96394837e-02],
[ 9.11869854e-02, -7.80789480e-02, -9.47006196e-02, ...,
-8.75181779e-02, -5.00206724e-02, -4.15807627e-02],
[-3.24688219e-02, 9.88084972e-02, 2.04912741e-02, ...,
3.22868414e-02, -7.98757095e-03, -6.28322810e-02],
...,
[-7.53486082e-02, -8.11391920e-02, 5.02582751e-02, ...,
6.68044761e-02, 7.12311268e-02, 1.40013034e-02],
[ 5.38392738e-02, -9.01111364e-02, 4.52190079e-02, ...,
5.37963808e-02, -7.72650167e-02, 2.46873423e-02],
[-5.07767871e-02, 2.49458514e-02, 5.14727421e-02, ...,
1.54662803e-02, 5.75904101e-02, 6.40576184e-02]],
[[ 9.30183530e-02, 8.54698047e-02, -1.22510791e-02, ...,
2.33316422e-02, 5.90705909e-02, -4.21627425e-02],
[ 9.17701274e-02, -5.00716716e-02, 5.03999703e-02, ...,
6.08416945e-02, 2.21106950e-02, 6.32006628e-03],
[ 1.84445754e-02, -1.78825986e-02, -9.59204882e-03, ...,
-7.35926777e-02, -5.12556843e-02, 2.03676205e-02],
...,
[-5.61602786e-02, -1.72908101e-02, -1.55365337e-02, ...,
-6.67966083e-02, -3.08601651e-02, 6.99174777e-02],
[-2.33309194e-02, -8.89001042e-02, 2.48466129e-03, ...,
-7.73471221e-02, -8.30952749e-02, 3.81505415e-02],
[ 4.84652333e-02, -7.13701639e-03, -9.14538801e-02, ...,
7.04966411e-02, 8.20040405e-02, -9.21923444e-02]]], dtype=float32),
'output.weights': Array([[[-0.04153565, 0.09195428, -0.01244184, ..., -0.03339841,
-0.02162927, -0.05637999],
[ 0.01475466, 0.007857 , 0.06213127, ..., -0.10160078,
-0.07778025, 0.05753492],
[ 0.04014307, 0.10520189, -0.09253264, ..., 0.02465431,
0.01508303, -0.06835672],
...,
[-0.00910518, 0.0430533 , 0.0014784 , ..., 0.04589728,
0.0372383 , -0.02727581],
[ 0.07981338, -0.10663319, -0.04067374, ..., -0.07852409,
-0.07333231, 0.08999678],
[ 0.06314308, 0.03145211, 0.08830152, ..., 0.06678845,
-0.02607497, 0.06493688]],
[[ 0.01239084, 0.05437459, -0.06656974, ..., 0.09382534,
-0.10686932, -0.06377684],
[ 0.09922767, 0.066201 , 0.04182281, ..., 0.04805463,
-0.07718468, 0.03450133],
[ 0.03691821, -0.00532679, -0.06904075, ..., -0.03995752,
0.05123168, -0.06153848],
...,
[ 0.06891211, 0.07145606, -0.02336014, ..., 0.05466897,
-0.04077912, 0.00073746],
[ 0.01303845, 0.00890957, -0.00340289, ..., 0.09062558,
0.06199773, 0.07566266],
[ 0.03106169, -0.00355986, 0.0333766 , ..., 0.09588771,
-0.02951832, -0.08973902]],
[[ 0.00113758, -0.03658953, 0.08980581, ..., 0.05254727,
-0.05651359, 0.02118875],
[ 0.04968337, -0.00739049, 0.07345299, ..., -0.04807096,
-0.09029738, 0.01192495],
[ 0.01663945, -0.10660162, 0.01294236, ..., 0.05593282,
0.08940264, 0.02041867],
...,
[-0.02979707, 0.07755595, -0.04366819, ..., 0.00663448,
0.07725712, -0.05840403],
[ 0.05698087, 0.03538626, 0.0470902 , ..., -0.08141897,
-0.0337114 , -0.03386189],
[ 0.07845226, -0.05876831, 0.01404706, ..., -0.08919108,
0.05991988, -0.02660515]],
...,
[[-0.04614436, -0.10140692, -0.07417714, ..., -0.00766312,
-0.09898168, 0.03562079],
[-0.01752906, 0.09800747, 0.01501916, ..., -0.03461099,
0.00800489, 0.04065962],
[-0.06746157, 0.02043689, 0.10115327, ..., -0.00564935,
0.0763222 , 0.10552185],
...,
[-0.0846168 , 0.00828965, -0.04784472, ..., 0.04003668,
0.05651335, 0.10748291],
[-0.08948845, 0.01083545, 0.07004206, ..., 0.05141354,
0.06254265, 0.03298891],
[-0.01179887, -0.04739168, -0.01153223, ..., -0.03259991,
-0.09882401, 0.05234766]],
[[-0.0600401 , -0.02016522, 0.05538008, ..., 0.03900053,
-0.05520801, 0.02227273],
[ 0.07116552, 0.02808202, 0.04852787, ..., 0.08479736,
-0.09929509, -0.10341009],
[ 0.05668271, -0.04167035, 0.0104346 , ..., 0.03618912,
-0.09929366, -0.06956141],
...,
[-0.04610822, -0.07338437, 0.04492767, ..., 0.10582393,
-0.03161006, -0.00216119],
[ 0.01778764, 0.02363454, 0.03343005, ..., 0.01182205,
0.04707338, 0.03061433],
[-0.02247417, -0.06529426, 0.02178178, ..., -0.02712204,
-0.05681928, -0.06397797]],
[[ 0.02978305, 0.01601582, -0.07215331, ..., 0.08467213,
0.06048506, -0.01232698],
[ 0.00402885, 0.0783093 , -0.080074 , ..., -0.06481265,
-0.00520034, -0.0322146 ],
[-0.00061597, -0.00033849, 0.02136307, ..., -0.08391811,
-0.04071377, -0.07887102],
...,
[ 0.10436961, 0.06024759, -0.00106039, ..., 0.06609862,
0.08243117, -0.04284556],
[ 0.03445998, -0.09555074, 0.09279913, ..., -0.10392344,
-0.03540028, -0.06269746],
[-0.08601366, 0.0592032 , -0.04733537, ..., 0.07286477,
0.04824678, 0.02591474]]], dtype=float32)},
'pre_ffw_norm': {'scale.weights': Array([1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1.], dtype=float32)},
'mlp': {'gating_linear.weights': Array([[-0.00919707, -0.03985155, 0.00277069, ..., -0.02037098,
-0.05282705, -0.07433735],
[ 0.02430362, 0.06468734, -0.04600343, ..., -0.07247271,
0.033275 , 0.08501213],
[ 0.04627404, -0.00102965, -0.0063793 , ..., 0.0155588 ,
-0.04000819, 0.00842301],
...,
[ 0.04377452, -0.03213362, 0.06367154, ..., 0.01864644,
-0.08590107, -0.02871277],
[ 0.00849763, -0.06056171, 0.05344926, ..., 0.07635268,
0.05290833, 0.08753373],
[ 0.03788637, -0.01471882, -0.00314928, ..., 0.07609116,
-0.07676279, 0.06870718]], dtype=float32),
'value_linear.weights': Array([[-0.01735538, -0.05237205, 0.03695389, ..., 0.02271603,
0.0478003 , -0.01584856],
[-0.07980724, 0.01101032, -0.07632042, ..., -0.05007159,
-0.06241091, 0.0360679 ],
[-0.05267033, 0.03482491, -0.08806719, ..., -0.05064091,
-0.00891682, -0.04027412],
...,
[ 0.07320815, -0.02409468, 0.03526707, ..., 0.00849143,
-0.07183015, 0.00383165],
[ 0.0132201 , -0.01701839, -0.02345004, ..., -0.07454559,
-0.08764587, -0.06400724],
[-0.03058864, -0.01201937, 0.0184843 , ..., -0.05794595,
-0.03299137, -0.03433651]], dtype=float32),
'out_linear.weights': Array([[-0.01750359, -0.02438317, 0.020488 , ..., -0.05932162,
0.07406198, 0.01428911],
[ 0.06397137, 0.0550986 , -0.06643721, ..., -0.08503746,
0.06862965, -0.01414688],
[ 0.08595247, 0.03926173, 0.04231643, ..., -0.0875468 ,
-0.03858118, -0.01353049],
...,
[-0.0861691 , 0.00513972, 0.03516499, ..., 0.00666182,
-0.00539821, 0.03406742],
[ 0.0238536 , -0.05243475, 0.04766111, ..., 0.0785231 ,
-0.08738928, 0.03006242],
[ 0.07041101, 0.07621492, -0.02132458, ..., -0.00335847,
-0.06100573, 0.00181033]], dtype=float32)}},
'block_4': {'pre_attention_norm': {'scale.weights': Array([1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1.], dtype=float32)},
'attention': {'query.weights': Array([[[ 0.00946109, -0.06165152, 0.08756559, ..., -0.06038262,
-0.04679282, 0.00313772],
[-0.01840439, 0.04578842, 0.02120935, ..., -0.03176022,
-0.00195969, -0.09538031],
[ 0.09750411, 0.08618093, -0.07484504, ..., 0.10232089,
-0.07429451, -0.10234521],
...,
[-0.00222641, -0.03400256, -0.10153897, ..., 0.00497898,
0.08469229, 0.06377002],
[ 0.02963736, 0.0813593 , -0.01649714, ..., 0.05036084,
0.08925514, -0.07352929],
[ 0.05171233, -0.05960578, -0.01651603, ..., -0.08680008,
-0.02531077, 0.0029855 ]],
[[-0.04015404, -0.01868657, -0.05949725, ..., 0.05517203,
-0.05508477, 0.04692463],
[-0.06148683, 0.00811964, 0.0671607 , ..., -0.00732589,
0.00645188, 0.08396495],
[-0.00446033, 0.04813523, -0.05082513, ..., -0.05179882,
-0.03440457, 0.07524367],
...,
[ 0.07017918, 0.02565218, -0.04608278, ..., 0.08841094,
0.10771128, 0.08818332],
[ 0.03054446, 0.10797595, -0.0934542 , ..., -0.05198287,
-0.07081217, -0.06124649],
[ 0.09073792, 0.02135724, 0.10155505, ..., -0.05134741,
0.05216305, 0.0165802 ]],
[[ 0.07285687, -0.02049127, 0.02650387, ..., 0.08190891,
0.0119536 , 0.08835521],
[-0.02585874, -0.08813689, 0.09196721, ..., 0.04603172,
-0.04500097, 0.04130793],
[-0.00449463, 0.10380735, -0.06702977, ..., -0.0913668 ,
-0.09516153, 0.08627971],
...,
[-0.10744832, 0.02221215, -0.05872616, ..., -0.02718509,
0.07209475, -0.060017 ],
[-0.02625365, -0.05203552, 0.01775672, ..., -0.01801761,
0.0650748 , 0.00668215],
[ 0.05446575, -0.05114865, -0.01648638, ..., 0.03795382,
0.08220308, -0.0500998 ]],
...,
[[-0.00279507, -0.01035227, -0.00023226, ..., 0.01940892,
-0.07541347, -0.03428551],
[-0.00258152, -0.04929421, -0.08921433, ..., 0.02159778,
-0.0014051 , -0.08737618],
[-0.03588273, 0.03046925, -0.06202927, ..., 0.08034611,
-0.00679081, -0.04363822],
...,
[ 0.00866572, -0.050884 , 0.09932761, ..., 0.0133356 ,
0.00331379, -0.06068738],
[ 0.05303737, -0.05591047, 0.06894948, ..., 0.0506338 ,
-0.03683129, -0.00743148],
[-0.08904714, 0.07872427, -0.10670385, ..., -0.06946359,
-0.08877012, -0.06798826]],
[[ 0.05473014, -0.00535881, -0.06605192, ..., -0.05219046,
-0.02737631, 0.07998065],
[ 0.01249067, 0.09565436, -0.00784539, ..., -0.08433413,
-0.00071978, 0.1066178 ],
[ 0.02430255, 0.06614412, 0.07423588, ..., 0.06040711,
-0.06204672, -0.00212387],
...,
[-0.06857744, -0.04108599, 0.04507249, ..., 0.0021696 ,
-0.06684474, -0.05642684],
[-0.05242027, 0.01157528, 0.02406595, ..., -0.07867585,
-0.01301835, -0.06310228],
[ 0.01445842, 0.00721808, 0.03091824, ..., -0.01451053,
0.03766842, 0.08284838]],
[[-0.01395879, -0.0618486 , 0.0219682 , ..., -0.06882658,
-0.10674775, 0.01224667],
[ 0.09653932, 0.09171487, -0.01686709, ..., 0.06990581,
0.10376403, 0.10724133],
[-0.02180663, 0.03010702, 0.01550783, ..., -0.06372052,
-0.0730338 , -0.07740021],
...,
[ 0.00586014, 0.00259678, -0.00181503, ..., -0.0391503 ,
0.02240185, -0.08398875],
[-0.00283647, 0.03246598, -0.08574891, ..., 0.01009451,
-0.10250273, 0.07933066],
[ 0.06799221, -0.06107143, 0.05591911, ..., 0.08260432,
-0.08241498, -0.10361532]]], dtype=float32),
'key.weights': Array([[[ 0.02966756, -0.05070919, 0.10086108, ..., 0.09894186,
0.00321928, 0.02347625],
[-0.05256454, 0.07328279, -0.02235697, ..., -0.05919649,
-0.05481625, 0.04065797],
[ 0.09704188, -0.08744881, 0.03356297, ..., -0.08281364,
-0.01791672, 0.06750985],
...,
[ 0.04217464, -0.00188854, -0.02707726, ..., 0.10698564,
-0.08629192, 0.08909976],
[-0.05754029, -0.04504608, -0.05986738, ..., -0.07114115,
-0.03428179, -0.00438208],
[-0.01955418, -0.04972056, 0.09457364, ..., -0.08133748,
0.06241515, -0.06061669]],
[[ 0.09191433, -0.00361551, 0.02085558, ..., 0.07085335,
-0.06294455, 0.06075304],
[ 0.05274095, 0.00096863, 0.06742716, ..., 0.04126382,
0.03818371, -0.10074326],
[-0.06348637, -0.02834678, -0.06460179, ..., 0.076653 ,
-0.09819552, 0.10601056],
...,
[-0.0406662 , 0.07818392, -0.05748281, ..., 0.03785541,
-0.00464959, 0.06684505],
[-0.07903117, 0.04375349, -0.00056531, ..., 0.02116194,
0.03349148, 0.08557268],
[-0.0446655 , 0.04775485, 0.10264049, ..., -0.00121034,
0.067717 , 0.00515727]],
[[ 0.08430512, -0.00473329, 0.03658821, ..., 0.03803442,
-0.03498443, -0.09895352],
[-0.10155763, -0.03883382, 0.08058728, ..., 0.04282878,
0.08955543, 0.01659501],
[ 0.0518962 , -0.01376615, -0.06962278, ..., -0.10751797,
-0.00953312, 0.03498998],
...,
[ 0.02977371, 0.03396315, 0.03907956, ..., 0.00635863,
0.01316066, 0.051005 ],
[-0.09316578, 0.06394439, -0.01351381, ..., 0.0701853 ,
-0.10029438, 0.09466159],
[ 0.0235479 , -0.06248901, -0.05534565, ..., -0.04923825,
-0.03371259, -0.05698296]],
...,
[[-0.10045463, 0.07745797, 0.01177621, ..., -0.0035284 ,
-0.00532356, 0.09414765],
[-0.0854082 , 0.08576532, 0.05104208, ..., -0.07588878,
0.05955808, -0.03947741],
[-0.00577146, -0.08670396, -0.0045471 , ..., -0.08714546,
0.0359144 , 0.03762787],
...,
[ 0.00810299, -0.02998801, -0.05919561, ..., -0.05317522,
-0.05829883, 0.04843504],
[ 0.08423324, -0.08756288, -0.0860282 , ..., 0.0802955 ,
0.04880269, -0.07933288],
[-0.00122082, -0.02205652, -0.05690352, ..., -0.05061042,
0.0459919 , 0.06981041]],
[[ 0.03171575, -0.06668513, -0.07215052, ..., 0.07279679,
0.07845843, -0.01666921],
[-0.00231961, -0.08844905, 0.04219678, ..., -0.03844374,
-0.07553118, 0.0469028 ],
[-0.02279248, 0.01944663, -0.08697202, ..., -0.0363368 ,
-0.06989014, -0.03294029],
...,
[-0.10440172, 0.04089526, -0.02945994, ..., -0.04093436,
0.01264307, -0.06160532],
[ 0.05439864, 0.053004 , 0.06698484, ..., 0.04794566,
-0.07925336, 0.08439024],
[ 0.02403408, 0.06779779, -0.06364737, ..., -0.09004401,
0.10474594, -0.00454808]],
[[ 0.01468105, -0.02732616, 0.01376706, ..., -0.02064783,
-0.06636342, 0.02920797],
[-0.04949134, 0.05331869, 0.07406975, ..., 0.00700624,
0.04963988, 0.08469719],
[-0.08758172, 0.07195341, 0.0783165 , ..., -0.04254558,
-0.03094371, -0.08194995],
...,
[-0.06390196, 0.10137361, 0.00189773, ..., 0.04486196,
0.00557164, 0.0724177 ],
[-0.04019007, 0.02562309, -0.01242245, ..., -0.02260616,
-0.08187877, -0.07221231],
[ 0.02764633, -0.06772394, -0.0588604 , ..., 0.07926263,
0.02385919, -0.04747384]]], dtype=float32),
'value.weights': Array([[[-0.10238901, -0.00332074, 0.08355561, ..., 0.05892409,
-0.02542792, 0.08451322],
[-0.02004936, 0.0253794 , 0.03344424, ..., -0.0669517 ,
-0.09971996, -0.03615822],
[ 0.01208288, 0.09426777, -0.04889726, ..., 0.05436277,
-0.00159996, -0.01427821],
...,
[ 0.06300657, -0.00661703, 0.07117435, ..., -0.07217047,
-0.03221979, 0.03577555],
[ 0.09406403, 0.10341016, 0.07415971, ..., 0.08433615,
0.09219874, -0.02773344],
[-0.08990037, -0.00642694, -0.05194046, ..., -0.07838348,
0.08939232, -0.05132697]],
[[ 0.10696079, 0.02298048, -0.03314787, ..., -0.04029937,
0.08655001, 0.06330006],
[ 0.00692004, -0.082724 , -0.04104047, ..., 0.03525646,
-0.00409895, 0.01452263],
[-0.09063112, -0.07169501, 0.05331748, ..., 0.07244578,
-0.07503345, 0.00844285],
...,
[-0.08657265, -0.03948061, 0.03838879, ..., 0.09800081,
-0.05576013, 0.02514381],
[ 0.017865 , -0.01396458, -0.05763393, ..., 0.08012196,
0.01875073, -0.01490665],
[ 0.03818735, -0.08124364, -0.0705097 , ..., -0.09002888,
-0.01058734, -0.03532741]],
[[-0.0403618 , 0.020198 , -0.04981038, ..., -0.04468674,
0.00227806, -0.0593291 ],
[-0.0124899 , 0.06265502, -0.02790484, ..., 0.04457852,
0.0834778 , -0.01516374],
[ 0.08904755, -0.10351588, 0.08789304, ..., 0.04400235,
-0.03460258, 0.05925242],
...,
[-0.03269582, 0.04083499, 0.0537626 , ..., 0.04810795,
0.06216374, 0.05118086],
[-0.06202803, 0.05424113, -0.07361013, ..., -0.01698448,
0.09705946, 0.00986875],
[ 0.05973387, 0.08304437, -0.00670669, ..., -0.00306171,
0.01621912, -0.09330146]],
...,
[[ 0.09932957, -0.09989911, -0.04669382, ..., -0.00042418,
0.0342549 , -0.02441967],
[-0.03097494, 0.01955898, 0.03894434, ..., 0.04452504,
-0.03848797, 0.04439104],
[-0.00499776, -0.08717161, 0.00606484, ..., -0.02516294,
0.10066364, -0.08659141],
...,
[-0.08666252, -0.05121622, -0.09465104, ..., -0.03611765,
0.00587286, -0.05273307],
[-0.10209233, -0.02764767, -0.00466758, ..., -0.07925303,
-0.01669644, -0.05575708],
[-0.06685447, 0.07392795, 0.04345722, ..., 0.03890681,
-0.03032402, -0.09653031]],
[[-0.0687222 , -0.02855627, -0.01716161, ..., 0.01590494,
-0.08810354, 0.04328308],
[-0.08390799, 0.07167805, -0.02689512, ..., 0.05169308,
-0.05251584, -0.02587554],
[ 0.06224233, -0.09249328, 0.0004679 , ..., 0.08630126,
0.08799666, -0.02997211],
...,
[ 0.10601179, 0.05115861, 0.09234599, ..., 0.08881163,
-0.04911378, 0.05357352],
[ 0.05902302, 0.10311601, 0.02395825, ..., 0.05636727,
-0.08026949, 0.01823849],
[-0.0823376 , -0.08379997, 0.06620869, ..., -0.01052506,
-0.01756491, -0.01399849]],
[[ 0.06666165, 0.03141001, -0.09848791, ..., -0.00505555,
-0.00866221, -0.06319816],
[ 0.02339062, 0.09748351, 0.05846809, ..., 0.09067624,
-0.03024448, -0.07553131],
[-0.00874944, -0.04680583, 0.09931261, ..., -0.0217592 ,
0.05522589, -0.07178007],
...,
[ 0.01204465, 0.01482997, 0.02369504, ..., -0.00610985,
-0.00255902, -0.05188957],
[-0.05735252, 0.08544451, 0.08447025, ..., -0.02879305,
-0.10488513, -0.06507408],
[ 0.02556451, -0.02076597, -0.02819726, ..., 0.09015321,
0.0242661 , -0.06672818]]], dtype=float32),
'output.weights': Array([[[ 0.04857322, -0.09944757, 0.02261003, ..., 0.06439366,
0.07141559, 0.01414826],
[-0.02585517, 0.02839654, 0.05859298, ..., 0.07272132,
-0.05552242, 0.0674812 ],
[-0.07220263, -0.01790825, 0.06659748, ..., -0.03632973,
-0.02636295, 0.04294392],
...,
[ 0.03484361, 0.03578337, 0.01303479, ..., 0.01899576,
0.0719453 , -0.0487311 ],
[ 0.07701885, -0.00028169, 0.0437762 , ..., -0.04568874,
-0.04237797, -0.09665219],
[-0.00433056, 0.06238797, -0.07366703, ..., -0.03780064,
0.09148663, -0.03774949]],
[[ 0.0406908 , -0.06031727, -0.07838962, ..., 0.02046242,
-0.00349831, -0.02587182],
[-0.09387077, -0.09494787, -0.07270519, ..., 0.0016266 ,
0.09013966, -0.10052494],
[ 0.05547217, 0.07333615, 0.057437 , ..., -0.03084981,
0.02958814, -0.02453563],
...,
[ 0.00184902, 0.06296618, -0.0612434 , ..., -0.0439475 ,
0.03974828, 0.04419641],
[ 0.01854497, 0.08480572, 0.01653676, ..., 0.038491 ,
0.04192178, 0.08473973],
[ 0.04932895, 0.08014632, 0.1065437 , ..., 0.02888638,
0.08530891, 0.08501981]],
[[ 0.04183285, -0.02391262, -0.09201402, ..., 0.09152251,
0.04033429, -0.05584889],
[-0.07877362, -0.09760092, -0.10495318, ..., 0.06614685,
-0.00926398, 0.08770726],
[-0.08111565, -0.01358231, 0.09856287, ..., 0.00792026,
0.10736801, -0.05578364],
...,
[ 0.05287232, -0.09868577, 0.06846648, ..., -0.06741536,
-0.04879415, 0.017143 ],
[ 0.08785345, -0.00722993, 0.10091066, ..., -0.05583505,
0.05126856, 0.07147351],
[-0.04154757, -0.02401962, 0.07846984, ..., 0.00843328,
0.08646923, -0.1015638 ]],
...,
[[-0.01659289, 0.07015567, 0.02606057, ..., 0.01179118,
-0.10371776, 0.02660863],
[ 0.0342469 , -0.05317574, -0.08386907, ..., -0.09350891,
-0.00231548, -0.01913924],
[ 0.09911047, -0.01371071, 0.02518503, ..., 0.05958345,
0.00488619, -0.01741415],
...,
[ 0.03690621, 0.00812986, -0.05531587, ..., 0.05297308,
-0.07360285, 0.03159698],
[ 0.01597158, -0.07352082, -0.03715752, ..., 0.03587326,
-0.07546222, -0.09305257],
[ 0.05042738, -0.00394264, 0.04551102, ..., 0.01941032,
-0.1018206 , 0.0297382 ]],
[[ 0.08891174, 0.10304739, -0.0743474 , ..., -0.03819475,
-0.08200053, 0.03297851],
[-0.06789047, -0.0271074 , 0.05041561, ..., 0.0410919 ,
-0.0571834 , -0.05765961],
[ 0.01082843, 0.04575763, -0.05942155, ..., 0.10416958,
-0.0698155 , -0.0936079 ],
...,
[ 0.09733722, -0.0089729 , 0.05789496, ..., -0.01427522,
0.00744905, 0.08376124],
[ 0.0666073 , -0.07341077, -0.01078241, ..., -0.03817615,
0.00150405, -0.03473699],
[-0.09913009, -0.01618913, -0.05309457, ..., 0.07854918,
0.02927203, -0.00855048]],
[[ 0.05394956, -0.09964981, -0.08956743, ..., -0.09096286,
0.02714072, -0.04866644],
[ 0.03474536, 0.05089172, -0.05076665, ..., -0.0586158 ,
-0.04323554, 0.05740308],
[-0.080293 , 0.0389239 , -0.07931347, ..., -0.0575735 ,
0.07694697, 0.06616644],
...,
[ 0.01906744, 0.0796563 , -0.07703242, ..., -0.02229087,
0.09857954, 0.03309313],
[ 0.04214383, 0.05895525, 0.00531416, ..., 0.10542824,
-0.03491278, -0.07623658],
[-0.04732548, -0.05335865, 0.09212413, ..., 0.09336126,
0.01199412, -0.06999436]]], dtype=float32)},
'pre_ffw_norm': {'scale.weights': Array([1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1.], dtype=float32)},
'mlp': {'gating_linear.weights': Array([[ 0.00249305, -0.02006169, 0.00034139, ..., 0.00103011,
-0.0792389 , -0.04116493],
[ 0.05398832, -0.03853764, -0.06424425, ..., -0.05485614,
-0.01026699, 0.02369367],
[ 0.02376069, -0.02120294, -0.01521961, ..., 0.00244249,
-0.0456754 , -0.00853644],
...,
[ 0.02708276, -0.02588442, 0.02307968, ..., -0.00096748,
-0.03422507, 0.01766398],
[-0.0612856 , -0.05787679, -0.03695971, ..., 0.04738225,
-0.03314257, -0.01562337],
[ 0.07826702, -0.00534574, -0.01575026, ..., -0.01286178,
0.05477586, 0.05564425]], dtype=float32),
'value_linear.weights': Array([[ 0.04178567, 0.02595907, 0.05917577, ..., -0.03610552,
-0.03217094, -0.01761523],
[-0.08377769, -0.05969197, 0.04138684, ..., 0.02902774,
-0.00996111, -0.05995385],
[ 0.08256498, 0.05270175, 0.08277042, ..., -0.08154973,
0.07069694, -0.01587731],
...,
[-0.00863498, 0.06980709, -0.07202983, ..., -0.06248517,
-0.01357927, 0.08403928],
[-0.0618956 , -0.03795958, -0.08459446, ..., 0.0663895 ,
-0.03810077, 0.08422004],
[ 0.0417348 , -0.02694994, 0.0702616 , ..., 0.01976508,
-0.08209807, 0.07139529]], dtype=float32),
'out_linear.weights': Array([[-0.01899929, -0.05121481, -0.08528872, ..., 0.04927557,
-0.02219733, 0.04735207],
[-0.02818714, -0.08725584, -0.03081095, ..., 0.01436851,
0.05834622, 0.02826863],
[-0.04688002, 0.08341584, 0.05823929, ..., 0.08149333,
0.04160098, 0.0016808 ],
...,
[ 0.01824762, 0.01780744, -0.00284873, ..., 0.03254097,
0.06864896, 0.07666741],
[ 0.04618473, -0.00246443, -0.02444646, ..., -0.02029029,
-0.03989785, -0.08682929],
[-0.07860562, 0.0043446 , -0.05057375, ..., 0.07874653,
0.03296353, 0.03570217]], dtype=float32)}},
'block_5': {'pre_attention_norm': {'scale.weights': Array([1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1.], dtype=float32)},
'attention': {'query.weights': Array([[[ 6.98920526e-03, -2.24690605e-02, 1.10609224e-02, ...,
-9.56980810e-02, 8.10946971e-02, 8.16590190e-02],
[ 4.43694880e-03, -5.54080047e-02, -7.54439756e-02, ...,
4.68662009e-02, -2.33808882e-03, 5.43762185e-02],
[-1.20694311e-02, 4.82318625e-02, 5.34766242e-02, ...,
1.07786462e-01, -7.77173564e-02, 3.55384313e-02],
...,
[-1.52390273e-02, -4.51295525e-02, -1.05882928e-01, ...,
5.19417562e-02, 7.93971494e-02, -6.67541474e-02],
[-9.50563774e-02, -9.99856964e-02, -7.71190152e-02, ...,
-1.68296450e-03, 2.78754942e-02, -3.14534754e-02],
[-5.81725985e-02, -9.97146219e-02, 6.89329673e-03, ...,
2.98466235e-02, -2.37189925e-05, 5.21386564e-02]],
[[-2.01397226e-03, 7.93305337e-02, 7.66330957e-02, ...,
3.08707990e-02, -7.98701346e-02, -8.20086151e-03],
[ 3.53536606e-02, -8.27091113e-02, -6.92761093e-02, ...,
5.60720339e-02, -1.05536073e-01, 1.38887996e-02],
[ 1.54691190e-02, 3.72621231e-02, 1.06880024e-01, ...,
4.01986577e-03, 3.64468023e-02, 2.88896561e-02],
...,
[-9.32704061e-02, 9.54070538e-02, -2.19185185e-03, ...,
-1.63316745e-02, 2.03209044e-03, -3.39890569e-02],
[ 2.79233456e-02, -7.62031376e-02, -9.98197943e-02, ...,
-8.89362618e-02, -9.55900177e-02, -6.20138086e-02],
[ 8.63494426e-02, 7.01185018e-02, -2.88739372e-02, ...,
4.30578962e-02, -4.74424511e-02, 1.06492341e-02]],
[[ 3.99856530e-02, -1.62992068e-02, -1.20764002e-02, ...,
3.35083567e-02, -2.25641429e-02, -6.27742857e-02],
[-9.24955532e-02, -6.67447299e-02, 4.37915847e-02, ...,
-8.24244022e-02, 6.78407103e-02, 3.46470475e-02],
[ 9.52842012e-02, -7.45838210e-02, -2.32784487e-02, ...,
-5.55109605e-03, -6.19025454e-02, 1.07742064e-01],
...,
[-4.76910993e-02, 8.56455639e-02, -9.05451248e-04, ...,
-7.47156003e-03, -6.79718703e-02, 1.43426610e-02],
[ 5.77346608e-02, 3.67607214e-02, -9.91240442e-02, ...,
9.64036398e-03, 9.01653394e-02, 8.21006224e-02],
[ 1.02119580e-01, 3.93297821e-02, -1.20621016e-02, ...,
3.47285271e-02, 1.74292270e-02, -1.01934835e-01]],
...,
[[ 3.67640778e-02, -4.94731739e-02, 1.36659341e-02, ...,
-1.87603813e-02, -7.31427688e-03, -3.04710343e-02],
[-8.40078443e-02, -6.03974611e-02, 8.25465322e-02, ...,
1.79005358e-02, -8.39532092e-02, 8.73618796e-02],
[ 1.00339442e-01, -4.18913551e-02, 6.66472735e-03, ...,
-1.83463693e-02, 6.99599832e-02, 2.90914085e-02],
...,
[-3.99481505e-02, 4.57173400e-02, 6.89297691e-02, ...,
7.73523822e-02, -7.52842352e-02, -7.71844387e-02],
[-9.92225632e-02, -3.75869893e-03, -2.73532383e-02, ...,
-6.85260817e-02, -7.26163313e-02, -2.67681363e-03],
[-2.79207900e-02, -2.22630240e-02, -8.54721293e-02, ...,
-7.59318545e-02, -1.04305081e-01, 5.72232977e-02]],
[[-1.05494566e-01, 2.12693289e-02, 6.58555403e-02, ...,
-3.16767022e-02, 7.30000883e-02, 1.01939820e-01],
[ 1.90407503e-03, 6.89884573e-02, -4.79736365e-02, ...,
-8.82072672e-02, 4.09926921e-02, -5.10257715e-03],
[-3.89541499e-02, 3.56504209e-02, -7.75896534e-02, ...,
-5.99349791e-04, 1.70077570e-02, -4.96134721e-03],
...,
[-3.22050229e-02, -1.61451232e-02, -7.46594667e-02, ...,
-1.05926648e-01, 4.86611016e-03, 2.44637281e-02],
[-9.50709358e-02, 2.85691004e-02, -9.09569934e-02, ...,
3.96662615e-02, 4.78826575e-02, -1.07134119e-01],
[-1.06374107e-01, -2.80806813e-02, 8.68832394e-02, ...,
9.60046723e-02, 2.67336015e-02, 7.45149553e-02]],
[[ 9.28851217e-02, -9.34095979e-02, -1.01641696e-02, ...,
-2.20973766e-03, -2.96217185e-02, 1.58239491e-02],
[-5.99094518e-02, 2.51244009e-02, 9.80514511e-02, ...,
-3.54542173e-02, 5.61420806e-02, 1.01302527e-01],
[ 8.48812386e-02, 5.92576303e-02, 7.95461684e-02, ...,
-6.59162924e-02, 7.14983866e-02, 3.33389416e-02],
...,
[-8.32986534e-02, 8.51696357e-02, -8.08895826e-02, ...,
-9.20301601e-02, 9.69092473e-02, -6.75970912e-02],
[ 8.65020603e-03, -5.54732271e-02, -1.05463468e-01, ...,
6.70254081e-02, 2.49014832e-02, -3.31384540e-02],
[ 6.68995604e-02, -2.16521882e-02, 8.96923020e-02, ...,
3.04944944e-02, 8.55549499e-02, -2.03708205e-02]]], dtype=float32),
'key.weights': Array([[[ 0.08138222, 0.0048946 , -0.05262971, ..., -0.01437879,
-0.0583536 , 0.05654125],
[ 0.0839911 , 0.09225225, -0.03013765, ..., -0.09344535,
-0.08567817, -0.06891549],
[ 0.10476483, 0.07473698, -0.08824462, ..., -0.0916537 ,
0.06347971, 0.08080284],
...,
[-0.10123686, -0.05591227, 0.03372443, ..., 0.10502677,
0.08280639, 0.02919798],
[ 0.10569782, 0.10625041, 0.06243144, ..., 0.01030646,
-0.0040098 , 0.0969387 ],
[ 0.10195192, 0.05867831, 0.00072607, ..., -0.00647103,
-0.07129728, 0.03072843]],
[[ 0.03428326, 0.1051544 , 0.05171525, ..., 0.0708119 ,
0.01893978, -0.10009634],
[ 0.05274712, 0.0794817 , 0.03159961, ..., 0.08400403,
-0.09397703, 0.08294684],
[-0.0558103 , -0.04259317, -0.09800197, ..., 0.06390578,
0.02000577, 0.06214195],
...,
[ 0.03720522, -0.10429182, 0.01689925, ..., -0.06555679,
-0.06698615, 0.08336844],
[-0.09353968, -0.06284454, -0.05067706, ..., 0.04001536,
-0.05530304, 0.10600416],
[-0.0278431 , 0.10695565, -0.06706902, ..., 0.01204052,
-0.09136881, -0.05219105]],
[[ 0.05480522, 0.02257132, 0.06794787, ..., 0.10718402,
0.05931364, 0.02683206],
[ 0.09495048, 0.07070342, -0.05458411, ..., -0.08104584,
-0.03051088, -0.00686646],
[ 0.06805026, 0.05702428, -0.0822418 , ..., 0.02644015,
0.03982685, 0.05763026],
...,
[-0.05523364, -0.09048922, 0.08686427, ..., 0.0115034 ,
-0.07131842, -0.07494476],
[ 0.07242043, 0.02698716, 0.05861177, ..., 0.03513867,
-0.09465706, -0.06609222],
[ 0.02750789, 0.05387218, 0.04637582, ..., 0.1022255 ,
-0.07451274, -0.02193186]],
...,
[[-0.03211257, 0.08465032, 0.07006216, ..., -0.07708136,
-0.06954202, -0.0467586 ],
[-0.03271432, -0.09799624, 0.0291063 , ..., 0.03972067,
0.06236183, -0.01568109],
[ 0.05218297, -0.00028881, 0.03346758, ..., -0.03996578,
-0.10817701, 0.06410585],
...,
[ 0.01199409, 0.06955854, 0.1022563 , ..., -0.09595817,
0.00670729, 0.04001691],
[ 0.05859131, 0.06222545, 0.09314389, ..., -0.01428867,
-0.07681322, 0.04751766],
[-0.077172 , 0.02285904, 0.06344934, ..., -0.08835234,
0.1067013 , -0.103095 ]],
[[-0.02724858, 0.02760787, 0.00017249, ..., 0.04190361,
-0.06104778, 0.09686576],
[-0.0116531 , 0.09779552, 0.07566666, ..., 0.09618658,
-0.09388832, -0.0552565 ],
[ 0.03933081, -0.10763712, -0.05874351, ..., -0.07874966,
0.08146532, 0.05779671],
...,
[ 0.05917747, 0.04291576, -0.03237082, ..., 0.00228415,
-0.061427 , 0.05410274],
[ 0.09730065, -0.01364908, 0.10537365, ..., -0.0055567 ,
0.01235401, 0.05756571],
[-0.02838895, 0.02678065, 0.04273174, ..., -0.03015427,
-0.00863281, -0.00951635]],
[[-0.00369547, -0.01874048, 0.06518433, ..., -0.03175743,
0.06369739, -0.08412407],
[ 0.01999978, 0.10400799, 0.07916834, ..., -0.04796006,
-0.02192626, -0.04325846],
[-0.0034297 , -0.0058022 , -0.08589759, ..., -0.04343118,
-0.00117717, 0.09952466],
...,
[ 0.05750609, 0.09877959, -0.0296506 , ..., -0.02238593,
-0.0879325 , 0.09682263],
[-0.09093036, 0.10405359, 0.04414755, ..., 0.0336691 ,
-0.08122062, 0.03576917],
[-0.00983037, -0.03438836, 0.09925131, ..., 0.01699 ,
-0.05463599, -0.0083202 ]]], dtype=float32),
'value.weights': Array([[[ 0.05329513, 0.03993057, 0.05790854, ..., 0.03925142,
-0.05571253, -0.0768395 ],
[ 0.00119181, 0.0962246 , -0.08794295, ..., 0.03143693,
-0.1066765 , 0.09242783],
[ 0.06311838, 0.01134793, -0.08287383, ..., -0.08428824,
0.05081961, 0.0436015 ],
...,
[ 0.0362555 , 0.08188684, 0.06568556, ..., 0.0416449 ,
0.05463013, 0.05918949],
[ 0.07046482, -0.03108156, 0.08730799, ..., -0.02892705,
-0.10166719, -0.00668001],
[-0.08829724, -0.04818858, -0.05582664, ..., -0.03932033,
0.04094211, 0.10014128]],
[[-0.02118248, -0.08649984, 0.00870175, ..., -0.06845779,
0.05233476, -0.0237207 ],
[-0.08688515, 0.0509133 , 0.07972663, ..., -0.04925557,
0.07195233, 0.08445967],
[-0.08882688, -0.01434749, -0.07409421, ..., -0.09540571,
-0.03725203, -0.02406368],
...,
[ 0.08430731, 0.00935607, -0.07264666, ..., 0.00814418,
0.04828028, -0.01868904],
[-0.03979877, 0.04075457, -0.09409077, ..., -0.10048465,
-0.04200412, -0.05684423],
[-0.07203494, -0.06472645, 0.01369182, ..., -0.08796079,
-0.03405013, -0.04611488]],
[[-0.0153729 , 0.01489899, -0.09033279, ..., 0.04818959,
0.00979744, -0.03922556],
[-0.06323089, -0.0383733 , 0.03286977, ..., -0.04224688,
0.0529745 , -0.06609552],
[-0.04154336, -0.07306077, -0.01832931, ..., 0.05628112,
-0.10801309, -0.0549599 ],
...,
[ 0.09821627, 0.03968025, 0.03790576, ..., -0.06596332,
0.06377867, 0.06810817],
[ 0.10551924, 0.06759154, 0.01727489, ..., 0.03330763,
-0.05204845, 0.01891485],
[ 0.09659696, -0.05622604, -0.0476521 , ..., 0.07142285,
0.06430908, 0.08562148]],
...,
[[ 0.02857798, 0.09359388, 0.03049207, ..., 0.06426084,
0.08746408, 0.02097358],
[ 0.04740668, -0.02784863, -0.02011285, ..., 0.03149836,
-0.05397694, 0.08933479],
[ 0.00556139, 0.10069456, -0.06182705, ..., -0.0278669 ,
0.03446653, 0.0300652 ],
...,
[-0.0530183 , -0.02344492, 0.07157189, ..., -0.00682395,
-0.020838 , -0.06035815],
[ 0.0746223 , 0.07486958, -0.08512685, ..., 0.07244544,
-0.06793362, -0.10008334],
[-0.02462359, 0.10284723, -0.04858463, ..., 0.01449659,
0.05176749, 0.00620916]],
[[-0.0503603 , 0.09373643, 0.07689948, ..., 0.07632826,
0.04400353, 0.05248642],
[-0.05665729, 0.09437354, 0.05191192, ..., -0.09624653,
0.03480854, -0.09468787],
[-0.00615896, -0.07936902, 0.08727327, ..., -0.03891461,
0.07283174, 0.05085752],
...,
[ 0.1007175 , 0.0799766 , -0.07096013, ..., -0.03092644,
0.06499866, -0.01515522],
[ 0.00590304, 0.01213099, -0.10798003, ..., -0.03060351,
0.02277687, 0.08626796],
[-0.02928199, 0.0199421 , -0.01813548, ..., 0.06888852,
-0.05494733, -0.0816386 ]],
[[ 0.0752297 , 0.06182297, 0.06920812, ..., -0.03413705,
0.0714048 , 0.05388927],
[-0.02936135, -0.0694488 , 0.03404801, ..., -0.06182215,
0.02636729, -0.06228004],
[-0.03831353, -0.04039977, 0.04189789, ..., -0.00838945,
-0.0265621 , 0.07277834],
...,
[-0.0943221 , -0.00934234, -0.07197142, ..., -0.1054908 ,
-0.0580523 , 0.0157609 ],
[-0.10422497, 0.0254417 , -0.09912136, ..., -0.09377445,
-0.07740651, -0.07448061],
[ 0.00237709, 0.08625514, 0.09249839, ..., -0.07215784,
0.0345147 , 0.09550784]]], dtype=float32),
'output.weights': Array([[[-0.09996033, -0.06129777, 0.04012768, ..., 0.03605958,
0.04687436, 0.01382043],
[ 0.04764825, 0.05969441, -0.08757135, ..., -0.05234681,
-0.02303873, 0.08158737],
[-0.02614747, 0.02333234, 0.01759273, ..., -0.04405724,
-0.04428971, -0.08673476],
...,
[ 0.10107907, -0.0068272 , 0.09803919, ..., -0.05665492,
0.10579897, -0.04290139],
[-0.09932864, 0.05931643, -0.02322136, ..., 0.03057288,
0.08074709, 0.05182429],
[ 0.07857426, 0.09258948, 0.02926921, ..., -0.07467955,
0.04500115, -0.05705791]],
[[-0.10462569, 0.00532206, -0.03973504, ..., -0.08740245,
0.06045295, -0.04050595],
[-0.07861047, 0.01845338, 0.08256692, ..., 0.08055166,
-0.01907046, 0.09861515],
[-0.07058533, -0.05510206, -0.06156289, ..., 0.03229732,
0.10247225, 0.09349611],
...,
[ 0.10092708, -0.09070179, 0.07034248, ..., -0.09211208,
0.08418913, -0.09351552],
[-0.08085635, 0.07715884, -0.05597623, ..., -0.04075834,
0.03722963, 0.03466395],
[-0.0309075 , 0.0437554 , 0.05286762, ..., 0.08027816,
-0.05465104, 0.10751316]],
[[ 0.04376041, -0.00891135, 0.0347387 , ..., 0.05429004,
0.07069716, 0.10279693],
[-0.07017826, 0.00395927, -0.00280509, ..., -0.05292311,
0.05216702, 0.07722865],
[-0.08653124, 0.10256263, 0.08578595, ..., 0.08268278,
-0.01584011, 0.03288921],
...,
[-0.07346699, 0.07958651, 0.04763848, ..., 0.00665962,
-0.05898418, -0.0673512 ],
[ 0.08256192, -0.09164175, 0.02006893, ..., 0.06372643,
-0.08332619, 0.03310059],
[ 0.05748896, 0.00455513, -0.08260192, ..., -0.06721756,
-0.05810991, 0.07589879]],
...,
[[-0.08049689, -0.08370095, -0.00049846, ..., -0.0421313 ,
-0.0458294 , 0.06312731],
[-0.0412385 , -0.05336407, -0.01263796, ..., -0.06386755,
-0.00194896, 0.02322226],
[-0.09231793, 0.03000419, 0.01933346, ..., 0.04440861,
0.09835871, 0.02431481],
...,
[ 0.01906228, -0.05782254, 0.00243209, ..., -0.08478338,
0.08936602, 0.04130106],
[ 0.08895843, 0.0272196 , 0.05174011, ..., -0.03806031,
0.02710869, 0.08511484],
[ 0.08745436, -0.05992551, 0.0398494 , ..., -0.02397877,
-0.03223563, 0.05560055]],
[[ 0.09305815, 0.09591004, 0.07349249, ..., 0.02235008,
0.00254624, -0.10733721],
[ 0.01593651, 0.09838027, 0.04717522, ..., 0.05084648,
0.02034537, -0.06696726],
[-0.01992086, -0.00456984, -0.09668545, ..., -0.08560494,
0.06180013, 0.03359487],
...,
[-0.06300738, 0.03710497, -0.08041732, ..., 0.01735691,
-0.1016492 , 0.05496018],
[-0.09562483, 0.02657725, -0.04225287, ..., -0.00620387,
-0.03894377, -0.10278942],
[ 0.07903398, 0.04228297, 0.03545708, ..., -0.03649669,
-0.09087438, -0.06631467]],
[[ 0.06222873, 0.09505181, 0.01712093, ..., -0.04263428,
0.03201318, 0.09761855],
[-0.02017387, -0.08131567, -0.08237975, ..., 0.06957054,
-0.04661745, 0.0809617 ],
[ 0.05588148, 0.08114517, 0.06951167, ..., -0.092124 ,
-0.04232591, -0.09203604],
...,
[ 0.0341267 , -0.06303488, -0.00492436, ..., -0.03925883,
-0.03210266, -0.09667758],
[ 0.07300368, -0.0753007 , 0.06670643, ..., 0.08020236,
-0.08507035, 0.0595204 ],
[-0.0139439 , 0.08868506, -0.0788703 , ..., 0.00124095,
-0.01374174, 0.07660837]]], dtype=float32)},
'pre_ffw_norm': {'scale.weights': Array([1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1.], dtype=float32)},
'mlp': {'gating_linear.weights': Array([[ 0.05430362, 0.02904912, -0.0452337 , ..., -0.00366334,
0.01592597, 0.0294484 ],
[-0.05599314, -0.08683039, 0.0042487 , ..., -0.05569424,
-0.05414384, -0.0270985 ],
[-0.08680175, 0.01056449, 0.01826368, ..., 0.06056639,
-0.02625491, 0.00839938],
...,
[-0.04258397, 0.04785595, -0.00653179, ..., -0.02408608,
-0.07660805, -0.07277802],
[ 0.00837268, -0.0316723 , -0.02185693, ..., -0.08548034,
0.03791802, 0.0030669 ],
[-0.03109428, -0.0221736 , -0.07866952, ..., 0.0090856 ,
0.00138322, -0.06431657]], dtype=float32),
'value_linear.weights': Array([[-0.03106115, -0.06296851, -0.05130656, ..., -0.05932555,
0.06954744, 0.08100747],
[-0.01958811, -0.06774292, -0.02820741, ..., -0.07921045,
-0.00224002, 0.00254063],
[ 0.05502562, 0.07381281, -0.04500074, ..., 0.04195171,
0.03400427, 0.01359767],
...,
[ 0.00824472, 0.00391237, -0.0717361 , ..., -0.04815739,
-0.02836314, -0.03091943],
[ 0.06169369, -0.00847531, 0.01411363, ..., -0.08334126,
0.06632289, 0.04509049],
[-0.06876524, 0.0706201 , -0.01847031, ..., 0.06558233,
-0.07433891, 0.05974113]], dtype=float32),
'out_linear.weights': Array([[ 0.04244289, 0.07936545, 0.0704156 , ..., -0.02124277,
-0.06062529, -0.02735204],
[-0.02798829, 0.05525877, 0.0489689 , ..., 0.03651451,
0.06354463, -0.05105264],
[ 0.06634508, -0.00225844, -0.07259274, ..., -0.02578757,
0.0878173 , -0.08290188],
...,
[-0.08019618, -0.00353905, 0.06506445, ..., -0.07842072,
-0.01674418, 0.06154245],
[-0.03039881, -0.06857836, 0.0590468 , ..., 0.06750734,
-0.07953709, 0.03763819],
[-0.00872564, -0.02786275, -0.08006101, ..., -0.03320084,
0.0225834 , 0.03734953]], dtype=float32)}},
'block_6': {'pre_attention_norm': {'scale.weights': Array([1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1.], dtype=float32)},
'attention': {'query.weights': Array([[[-0.01223384, -0.02353789, 0.10166123, ..., 0.08223628,
-0.00783803, -0.01136271],
[ 0.05898821, -0.09797247, -0.0005754 , ..., -0.07422047,
0.03455501, -0.04744276],
[-0.01935916, 0.04095498, 0.0140426 , ..., -0.10406472,
-0.01180633, -0.01487506],
...,
[-0.09738443, -0.06179324, 0.06713659, ..., 0.07606675,
-0.02227926, -0.08849432],
[-0.00700918, -0.06411525, 0.05905405, ..., -0.00484407,
0.04862788, 0.02693992],
[-0.01393278, -0.08004455, -0.0380521 , ..., -0.10097937,
-0.02414077, -0.02194363]],
[[-0.02443407, -0.01972687, -0.10808671, ..., 0.07713716,
0.10273927, -0.03580659],
[-0.10455531, 0.00762237, 0.02335892, ..., -0.06662392,
-0.08522562, 0.05325789],
[-0.04314366, 0.02062605, 0.102902 , ..., -0.07168814,
0.00989134, 0.0437674 ],
...,
[ 0.00165233, 0.00914061, 0.01848587, ..., -0.04864688,
0.04303787, -0.08747657],
[-0.02348046, 0.05026468, 0.10556864, ..., -0.07450164,
-0.00807677, 0.01766745],
[ 0.00411157, -0.08115153, 0.04265795, ..., -0.09914521,
0.07116209, -0.00015839]],
[[ 0.06087909, -0.04555352, 0.01817381, ..., 0.05491703,
-0.05197162, 0.08188078],
[-0.0617658 , 0.05753794, -0.07124561, ..., -0.09891166,
0.01667647, -0.02240461],
[ 0.04254723, 0.05911762, 0.08449575, ..., 0.02162837,
0.06323161, -0.06888486],
...,
[-0.09965621, -0.0202315 , 0.07366523, ..., -0.10088024,
-0.08828127, -0.10120437],
[-0.01840405, -0.10726596, -0.07384502, ..., -0.07897323,
0.10048836, 0.08829208],
[ 0.01798137, 0.08385975, -0.09844206, ..., 0.0410534 ,
-0.07303014, -0.06257901]],
...,
[[ 0.10727525, -0.09957409, 0.08653582, ..., -0.07584312,
-0.05908597, 0.01875873],
[ 0.0573699 , -0.10198762, -0.06945466, ..., -0.02384499,
0.0980783 , -0.01957493],
[ 0.071886 , -0.10602173, 0.10653297, ..., 0.02875235,
0.0376663 , -0.03971845],
...,
[-0.04722108, -0.0979031 , -0.0423988 , ..., 0.00802301,
-0.07137561, 0.07143087],
[-0.05770142, -0.05728478, -0.09720892, ..., -0.09056856,
0.0964902 , -0.08077171],
[ 0.01444889, 0.04877128, 0.06666539, ..., -0.09525321,
-0.05710656, 0.0265868 ]],
[[-0.05421522, 0.09924243, 0.04568551, ..., -0.09205042,
-0.09138644, 0.0079868 ],
[ 0.00272998, 0.04709265, -0.02460927, ..., -0.08743352,
-0.08605917, 0.0282694 ],
[-0.09684797, -0.00193487, -0.00306357, ..., 0.07836518,
0.06977769, -0.03514357],
...,
[ 0.10393903, -0.05941494, -0.0336302 , ..., 0.04635293,
0.08331697, -0.00417039],
[-0.00086152, -0.00300932, -0.04166906, ..., -0.02969889,
-0.04187672, -0.10444314],
[-0.01093404, -0.05094865, -0.07608671, ..., -0.03764243,
-0.06203833, -0.09374613]],
[[ 0.10494828, -0.02097941, -0.08338325, ..., -0.01760656,
-0.04703135, 0.02940618],
[-0.09191327, -0.03962034, 0.02117123, ..., 0.10430583,
-0.04971798, 0.01812923],
[-0.09103863, -0.05681099, 0.01756991, ..., 0.08069085,
-0.02845858, -0.03119548],
...,
[-0.01864406, 0.00212738, 0.05112702, ..., 0.04700686,
-0.05218731, 0.02747524],
[-0.00869284, 0.07847177, 0.05876828, ..., -0.08050438,
0.01525942, -0.09208162],
[-0.05695911, -0.08270462, 0.10274518, ..., -0.06959219,
0.02919937, -0.0848289 ]]], dtype=float32),
'key.weights': Array([[[ 0.02153081, -0.01446776, -0.03603449, ..., 0.05540271,
0.07394237, 0.08697651],
[-0.04470184, 0.00680764, -0.07918662, ..., 0.05670925,
-0.0363655 , 0.07245605],
[ 0.06002412, -0.04729838, -0.08519537, ..., 0.07007896,
0.0730538 , -0.07271123],
...,
[-0.04234878, 0.10652148, 0.05896955, ..., -0.07465626,
0.05030452, -0.01404515],
[ 0.02026722, -0.00743545, 0.06425034, ..., 0.02058171,
0.02194342, -0.05503663],
[-0.05411242, 0.04571721, 0.07319255, ..., 0.09883152,
0.0929616 , 0.05118414]],
[[ 0.00591243, -0.09952611, -0.09606706, ..., 0.06742032,
-0.03508013, 0.07212187],
[-0.01253891, -0.08355772, -0.04517459, ..., -0.03208284,
-0.0015017 , -0.05109838],
[-0.05433995, 0.00057478, 0.01364113, ..., 0.05622906,
-0.0250161 , -0.06714671],
...,
[ 0.09167574, -0.07037851, 0.02203644, ..., -0.01004547,
0.03984664, 0.00138866],
[-0.08993478, -0.00480755, 0.09134406, ..., -0.0363869 ,
0.03361787, -0.02305295],
[-0.04507909, 0.04410515, 0.1033589 , ..., -0.07644404,
-0.01724719, -0.06354034]],
[[ 0.00229646, -0.10048445, 0.02291624, ..., -0.09019151,
0.06193994, 0.02220947],
[ 0.0012179 , 0.03424757, -0.02054744, ..., -0.09883469,
0.0666339 , 0.10767604],
[ 0.01064299, -0.0062113 , -0.05228022, ..., -0.10040613,
-0.10266445, -0.05446978],
...,
[ 0.07676981, -0.04157746, -0.06680278, ..., 0.08249375,
0.07931649, -0.09212072],
[-0.03390936, -0.07557046, 0.09868567, ..., 0.01733639,
0.09112871, -0.01047435],
[-0.09030425, 0.08428889, -0.08743373, ..., -0.09811474,
0.00194671, -0.06333967]],
...,
[[-0.06075283, 0.06538367, -0.01925363, ..., 0.10786479,
-0.01977351, -0.10669719],
[ 0.02375128, -0.02488512, -0.08683278, ..., -0.08698779,
0.09944974, -0.10709985],
[ 0.02069163, 0.08849471, 0.0719284 , ..., 0.04595541,
0.05350445, -0.07607938],
...,
[ 0.03264343, -0.03454228, -0.0219497 , ..., -0.03575461,
0.05124931, -0.07808078],
[-0.078106 , 0.07336847, -0.05191559, ..., 0.03100204,
-0.02083232, 0.04390587],
[-0.00052806, 0.10168596, 0.03723433, ..., 0.01502522,
-0.04521617, -0.01813313]],
[[ 0.04402679, 0.07690348, 0.07642074, ..., 0.00516911,
-0.08151343, -0.0995348 ],
[-0.08404589, -0.01756261, -0.04877892, ..., 0.07006482,
0.03681874, 0.03026892],
[ 0.01057382, -0.01614079, 0.02799197, ..., 0.05695857,
-0.08804297, 0.02647969],
...,
[ 0.00562796, 0.08777609, 0.01673361, ..., 0.04533685,
0.0512729 , 0.03170241],
[ 0.0197858 , 0.01421506, 0.01003584, ..., 0.01115466,
-0.05406258, -0.09703264],
[ 0.09977426, 0.03877841, -0.09875515, ..., -0.08562595,
-0.02619072, 0.08024966]],
[[ 0.01684601, -0.06100791, -0.10699914, ..., 0.08502804,
-0.07493323, -0.0436494 ],
[-0.01323393, -0.01736194, -0.03105203, ..., 0.06721919,
0.05824964, 0.08962974],
[ 0.02446282, 0.10777964, 0.01439776, ..., 0.09731087,
0.00689433, -0.07815485],
...,
[-0.05378208, 0.06627136, 0.02857852, ..., -0.09627999,
0.01131017, 0.07949125],
[-0.04562943, -0.07723317, 0.07479244, ..., -0.03778113,
0.00287067, 0.03878282],
[ 0.04072081, -0.07474902, 0.10816563, ..., -0.03069978,
0.01622604, 0.05151905]]], dtype=float32),
'value.weights': Array([[[ 0.01674669, -0.05630788, -0.05810637, ..., 0.02010994,
-0.04390607, -0.03165905],
[-0.05171954, 0.00746802, 0.10443124, ..., 0.03383833,
0.07803757, -0.05892681],
[-0.05555768, -0.06673092, -0.02143495, ..., 0.02156841,
-0.0887685 , -0.09595365],
...,
[ 0.10780331, -0.0138602 , -0.06333632, ..., -0.01019034,
-0.02195646, -0.02572522],
[ 0.01100535, -0.06472552, -0.03740139, ..., -0.03554372,
-0.09377509, -0.06047153],
[ 0.02910509, 0.04776078, -0.09370834, ..., -0.06196805,
0.09323242, -0.04508635]],
[[ 0.08511773, 0.01868104, -0.05210028, ..., 0.05243338,
0.06643897, -0.03722909],
[ 0.06421638, 0.0475914 , -0.01044756, ..., -0.0130265 ,
0.09522922, -0.06480664],
[-0.08043604, 0.08191325, -0.09429779, ..., -0.03489861,
-0.08391147, 0.06434741],
...,
[-0.02873593, -0.01331967, -0.01387378, ..., -0.05397844,
-0.02633877, 0.03893647],
[ 0.0607256 , 0.01570781, -0.05293367, ..., 0.04778019,
0.04710636, 0.09969679],
[-0.03989986, 0.03908343, 0.02155904, ..., 0.0677664 ,
-0.06751385, -0.05619587]],
[[-0.04344847, -0.02507787, -0.08110236, ..., -0.03648027,
-0.00754109, 0.08465422],
[-0.05809471, 0.07178464, 0.05460221, ..., 0.05136804,
0.05912102, 0.08880518],
[-0.0340285 , 0.08675791, -0.09699194, ..., 0.04587524,
-0.08757403, -0.03454105],
...,
[-0.09457044, -0.06062903, -0.07097851, ..., -0.08817095,
-0.06541531, 0.00657323],
[-0.09197526, 0.00987309, -0.00032884, ..., -0.06760997,
-0.00064488, 0.01676752],
[ 0.00968566, 0.02711114, -0.08651837, ..., -0.04330032,
-0.05755923, -0.03914829]],
...,
[[-0.09705817, -0.08199942, 0.02158511, ..., 0.00423736,
0.09374966, -0.07641444],
[-0.0191754 , -0.06124739, -0.0911682 , ..., 0.09392244,
0.09814599, 0.04103518],
[ 0.0008735 , 0.08032369, 0.02099149, ..., 0.10316443,
-0.00556281, 0.00364248],
...,
[-0.03975858, -0.00294023, -0.03033117, ..., -0.05822017,
-0.08294813, -0.05194707],
[-0.00760907, 0.08565047, 0.0565889 , ..., -0.00712091,
-0.04734546, 0.00159898],
[-0.07394294, 0.04985642, 0.09999274, ..., 0.0641209 ,
-0.098013 , 0.10635248]],
[[ 0.08573448, -0.00749737, 0.03940625, ..., -0.06322898,
-0.07906756, 0.04353284],
[-0.10578026, -0.05110844, 0.01841389, ..., -0.09031465,
0.01314615, -0.03741732],
[-0.01665662, -0.05902179, 0.03643996, ..., 0.08543269,
-0.04047297, -0.00734308],
...,
[ 0.01205366, -0.00485855, -0.00708535, ..., -0.08434706,
0.04755054, 0.02171176],
[ 0.04206074, 0.09046003, -0.08542033, ..., -0.0919989 ,
0.07218812, -0.03495464],
[ 0.04492627, 0.05256544, 0.02146262, ..., 0.03682553,
-0.10770655, 0.05127569]],
[[ 0.00370375, 0.0781602 , 0.01044678, ..., -0.01500331,
-0.02207343, -0.07985834],
[ 0.02569203, 0.02372687, 0.05719016, ..., -0.05871597,
0.03498046, 0.07693068],
[-0.00469786, -0.02293495, 0.03416461, ..., 0.06560913,
-0.08019041, -0.08597794],
...,
[ 0.0828101 , -0.06514534, -0.10279483, ..., -0.02669933,
-0.09984811, -0.09321667],
[-0.01907681, 0.03470267, 0.10641819, ..., -0.06644183,
0.03031951, -0.0236507 ],
[ 0.02739206, -0.01719359, -0.05005355, ..., -0.06615473,
0.01581234, -0.08844087]]], dtype=float32),
'output.weights': Array([[[ 0.00588543, 0.04634384, 0.02380502, ..., 0.02838846,
0.09833858, 0.09731949],
[ 0.09672242, -0.03393592, -0.03992255, ..., -0.0207774 ,
-0.02347011, -0.06679465],
[-0.02431863, 0.03039663, -0.08568443, ..., 0.0911659 ,
0.09368662, 0.06827296],
...,
[-0.05849914, 0.09885387, 0.05442301, ..., -0.06044041,
-0.06029412, 0.00508918],
[-0.03922966, -0.08238641, -0.02182122, ..., 0.10338072,
0.04180164, 0.00774979],
[ 0.0663205 , 0.00199973, -0.05379971, ..., 0.06089801,
0.05107076, 0.02249988]],
[[-0.03933747, 0.04876746, 0.09295218, ..., -0.08682413,
-0.03604724, -0.0560677 ],
[ 0.08634949, 0.10519774, -0.04636116, ..., -0.00047923,
-0.08312152, 0.08629455],
[ 0.06408232, 0.10485268, 0.05584019, ..., 0.05099638,
-0.09717473, -0.07431176],
...,
[-0.0857472 , 0.0906397 , 0.10352556, ..., -0.09282767,
-0.09196419, 0.02346384],
[ 0.03600295, -0.06547121, 0.05090403, ..., -0.00116948,
-0.08076201, 0.05223098],
[-0.03722829, 0.01178787, -0.09479395, ..., 0.03856282,
-0.05689799, -0.0240405 ]],
[[-0.09761842, -0.05858617, 0.09230743, ..., -0.001618 ,
0.02519042, -0.08068094],
[ 0.02428254, -0.07336432, -0.07421675, ..., -0.08031914,
0.03730613, 0.00505935],
[-0.09655331, 0.02517997, 0.04929713, ..., 0.00460936,
0.09505042, -0.03409764],
...,
[ 0.09306135, -0.08198174, 0.10390666, ..., 0.07572027,
0.06723658, 0.08651478],
[-0.04704883, -0.05007805, -0.06602913, ..., 0.01671332,
-0.00174393, 0.0273991 ],
[ 0.00090186, 0.03636878, 0.00770031, ..., 0.09038007,
-0.0007915 , -0.09847973]],
...,
[[-0.09494137, -0.09406457, 0.02975028, ..., -0.05521275,
-0.03369625, 0.05038466],
[ 0.04132238, 0.05268298, 0.07581738, ..., 0.05331679,
0.01129853, 0.01134449],
[-0.05164254, 0.05297703, -0.10541812, ..., -0.09950215,
0.07765144, 0.01491104],
...,
[-0.05781823, 0.10484386, -0.03988407, ..., -0.01388209,
-0.07905708, -0.07621384],
[-0.01874962, -0.10177012, -0.1066065 , ..., -0.03042223,
0.08387087, -0.10748719],
[-0.07822885, 0.00741419, 0.05201005, ..., -0.09704408,
0.07760924, 0.05827447]],
[[-0.09110893, 0.01062915, -0.07505407, ..., -0.08281508,
-0.01165547, 0.07199845],
[-0.03100552, -0.04160541, -0.00539404, ..., -0.10585603,
-0.1069422 , -0.0025361 ],
[ 0.04058671, 0.09900945, -0.01497427, ..., 0.04670303,
-0.08978279, -0.07830778],
...,
[-0.10465579, 0.09529752, 0.03410443, ..., -0.03280022,
-0.00988303, 0.09496238],
[-0.03048523, -0.06765232, 0.09354389, ..., 0.05783751,
0.01838607, -0.05888252],
[ 0.04455772, -0.01158155, -0.09908778, ..., 0.10394024,
0.0988012 , -0.07160645]],
[[ 0.06908762, 0.05069567, 0.06198643, ..., -0.06366103,
0.0874368 , 0.02867252],
[-0.04519062, 0.00938346, 0.03992433, ..., 0.05731647,
-0.02515235, -0.08775686],
[ 0.03648314, 0.08852491, -0.00790147, ..., -0.01143083,
-0.09166665, -0.04871982],
...,
[-0.00654683, 0.0279316 , -0.05816646, ..., -0.01921744,
-0.05128101, 0.01961367],
[ 0.06471586, 0.101045 , 0.05198625, ..., -0.0121538 ,
-0.03731428, -0.07555947],
[-0.06071796, -0.01045502, 0.03594328, ..., 0.0269032 ,
-0.03600143, 0.05694641]]], dtype=float32)},
'pre_ffw_norm': {'scale.weights': Array([1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1.], dtype=float32)},
'mlp': {'gating_linear.weights': Array([[-0.00123084, -0.04514119, 0.03579673, ..., 0.05871787,
0.0528171 , -0.07736286],
[ 0.03028335, -0.07560788, -0.08820161, ..., -0.07507264,
-0.02685089, -0.0076581 ],
[-0.0558075 , 0.01047585, -0.02508064, ..., 0.08823742,
-0.02641946, -0.0880558 ],
...,
[ 0.04958846, 0.0641031 , 0.0637101 , ..., -0.02504968,
-0.02165583, -0.03315994],
[ 0.03130015, -0.02695774, -0.0258101 , ..., 0.04227363,
0.0826306 , 0.00493999],
[-0.06670655, 0.06165688, 0.04407037, ..., 0.0764217 ,
-0.07342725, -0.04420125]], dtype=float32),
'value_linear.weights': Array([[-0.08228298, -0.08391917, -0.0754526 , ..., 0.02003063,
-0.08521492, -0.06497841],
[ 0.02560021, -0.08397669, 0.03775854, ..., -0.05371217,
0.02756315, 0.02553321],
[ 0.02202575, -0.04350768, 0.07286762, ..., 0.05969216,
0.07516016, -0.03858734],
...,
[ 0.07901482, -0.08766976, 0.04702545, ..., 0.0788388 ,
-0.00812442, 0.08169709],
[ 0.06137755, 0.04901019, 0.05495469, ..., -0.04430538,
-0.00505469, 0.06859718],
[-0.05233334, 0.0313801 , -0.06071386, ..., 0.06891893,
0.08679583, -0.05469652]], dtype=float32),
'out_linear.weights': Array([[-0.04437562, 0.08185953, -0.01521809, ..., 0.08608235,
-0.01774797, -0.08376244],
[ 0.04305138, -0.054964 , -0.06081209, ..., 0.05765019,
0.0581353 , -0.05897126],
[-0.02144684, 0.08668531, -0.08117946, ..., -0.00881826,
0.07438052, 0.04763403],
...,
[-0.02661774, 0.08259037, -0.00870859, ..., -0.07332407,
-0.04156006, -0.08168837],
[-0.0178968 , -0.0685404 , -0.04431031, ..., -0.07066294,
0.01078369, -0.08386905],
[ 0.07432064, 0.05199073, -0.06332663, ..., -0.06145025,
-0.02344846, 0.05295355]], dtype=float32)}},
'block_7': {'pre_attention_norm': {'scale.weights': Array([1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1.], dtype=float32)},
'attention': {'query.weights': Array([[[ 0.0826029 , -0.00425388, 0.03169761, ..., -0.08681224,
-0.10259559, -0.01046911],
[-0.09035416, 0.00422309, -0.01119245, ..., 0.10488611,
0.06731731, -0.08263723],
[-0.02623357, 0.07983278, 0.10495631, ..., -0.05102853,
0.02765278, -0.03441943],
...,
[ 0.04226084, -0.0077094 , -0.06297021, ..., -0.08256646,
0.04217712, -0.10795575],
[-0.02349254, -0.05347505, 0.01098422, ..., -0.0110059 ,
-0.06365795, 0.05700593],
[ 0.04058761, -0.03002662, -0.04325864, ..., 0.03400356,
0.00059858, 0.05952698]],
[[ 0.00545328, 0.02075846, 0.07557929, ..., -0.04247168,
-0.06978706, 0.08459496],
[-0.03143541, 0.0868286 , -0.04799568, ..., -0.09478574,
-0.00448121, -0.03394394],
[-0.02828287, 0.08909406, 0.05866461, ..., 0.0737257 ,
0.0459909 , -0.03282084],
...,
[-0.0963419 , 0.02381648, 0.00225145, ..., 0.05957538,
0.006216 , 0.09487171],
[-0.01388271, -0.09366029, -0.05563544, ..., 0.09679835,
-0.07978298, 0.10751657],
[ 0.01017909, 0.06582723, 0.08128956, ..., 0.08524464,
0.10614696, -0.00876372]],
[[-0.0525378 , 0.06104505, 0.08996472, ..., 0.01856694,
-0.0669741 , -0.08641382],
[-0.10139678, -0.10564042, -0.05391188, ..., -0.02104262,
-0.02504715, 0.10803774],
[ 0.0504121 , 0.00684023, -0.07088043, ..., -0.04863635,
0.01588452, -0.07743639],
...,
[ 0.0797791 , 0.04452605, 0.06135556, ..., -0.10414403,
0.0537828 , -0.03579436],
[-0.03514618, 0.10434571, -0.10188632, ..., -0.01831703,
0.097448 , -0.04344217],
[ 0.05777588, 0.03529319, -0.10805872, ..., -0.01008367,
0.01792167, 0.09387507]],
...,
[[-0.02773894, -0.06846757, -0.03936978, ..., 0.10487713,
0.0701421 , 0.00225145],
[ 0.05984658, -0.0676054 , 0.0156725 , ..., -0.04942597,
0.0819554 , 0.10143019],
[ 0.01194831, 0.10274827, 0.00625149, ..., -0.10771708,
0.06191563, -0.04034048],
...,
[ 0.05493623, 0.02964407, 0.05427334, ..., 0.00842582,
-0.09563957, -0.02857486],
[ 0.06861918, 0.00690517, -0.07286867, ..., -0.05873612,
-0.08873668, 0.01020792],
[ 0.02131065, 0.03677667, 0.04014327, ..., 0.0125714 ,
0.07728139, -0.09066609]],
[[ 0.02460999, -0.01743937, 0.02986381, ..., 0.09301629,
-0.08727023, -0.04198029],
[-0.08450659, 0.02830845, -0.03701004, ..., 0.09920783,
-0.00908662, 0.10740654],
[-0.0964445 , 0.03100263, 0.04736304, ..., -0.06214066,
0.01107852, -0.09054923],
...,
[ 0.0782161 , 0.09673364, -0.02409638, ..., 0.10100949,
0.00412881, 0.06968271],
[-0.00452047, 0.05621959, -0.10601801, ..., 0.03435166,
-0.07504609, 0.09747899],
[-0.0590955 , 0.05903193, 0.10191419, ..., 0.07193331,
-0.02463252, -0.00301311]],
[[-0.02183438, -0.03859054, 0.03731692, ..., 0.06219203,
0.05760662, 0.03623429],
[ 0.05513099, -0.04138082, 0.01079508, ..., -0.07138836,
-0.05001982, -0.00802567],
[ 0.05037845, -0.01571671, -0.06192146, ..., 0.02586237,
-0.07238285, -0.07148714],
...,
[ 0.08859754, -0.07239974, 0.02671928, ..., 0.04004272,
0.02395107, -0.05375098],
[ 0.02300755, 0.04339897, 0.02502263, ..., 0.05734019,
-0.01367548, 0.09936856],
[ 0.1017567 , 0.01798929, 0.08238352, ..., -0.07743851,
0.07668409, 0.03466973]]], dtype=float32),
'key.weights': Array([[[ 0.02238185, 0.09878787, -0.09140988, ..., -0.04741685,
0.1026375 , 0.09877153],
[ 0.06688676, -0.06107251, 0.04842422, ..., -0.03628033,
0.06577093, 0.04291914],
[ 0.01781196, -0.05660833, -0.09129949, ..., 0.05600916,
0.00850738, -0.03649873],
...,
[-0.10642766, -0.01046973, -0.09319964, ..., 0.04320083,
0.09599407, -0.07279581],
[-0.0670133 , -0.0227119 , -0.02838309, ..., -0.04185749,
-0.0184519 , 0.07676739],
[-0.0595635 , 0.07341382, -0.01824535, ..., 0.0208403 ,
0.06752085, 0.0682868 ]],
[[ 0.01315114, 0.04962473, 0.10395299, ..., 0.04110783,
-0.03748928, 0.08565646],
[-0.03326778, 0.10059416, -0.03440093, ..., -0.10214996,
-0.02161523, 0.02816887],
[-0.10436834, -0.04333024, -0.03876133, ..., -0.00980779,
0.04925704, 0.07431199],
...,
[ 0.10165308, 0.05172617, 0.00535636, ..., 0.03623746,
-0.07595082, -0.10292556],
[ 0.09317344, -0.02191973, -0.05932417, ..., -0.10523299,
0.07482545, -0.06258689],
[ 0.06575638, -0.08241609, 0.01992447, ..., 0.01741201,
-0.09897267, 0.02702649]],
[[-0.02237586, -0.06037813, -0.0224614 , ..., -0.01005417,
-0.01773259, -0.02805688],
[-0.09769043, -0.09113451, 0.06447072, ..., -0.04378518,
0.06742786, 0.08383306],
[-0.09379607, 0.1024415 , 0.01490523, ..., 0.0504392 ,
-0.04718141, -0.02362365],
...,
[-0.06543608, 0.04179914, -0.06277403, ..., -0.07891322,
0.05108206, 0.00056652],
[ 0.06179933, 0.06033067, 0.02285817, ..., 0.08770243,
0.06531289, -0.05389544],
[-0.10808866, 0.00610236, 0.04422839, ..., 0.01212652,
0.10089404, -0.08793996]],
...,
[[ 0.06749161, -0.03296488, 0.05733794, ..., 0.06272574,
0.09311555, -0.00198011],
[ 0.02237746, -0.09613981, -0.02146502, ..., -0.01539448,
0.09633 , -0.07056344],
[-0.07981785, 0.09954737, -0.09503222, ..., 0.08850826,
0.02152853, 0.09340823],
...,
[ 0.06915952, 0.08071617, -0.06486099, ..., -0.01289049,
0.06445619, 0.10795701],
[ 0.00146735, -0.00216127, 0.05383582, ..., 0.09378178,
0.06859829, -0.03172192],
[-0.08367976, -0.05659873, -0.08834778, ..., -0.04120196,
-0.08029919, -0.09321546]],
[[ 0.00747636, -0.05883085, -0.04243287, ..., -0.08255131,
0.02090709, 0.06904524],
[-0.02228269, 0.00529297, -0.06945747, ..., 0.01240114,
-0.08963627, -0.03705508],
[-0.02722223, -0.05726555, 0.06575494, ..., -0.04413038,
0.0700402 , 0.0628352 ],
...,
[ 0.06565631, 0.04534158, 0.00174638, ..., -0.10765857,
0.03929254, 0.08643704],
[-0.00798698, 0.07758772, 0.04473399, ..., 0.00145961,
-0.03854907, 0.00841062],
[ 0.0491758 , -0.0861505 , -0.08816499, ..., -0.07537086,
0.05542896, 0.01146972]],
[[-0.02553746, -0.03606064, -0.0509119 , ..., 0.00079189,
0.0401321 , -0.0935475 ],
[-0.08817155, -0.09735642, -0.02300838, ..., -0.10357418,
-0.10399963, -0.10262431],
[-0.0834941 , -0.07988371, -0.07319583, ..., -0.05359999,
-0.09561061, -0.02598451],
...,
[ 0.02277361, -0.05475159, -0.03999585, ..., 0.01315767,
-0.10493138, 0.0503522 ],
[-0.05044895, 0.02801226, -0.06305752, ..., 0.06601465,
0.02946549, 0.01241915],
[ 0.07687099, 0.09334053, -0.05831866, ..., -0.10369946,
-0.0194734 , -0.01031141]]], dtype=float32),
'value.weights': Array([[[ 0.07966152, -0.03417633, 0.08029238, ..., -0.10033588,
0.04478801, -0.02268971],
[-0.06254598, -0.09460533, 0.01164943, ..., -0.09950528,
-0.07807454, 0.06586467],
[ 0.04426702, -0.04034802, -0.1054906 , ..., -0.0926253 ,
0.06462298, 0.00334998],
...,
[ 0.02272731, 0.02428319, -0.01978141, ..., -0.06017976,
-0.07881662, -0.08143339],
[-0.06821278, 0.05135188, -0.03291696, ..., -0.01090173,
-0.05241017, -0.06336869],
[-0.05888453, -0.10669115, 0.09851677, ..., 0.08541682,
0.10429517, 0.06476137]],
[[ 0.01080964, -0.03232754, -0.00490201, ..., 0.04000692,
-0.09138277, 0.06407207],
[-0.04814261, 0.04331227, 0.00710744, ..., -0.03258107,
0.09604752, 0.08721443],
[-0.01713502, 0.06985145, -0.06879003, ..., -0.05083409,
-0.04508237, 0.1027004 ],
...,
[ 0.07685426, 0.06566793, 0.01109357, ..., -0.05058477,
-0.07761611, -0.05392605],
[-0.06933679, 0.06462599, 0.08492512, ..., -0.04080155,
0.10716736, -0.01471847],
[ 0.05798238, -0.03147879, 0.00698077, ..., 0.09009606,
-0.0056476 , -0.02337062]],
[[-0.06404203, -0.04201137, -0.07908472, ..., -0.00490787,
-0.06601545, -0.03157254],
[-0.01012138, -0.04770542, 0.00387786, ..., -0.10246757,
-0.06730095, -0.06444278],
[-0.05194875, -0.10231553, -0.08285751, ..., 0.04972665,
0.10491623, -0.08617058],
...,
[ 0.09816501, -0.070894 , -0.02219282, ..., 0.10595328,
-0.06416971, 0.04698689],
[-0.01729765, 0.04276909, -0.07234798, ..., 0.10665507,
0.01450451, 0.09596795],
[ 0.07920572, -0.01333629, -0.08241282, ..., -0.1010815 ,
0.10322439, -0.06785779]],
...,
[[ 0.06745016, 0.01120558, 0.08512767, ..., -0.07388645,
-0.04053057, 0.02690771],
[-0.08568119, -0.01573134, -0.05411998, ..., 0.02491271,
-0.08063526, -0.09256501],
[ 0.06698538, -0.09422325, 0.00677679, ..., -0.10344554,
0.09739878, 0.09740087],
...,
[-0.06111775, -0.01020727, 0.02519442, ..., -0.09393893,
-0.05062301, -0.08257484],
[-0.04698098, 0.01888811, 0.08286539, ..., 0.08331031,
-0.04749624, 0.08220479],
[-0.10793414, 0.00057723, 0.06526572, ..., 0.05784714,
-0.04017734, -0.06435306]],
[[-0.010008 , -0.01795548, 0.09489608, ..., -0.06344121,
-0.01590164, 0.06866547],
[ 0.0696984 , 0.07997707, 0.05653983, ..., -0.08688314,
-0.10192697, -0.10750145],
[ 0.0060979 , -0.05386924, 0.10329144, ..., -0.01615803,
-0.02385109, -0.03482911],
...,
[-0.06567702, 0.06020575, -0.01276038, ..., -0.02620791,
0.0640068 , -0.06302815],
[ 0.04573143, 0.02332831, -0.0237997 , ..., 0.10823036,
0.09633881, -0.05601956],
[-0.05884909, 0.07528106, -0.02520224, ..., -0.00206722,
-0.03033306, 0.02232659]],
[[ 0.08713702, 0.09832067, 0.00404482, ..., 0.05358918,
-0.04722323, 0.10669637],
[ 0.07040212, -0.01281881, 0.00158589, ..., -0.06199102,
-0.07344709, -0.07004903],
[ 0.0159755 , 0.07489061, -0.08109459, ..., -0.0879827 ,
-0.02050374, 0.00834108],
...,
[ 0.02649507, 0.01574822, 0.06015093, ..., -0.03870232,
-0.00481519, -0.0584406 ],
[-0.06233999, 0.03414474, 0.08690412, ..., 0.01247991,
-0.03583189, 0.07169898],
[ 0.09730726, 0.00294436, 0.02900469, ..., 0.09710238,
-0.08910412, -0.00109306]]], dtype=float32),
'output.weights': Array([[[-0.03458531, -0.02287853, 0.06896035, ..., -0.01827361,
0.10193873, -0.0286154 ],
[ 0.04892743, 0.08197823, 0.00579262, ..., 0.0909356 ,
-0.05621538, -0.0001744 ],
[ 0.00591666, 0.0056868 , 0.04349423, ..., -0.04461292,
0.10407316, 0.04133351],
...,
[-0.10061961, -0.06635217, 0.04653323, ..., -0.0171326 ,
-0.05236539, 0.05872487],
[ 0.00898008, -0.10025889, 0.07379854, ..., 0.0137095 ,
-0.02379929, -0.03496575],
[-0.08254486, 0.03268937, 0.09432811, ..., 0.04468663,
-0.09577069, -0.10760912]],
[[ 0.08876643, 0.08105177, 0.10800483, ..., -0.00062942,
-0.0335694 , 0.07347674],
[ 0.02283685, -0.02366082, -0.06368676, ..., 0.05191812,
-0.02481925, -0.09558695],
[ 0.00841531, 0.07931291, -0.05449151, ..., -0.00968403,
-0.10432865, -0.01249645],
...,
[ 0.00649237, 0.06676525, 0.0861425 , ..., 0.0696459 ,
0.07849813, 0.09440941],
[ 0.0468733 , -0.08486968, 0.07354444, ..., 0.05676257,
0.03424026, -0.05161968],
[-0.06857486, 0.09004083, 0.05060371, ..., -0.03515418,
0.09032193, 0.08030789]],
[[-0.06991293, -0.10135634, 0.09854256, ..., -0.03500539,
0.04997747, 0.07146943],
[-0.06035916, 0.0950881 , 0.06121255, ..., -0.02080427,
0.02748536, 0.00040085],
[-0.08521413, -0.07161293, 0.01130358, ..., -0.10779908,
-0.06482618, 0.02695693],
...,
[ 0.1005047 , 0.0284001 , 0.04512282, ..., -0.00036273,
-0.04204381, 0.01141681],
[ 0.04709288, -0.07388595, -0.05047528, ..., -0.06504829,
0.05944527, 0.09298255],
[ 0.09222344, 0.08260876, 0.09346042, ..., 0.09468498,
0.05628065, 0.10133727]],
...,
[[-0.10566512, -0.07689653, 0.06376127, ..., -0.07452133,
0.01609082, 0.04277107],
[-0.00277133, 0.06891312, 0.04815229, ..., 0.09156386,
0.10597499, -0.04824988],
[-0.03816587, -0.03968562, 0.10354014, ..., -0.10534183,
-0.06829198, -0.03447807],
...,
[ 0.06442641, 0.00661582, 0.0080139 , ..., -0.01889271,
-0.08389039, 0.07077239],
[ 0.02196934, -0.08954885, -0.05603169, ..., -0.09989072,
0.09170963, 0.06936239],
[ 0.00757348, 0.00404253, 0.04287875, ..., 0.09445179,
-0.06153383, 0.06870742]],
[[ 0.08226952, -0.06461275, -0.05913584, ..., -0.09530079,
-0.01464463, -0.0515743 ],
[-0.04707838, -0.01362611, -0.04240347, ..., -0.07829954,
-0.06640879, 0.04033424],
[-0.05476953, -0.04392205, 0.08152917, ..., 0.00381533,
0.10417911, -0.01282015],
...,
[-0.10551246, -0.04023198, -0.10599182, ..., 0.03795668,
0.07397719, 0.07732905],
[ 0.08589757, -0.04179034, -0.10341997, ..., 0.09933031,
-0.04539273, -0.00094238],
[ 0.02062693, 0.024514 , 0.05099514, ..., 0.06980265,
0.05196176, -0.07334656]],
[[ 0.09747326, 0.09790434, -0.05666738, ..., -0.05903786,
-0.03141427, -0.05878823],
[-0.04754177, -0.09608548, 0.10344217, ..., -0.07611967,
0.00180388, 0.09343311],
[-0.04334012, -0.08388966, -0.03569972, ..., -0.01400435,
-0.06098679, 0.02220583],
...,
[-0.10785021, -0.02400365, -0.0205104 , ..., -0.05236601,
-0.01224584, 0.03020512],
[ 0.03796742, 0.00646739, 0.05408586, ..., -0.04957006,
-0.10475907, 0.09361149],
[-0.07970635, 0.10217187, -0.10375689, ..., 0.06729084,
-0.10414411, -0.05656572]]], dtype=float32)},
'pre_ffw_norm': {'scale.weights': Array([1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1.], dtype=float32)},
'mlp': {'gating_linear.weights': Array([[ 0.07736463, -0.00892594, 0.04739518, ..., 0.02073821,
0.05673385, -0.02703472],
[ 0.08115689, 0.07910247, 0.05098418, ..., 0.07110944,
0.07792592, 0.01871366],
[-0.02565845, -0.05478924, -0.07211117, ..., -0.04485455,
-0.04728276, -0.02386197],
...,
[-0.0671441 , 0.05459157, -0.05684333, ..., -0.01082394,
-0.01310487, 0.08362027],
[ 0.07506417, -0.075744 , -0.0563831 , ..., 0.03734297,
0.03952689, 0.00160508],
[ 0.04146955, -0.05362841, 0.05816006, ..., -0.00589801,
0.08831246, -0.04726815]], dtype=float32),
'value_linear.weights': Array([[ 0.0814047 , -0.06475977, 0.01620506, ..., -0.04261436,
-0.01520829, -0.02937568],
[ 0.06288588, 0.04389459, 0.07258547, ..., 0.0779661 ,
0.01168024, 0.07391839],
[ 0.04160402, 0.03223121, 0.04517603, ..., 0.06788778,
-0.06511201, 0.0210816 ],
...,
[-0.0001084 , -0.04288362, -0.01602853, ..., 0.07885696,
0.04898703, 0.06088405],
[-0.04031274, -0.07514089, -0.02111013, ..., 0.03655177,
-0.01097268, 0.03941086],
[ 0.07512347, 0.04708536, 0.02899705, ..., 0.02598269,
0.08192534, 0.03254202]], dtype=float32),
'out_linear.weights': Array([[ 0.08800222, 0.00723187, 0.04301554, ..., 0.00355842,
-0.05786981, -0.08025222],
[-0.08549616, -0.07821018, 0.07963284, ..., 0.03710836,
0.04175682, 0.02466408],
[-0.06009689, 0.01207488, -0.05097592, ..., 0.01500548,
0.02164856, 0.05903066],
...,
[-0.07967521, 0.08674563, 0.05182031, ..., -0.01183467,
0.02822747, -0.00141557],
[-0.08413558, -0.01471816, -0.0061244 , ..., 0.05676042,
-0.06950022, -0.08400596],
[-0.066227 , -0.07541788, 0.01630202, ..., -0.00100678,
-0.08654082, -0.02470008]], dtype=float32)}},
'block_8': {'pre_attention_norm': {'scale.weights': Array([1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1.], dtype=float32)},
'attention': {'query.weights': Array([[[ 0.08473152, 0.09540545, 0.08482038, ..., -0.05944723,
0.06739417, -0.01572133],
[ 0.07021648, -0.06134653, 0.02879413, ..., -0.03368182,
-0.06385096, 0.02839169],
[-0.06133933, 0.0559322 , 0.10453773, ..., -0.07663323,
-0.05759482, -0.02627208],
...,
[ 0.07719316, 0.029461 , -0.06022134, ..., -0.01675152,
-0.09586807, -0.06737485],
[-0.02355025, -0.06573692, -0.10713198, ..., 0.03350639,
0.0770079 , -0.01585804],
[ 0.07025176, 0.10455628, -0.06263551, ..., 0.07364401,
-0.04708326, 0.01101217]],
[[-0.04558803, -0.09333741, 0.04621118, ..., -0.05828219,
-0.01355805, -0.05721385],
[ 0.02879297, 0.05314585, -0.03738413, ..., -0.10253432,
-0.10238601, -0.05795693],
[-0.09228017, 0.10095719, -0.09628022, ..., 0.0196591 ,
-0.0489529 , 0.02894695],
...,
[ 0.09670089, 0.00440252, -0.07056039, ..., -0.05337241,
0.06694439, 0.0845435 ],
[-0.09504025, -0.00027606, 0.0873005 , ..., 0.06641001,
0.06717969, 0.06856843],
[ 0.09722611, 0.10327234, 0.05677893, ..., -0.07737535,
0.01275282, 0.06855889]],
[[-0.07735786, 0.08785089, 0.09462348, ..., 0.02291477,
0.02728335, -0.01623239],
[-0.00012347, 0.09833262, -0.00689786, ..., 0.02462816,
-0.07798268, -0.0934734 ],
[-0.10815569, -0.06147452, -0.1001762 , ..., -0.04382534,
0.03024231, -0.07612801],
...,
[-0.00614371, 0.01222888, 0.04756804, ..., -0.0031346 ,
-0.08028737, 0.00999186],
[ 0.06571193, 0.06226517, -0.05775345, ..., 0.05101875,
0.00568014, -0.07373146],
[ 0.01340539, 0.08158915, 0.02031376, ..., 0.00816764,
-0.10820875, -0.02900226]],
...,
[[ 0.05808035, -0.07720271, -0.10415621, ..., -0.05232874,
0.05411363, -0.02513449],
[ 0.08450801, -0.00992682, -0.09986713, ..., 0.00583518,
-0.05770619, -0.05421816],
[ 0.08375729, -0.01496382, 0.0009548 , ..., 0.04488733,
0.02629043, -0.08656271],
...,
[ 0.06651407, 0.00301494, -0.05437487, ..., -0.04896813,
0.02919872, -0.07529017],
[ 0.00381739, -0.03913732, -0.10546876, ..., -0.02674287,
0.02772474, -0.00189656],
[ 0.07121704, 0.03979105, 0.0225284 , ..., -0.03067439,
-0.08034118, -0.06338892]],
[[ 0.10755727, 0.02054648, 0.00373224, ..., 0.04295288,
-0.0824848 , -0.00274105],
[-0.0713609 , 0.02727114, 0.06489568, ..., 0.09341164,
-0.03499635, -0.06791251],
[ 0.03517767, -0.00141867, -0.05830807, ..., -0.09197229,
0.0862843 , -0.01863482],
...,
[-0.08816069, -0.0433585 , -0.03020109, ..., -0.0220527 ,
-0.00613739, 0.04710066],
[ 0.06704552, -0.09379932, 0.08859756, ..., 0.01865758,
0.07175475, 0.0562278 ],
[-0.08152393, 0.08040429, -0.06993732, ..., -0.01113987,
0.07179535, 0.06499807]],
[[-0.02395146, -0.00742686, -0.00088251, ..., 0.06024157,
-0.01633302, -0.07678842],
[ 0.07598982, 0.07578579, 0.06655715, ..., 0.01689691,
0.07265094, 0.09277391],
[ 0.09386225, -0.02845402, -0.0203399 , ..., 0.07491527,
-0.02941937, 0.07631959],
...,
[-0.05212893, -0.05267196, 0.04727712, ..., -0.05267717,
0.06334279, 0.00384723],
[ 0.0465961 , 0.04243516, -0.10519456, ..., 0.08348755,
-0.06425158, 0.00420564],
[-0.10140765, -0.10376851, 0.08910508, ..., 0.05244871,
0.06024425, -0.06943724]]], dtype=float32),
'key.weights': Array([[[ 0.03641722, -0.03493681, 0.07986254, ..., 0.08018669,
0.01739067, -0.0193985 ],
[ 0.06245794, -0.0114165 , 0.06411386, ..., 0.0238093 ,
0.00564811, -0.0907564 ],
[-0.08109477, -0.0526787 , -0.09214103, ..., 0.02864416,
-0.08472133, 0.08280718],
...,
[-0.09712419, -0.0280853 , 0.01814428, ..., 0.0421559 ,
-0.03282954, -0.04875913],
[ 0.00663151, 0.07843123, -0.09421261, ..., 0.00514318,
-0.00659476, -0.0278357 ],
[-0.08919869, 0.05102043, 0.07069367, ..., 0.06527227,
0.02198926, -0.02783097]],
[[ 0.10558988, 0.10609087, 0.10287412, ..., -0.09887328,
0.10515081, 0.05987422],
[ 0.02981005, 0.08836012, -0.09181681, ..., 0.03606257,
-0.08680037, -0.01325195],
[ 0.03877129, 0.07596567, -0.01993479, ..., 0.01186267,
-0.04986107, -0.04028548],
...,
[ 0.06198705, -0.01027414, 0.09172568, ..., -0.07786109,
-0.06230035, 0.0774589 ],
[-0.06533738, 0.06374824, -0.00455456, ..., 0.00912776,
0.01137913, -0.06272309],
[ 0.07098098, 0.05109135, 0.05240524, ..., 0.03481799,
-0.05979984, 0.03967689]],
[[-0.08901674, -0.03927052, -0.05515246, ..., -0.00861381,
0.10168337, -0.0330724 ],
[ 0.07681366, -0.02413053, -0.00476938, ..., 0.04083894,
-0.10812581, 0.03105575],
[-0.08092316, -0.10218859, 0.05496883, ..., 0.03914359,
0.0753978 , 0.00280168],
...,
[-0.07009024, 0.05329302, 0.01919192, ..., -0.06262705,
-0.01261936, -0.10358285],
[ 0.03824196, -0.05891697, -0.08960705, ..., 0.09849589,
0.10061543, -0.03545711],
[-0.03561855, -0.01458981, 0.01700879, ..., 0.02247332,
-0.0211489 , -0.0682646 ]],
...,
[[ 0.09618398, -0.09278501, 0.08372255, ..., -0.0805774 ,
0.02391437, -0.04240442],
[ 0.05190082, -0.00677478, -0.07789307, ..., 0.07127186,
0.02280872, -0.00337574],
[-0.02952016, -0.06271121, -0.08776131, ..., -0.00563428,
0.0938041 , 0.05688062],
...,
[ 0.06068991, 0.04619699, -0.027655 , ..., 0.1066377 ,
-0.07928196, -0.07430766],
[-0.02351853, -0.07563896, -0.04964749, ..., 0.10327774,
0.06736457, 0.02327631],
[-0.01397774, -0.01248582, -0.10383341, ..., -0.03237302,
-0.0815897 , 0.07581527]],
[[-0.02997051, 0.06018458, 0.00789169, ..., 0.03680091,
0.00584868, 0.04397659],
[ 0.02269404, 0.05319793, 0.03355507, ..., 0.08254168,
0.06860653, 0.0356275 ],
[ 0.09519449, -0.07469934, -0.07698788, ..., -0.08347689,
-0.09526314, -0.08114314],
...,
[-0.03827334, 0.02042915, -0.0531969 , ..., -0.02049148,
-0.04962625, -0.09958268],
[-0.02870816, -0.1027108 , -0.03945898, ..., 0.0307696 ,
0.00297791, -0.02771775],
[-0.0623801 , 0.09898594, -0.10283608, ..., 0.05617533,
-0.0920882 , 0.09767577]],
[[-0.10545588, 0.05136909, -0.10752109, ..., 0.053488 ,
0.04468575, 0.03416712],
[-0.04560984, -0.07390417, 0.06994313, ..., 0.01528567,
-0.09367719, -0.07748254],
[-0.02744331, -0.0413971 , 0.03966925, ..., 0.07975959,
0.10279626, -0.04648151],
...,
[ 0.00677362, 0.0066287 , 0.07890449, ..., -0.07199737,
0.0626455 , -0.06254835],
[-0.10381008, 0.03518275, -0.0719402 , ..., -0.07323039,
-0.04645129, 0.09724361],
[ 0.02405836, 0.10168776, -0.04927509, ..., 0.09630819,
0.0772764 , 0.0049137 ]]], dtype=float32),
'value.weights': Array([[[ 0.00805186, 0.07114813, 0.0426637 , ..., 0.07362053,
-0.0267679 , 0.09954051],
[ 0.07569187, 0.08217586, -0.06700649, ..., -0.07635722,
-0.05809682, -0.02585213],
[ 0.09726635, 0.04791536, -0.07227603, ..., 0.03952204,
-0.05492653, -0.05716122],
...,
[-0.01243327, -0.00080405, 0.04425861, ..., -0.07654601,
-0.07170079, -0.10733347],
[-0.02757448, 0.00438948, -0.07918827, ..., 0.08775537,
0.09762562, -0.10473435],
[ 0.09688886, -0.02061121, 0.00732943, ..., -0.04200479,
0.00212364, 0.10375663]],
[[ 0.04189383, -0.03066143, 0.00991454, ..., 0.01829331,
0.00848363, 0.10674778],
[-0.04826351, -0.10783893, 0.03698707, ..., 0.06718706,
0.03786961, 0.10738084],
[-0.07687483, 0.07349741, 0.09514406, ..., -0.06943716,
-0.00193216, 0.00066759],
...,
[-0.09906118, -0.09894965, -0.07596845, ..., 0.05145635,
-0.02759915, 0.01238888],
[-0.01877236, -0.02462981, -0.08828963, ..., -0.03915397,
-0.09234922, 0.07794799],
[-0.10750078, -0.10369105, -0.07498487, ..., 0.04922058,
-0.02659815, 0.10003634]],
[[-0.04807667, 0.06243453, -0.06102502, ..., -0.03475818,
0.08394314, -0.03680966],
[ 0.08704814, 0.05768415, 0.00770222, ..., -0.10470443,
-0.07721709, 0.03568253],
[ 0.0450292 , -0.07560188, -0.04941291, ..., -0.06572205,
0.09024548, 0.08656269],
...,
[-0.02630031, 0.10632251, -0.00554986, ..., 0.049058 ,
0.04486596, 0.10626129],
[-0.01067535, 0.00510702, 0.02629437, ..., 0.08350521,
-0.03633879, 0.05595783],
[ 0.05150046, 0.04102725, 0.04855667, ..., 0.0474558 ,
0.07901596, -0.07070673]],
...,
[[-0.106392 , 0.06822094, -0.09278186, ..., -0.061297 ,
-0.00307299, 0.10664266],
[ 0.06138666, 0.01416976, -0.10650824, ..., -0.08375576,
-0.03745087, -0.01439851],
[ 0.08416105, -0.06020853, -0.03671202, ..., 0.00305714,
0.0889095 , 0.07791906],
...,
[-0.05417069, 0.02284756, 0.00773128, ..., -0.05419911,
-0.04002974, -0.03133865],
[-0.09871202, -0.00021887, 0.07789364, ..., 0.05160812,
-0.03277799, -0.07921421],
[-0.00259861, 0.07310057, 0.09239562, ..., 0.00122936,
-0.02864078, -0.09868252]],
[[ 0.06039723, 0.10675152, -0.06782016, ..., 0.00700978,
-0.0529802 , -0.01846946],
[-0.07885342, 0.00856413, 0.02825647, ..., -0.09467084,
-0.05767891, 0.02736785],
[-0.10296732, 0.09249999, -0.04144932, ..., 0.03437266,
-0.03847009, -0.08281606],
...,
[ 0.09412777, -0.07352433, 0.06192102, ..., 0.0884393 ,
-0.06223872, -0.0218861 ],
[-0.10290076, -0.03397884, -0.09086377, ..., 0.10038283,
-0.02601604, -0.10277597],
[-0.04685562, 0.06682143, 0.04047611, ..., 0.06590065,
-0.08070637, 0.00652776]],
[[-