Zjy}PSrSSKrSSKJr SSKrSSKJr SSKJr SSK J r SSK J r SS K Jr SS KJrJrJrJrJrJrJr S S KJr \R2"\5rSqS r"SS\R<R>5r S'Sjr!S'Sjr""SS\RF5r$"SS\RF5r%"SS\ 5r&\"SS\55r'\"SS9\"SS\555r(\"SS9\"SS \555r)\"S!S"\'55r*\"S#S9"S$S%\'\ 55r+/S&Qr,g)(zPyTorch RWKV model.N) dataclass)nn)initialization)GenerationMixin)GradientCheckpointingLayer)PreTrainedModel) ModelOutputauto_docstringis_bitsandbytes_availableis_kernels_availableis_ninja_availableis_torch_cuda_availablelogging) RwkvConfigcj[5(d [S5eSSKJn U"S5qU[lg)NzFkernels is not installed, please install it with `pip install kernels`r) get_kernelzkernels-community/rwkv)r ImportErrorintegrations.hub_kernelsrrwkv_cuda_kernelmax_seq_length)context_lengthrs w/root/GenerationalWealth/GenerationalWealth/venv/lib/python3.13/site-packages/transformers/models/rwkv/modeling_rwkv.pyload_wkv_cuda_kernelr-s/  ! !bcc6!":;&4#c<\rSrSr\SSj5r\SSj5rSrg)RwkvLinearAttention8Nc UR5upxn U[R:a [SUS[RS35eXy-[ U S5-S:wa[SUSU S[ U S5S 35eUR UlURRS :wdNURRS :wd4URRS :wdURRS :wa [S 5e[R"UR5R55*nUR [R:Xa0UR5nUR5nUR5nUR5nUR5nUR5n[R"U[RS 9n U(dUbUcT[R "UU S [R"UR[RS9nUSS2SS2S4==S-ss'OB[R$"UV s/sHoR'S5PM sn SS9R5nUR [R(:Xa[R*n O[R,n U "XX4X5 OHUR [R(:Xa[R.O[R0n U "XX4U 5 UR3XX4U 5 Ub4[R4"US SS9V s/sHoR7S5PM nn U R9UR 5U4$s sn fs sn f)NzCannot process a batch with z+ tokens at the same time, use a maximum of z with this model. rzThe product of batch size (z) and hidden size (z") needs to be a round multiple of .cudazUCalling the CUDA kernel for wkv attention requires all tensors to be on CUDA devices. memory_formatr)dtypedevicer%籡*G)dim)sizerr ValueErrorminr& input_dtyper'typetorchexpfloat contiguousfloat16 empty_likecontiguous_formatzerosfloat32cat unsqueezebfloat16forward_with_state_bf16forward_with_state forward_bf16forwardsave_for_backwardchunksqueezeto) ctx time_decay time_firstkeyvaluestate return_state batch_sizeseq_len hidden_sizeoutputs forward_funcs rr?RwkvLinearAttention.forward9s+.88:( [ %44 4.wi7b#2233DF   #c+r&: :a ?-j\9L[MZ";34A7  ))    " "f ,  %%/zz&(||  F*tu uii 0 0 2 = = ?@@ 99 %#))+J))+CKKME**, nn  "!!#U5L5LM 5,} --::"'"9"9 aAg$& 5"A5a;;q>5"AqITTVyyENN*/GG /BB V K>$A%..u}} '' %BYBYZ   E4K4KL""58O8OP %-- '~~'H:E:W(66]m]v]v          !      OOK ( OOK ( HH[ ! JJ{ #     rNFN)__name__ __module__ __qualname____firstlineno__ staticmethodr?rU__static_attributes__r^rrrr8s)<1<1|% % rrcUR5upgn[R"U5nUc[R"USS2S4[RS9n [R"USS2S4[RS9n [R"USS2S4[RS9S- n OUupn [R"U5*n[ U5GH n USS2U 4R 5n USS2U 4n[R"XU-5n[R"X- 5n[R"X-U- 5nUU -UU--nUU -U-nUU- RUR5USS2U 4'[R"X-U 5n[R"X-U- 5n[R"U U- 5nUU -UU--n UU -U-n Un GM U(dUbXU /nX4$)Nr)r&r)) r+r0 zeros_liker8r1ranger2maximumrCr&)rErFrGrHrIrJ_ seq_lengthrN num_state den_state max_state current_index current_key current_valuemax_for_outpute1e2 numerator denominator max_for_states rrwkv_linear_attention_cpurysxxzA1   c "F }$$SAYemmD $$SAYemmD $$SAYemmDtK */' i ))J''Jz* !]*+113 a./ y 2JK YYy1 2 YY{/.@ ANR-%77 9nr) $- $;#?#? #Mq-  i& YY{]2 3NR-%77 NR' ! %+(u(y1 =rc [SXX#455nURS5S:Hn[bU(dU(a [XX#XES9$[R XX#XE5$)Nc3R# UHoRRS:gv M g7f)r#N)r'r/).0ts r (rwkv_linear_attention..sX3Wa((--6)3Ws%'rrIrJ)anyr+rryrapply)rErFrGrHrIrJno_cuda one_tokens rrwkv_linear_attentionrsZXJC3WXXG q I7i(SXtt"((Uaarc@^\rSrSrSU4SjjrSSjrSSjrSrU=r$) RwkvSelfAttentionc>[TU]5 Xl[SL=(a [RUR :Hn[ 5(a,[5(aU(d[UR 5 X l URnURb UROUnXPl[R "["R$"U55Ul[R "["R$"U55Ul[R "["R$"SSU55Ul[R "["R$"SSU55Ul[R "["R$"SSU55Ul[R0"S5Ul[R4"XESS9Ul[R4"XESS9Ul[R4"XESS9Ul[R4"XTSS9Ulg![a [RS5 GNf=f)Nz9Could not load the custom CUDA kernel for RWKV attention.rrrrFbias)super__init__configrrrrrr Exceptionloggerinfolayer_idrMattention_hidden_sizer Parameterr0emptyrErF time_mix_keytime_mix_valuetime_mix_receptance ZeroPad2d time_shiftLinearrGrH receptancerN)selfrr kernel_loadedrMr __class__s rrRwkvSelfAttention.__init__s  (4q9I9X9X\b\q\q9q   $;$=$=m Y$V%:%:;! (( ,2,H,H,TF ( (Ze &;",,u{{3H'IJ,,u{{3H'IJLLQ;)GH ll5;;q![+IJ#%<< Aq+0N#O ,,}599[eLYY{N ))KUSii 5O ) Y WX Ys(H%%IIcpURS5S:XaUbUSSS2SS2UR4nO4URU5nUb USSS2SS2UR4USS2S4'XR-USUR- --nXR-USUR- --nXR -USUR - --nUR U5nURU5n[R"URU55nUb USS2S4USSS2SS2UR4'XdXR4$Nrrr) r+rrrrrrGrHr0sigmoidr)rhiddenrIshiftedrGrHrs rextract_key_value#RwkvSelfAttention.extract_key_values2 ;;q>Q 5#4Ahq!T]]23Goof-G %aAt}})< =1 (((7a$:K:K6K+LL,,,w!d>Q>Q:Q/RR666AH`H`D`9aa hhsm 5!]]4??:#>?  ,21b5ME!HQ4==( ),,rc ^TRXS9upEpbUb[U4SjUSS55OSn[TRTRUUUUS9upUbTUSUSSS2SS2TR 4'USUSSS2SS2TR 4'USUSSS2SS2TR 4'TR XH-5U4$) NrIc3N># UHoSS2SS2TR4v M g7fr`r)r|rOrs rr~,RwkvSelfAttention.forward.. sFIqaDMM12Is"%r(rrrr)rtuplerrErFrrN) rrrI use_cacherrGrH layer_staterwkvs ` rr?RwkvSelfAttention.forward s(,(>(>v(>(S% JOJ[eFE!"IFFae 1 OO OO  "    ",7NE!HQ4==( ),7NE!HQ4==( ),7NE!HQ4==( ){{:,-u44r) rrrGrrNrrErFrrrrrHrr`r_) rarbrcrdrrr?rf __classcell__rs@rrrsP<-&55rrc6^\rSrSrSU4SjjrSSjrSrU=r$)RwkvFeedForwardic8>[TU]5 XlX lURnUR b UR OSUR-n[ R"S5Ul[ R"[R"SSU55Ul [ R"[R"SSU55Ul [ R"X4SS9Ul[ R"X3SS9Ul[ R"XCSS9Ulg)NrrrFr)rrrrrMintermediate_sizerrrrr0rrrrrGrrH)rrrrMrrs rrRwkvFeedForward.__init__ s   (( (.(@(@(LF $ $RSV\VhVhRh ,,}5LLQ;)GH#%<< Aq+0N#O 99[%H))K5IYY0EJ rcURS5S:XaUbUSSS2SS2UR4nO4URU5nUb USSS2SS2UR4USS2S4'XR-USUR- --nXR-USUR- --n[ R "[ R"URU555nURU5n[ R"URU55nUb USS2S4USSS2SS2UR4'XV-U4$r) r+rrrrr0squarerelurGrHrr)rrrIrrGrrHs rr?RwkvFeedForward.forward1s# ;;q>Q 5#4Ahq!T]]23Goof-G %aAt}})< =1 (((7a$:K:K6K+LL666AH`H`D`9aa ll5::dhhsm45 3]]4??:#>?  ,21b5ME!HQ4==( )!5((r)rrGrrrrrrHrr`rarbrcrdrr?rfrrs@rrrsK"))rrc2^\rSrSrU4SjrSSjrSrU=r$) RwkvBlockiEc>[TU]5 XlX lUS:Xa.[R "UR URS9Ul[R "UR URS9Ul [R "UR URS9Ul [X5Ul [X5Ulg)Nr)eps)rrrrr LayerNormrMlayer_norm_epsilonpre_lnln1ln2r attentionr feed_forward)rrrrs rrRwkvBlock.__init__Fs   q=,,v'9'9v?X?XYDK<< 2 28Q8QR<< 2 28Q8QR*6<+F=rcURS:XaURU5nURURU5X#S9upRX-nUR UR U5US9upbX-nX4nU(aXu4- nU$US- nU$)Nr)rIrrr`)rrrrrr)rrrIroutput_attentionsrroutputss rr?RwkvBlock.forwardTs ==A [[(F>>$((6*:%>] #"//0@/N &/  | #G w Gr)rrrrrrr)NFFrrs@rrrEs >rrc\rSrSr%\\S'SrS/rSS/rSr Sr \ R"5S\ R4S j5rS rg ) RwkvPreTrainedModeligrrrrErFTmodulec  [U[5(GaURnURRnURR nUR nX#S- - nSX#- - n[R"[U5Vs/sHoU- PM snURRURRS9n U SSSS24n [U5V s/sHn SSXS- - SSU-----PM n n [R"XRRURRS9n [R"[U5Vs/sH oS-S -S- PM snURRURRS9S -n [R "URU 5 [R "UR[R""UR[$R&"S 5-U -55 [R "UR[R("X55 [R "UR*[R("X5S U--5 [R "UR,[R("U S U-55 g[U[.5(GaURnURRnURR nSX#- - n[R"[U5Vs/sHoU- PM snURRURRS9n U SSSS24n [R "UR[R("X55 [R "UR,[R("X55 g[U[0R25(aUR4R6n SnSnUR8b [R:"UR85 U S U S:a[$R<"U S U S- 5nU S URR>:XaU SURR :XaS nX-n[R@"UR4US 9 g[U[0RB5(a_UR4R6n S[$R<"[EU S U S55-n[R@"UR4US 9 g[U[0RF5(aA[RH"UR45 [R:"UR85 ggs snfs sn fs snfs snf)zInitialize the weights.rg?r&r'Ngffffff?g?rg?g333333?r)gaing-C6?)% isinstancerrrnum_hidden_layersrMrr0tensorrirr&r'rErFinitcopy_ ones_likemathlogpowrrrrrweightshaperzeros_sqrt vocab_size orthogonal_ Embeddingmaxrones_)rrrrrMr ratio_0_to_1ratio_1_to_almost0i time_weighth decay_speedzigzagrrscales r _init_weights!RwkvPreTrainedModel._init_weightspsu f/ 0 0H & ? ?  --33K$*$@$@ !#1'<=L!$(D!E ,,*/ *<=* L*>sV4 V9V>2Wr^N)rarbrcrdr__annotations__base_model_prefix_no_split_modules_keep_in_fp32_modulessupports_gradient_checkpointing _is_statefulr0no_gradrModulerrfr^rrrrgsR $ )<8&*#L ]]_I%BIII%I%rrz+ Class for the RWKV model outputs. ) custom_introc\rSrSr%SrSr\RS-\S'Sr \ \RS-\S'Sr \ \RS4S-\S'Sr \ \RS4S-\S'S rg) RwkvOutputiz state (list of five `torch.FloatTensor` of shape `(batch_size, hidden_size, num_hidden_layers)`): The state of the model at the last time step. Can be used in a forward method with the next `input_ids` to avoid providing the old `input_ids`. Nlast_hidden_staterI. hidden_states attentionsr^)rarbrcrd__doc__rr0 FloatTensorrrIlistrrrrfr^rrrrsw 37u((4/6,0E4!! "T )0:>M5**C/047>7;Je'',-4;rrzK Base class for causal language model (or autoregressive) outputs. c\rSrSr%SrSr\RS-\S'Sr \RS-\S'Sr \ \RS-\S'Sr \ \RS4S-\S'Sr\ \RS4S-\S 'S rg) RwkvCausalLMOutputiaP loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). state (list of five `torch.FloatTensor` of shape `(batch_size, hidden_size, num_hidden_layers)`): The state of the model at the last time step. Can be used in a forward method with the next `input_ids` to avoid providing the old `input_ids`. NlosslogitsrI.rrr^)rarbrcrdrr r0rrr rIrrrrrfr^rrrrs&*D%  d ")'+FE   $+,0E4!! "T )0:>M5**C/047>7;Je'',-4;rrc^\rSrSrU4SjrSrSr\SS\RS-S\RS-S\RS-S \ \RS-S \ S-S \ S-S \ S-S \ S-S\ \-4Sjj5rSrSrSrU=r$) RwkvModelic >[TU]U5 [R"URUR 5Ul[R"[UR5Vs/sH n[XS9PM sn5Ul [R"UR 5Ul SUlSUlUR!5 gs snf)NrF)rrrrrrM embeddings ModuleListrirrblocksrln_outlayers_are_rescaledgradient_checkpointing post_init)rridxrs rrRwkvModel.__init__s  ,,v'8'8&:L:LMmmPUV\VnVnPo$pPoYv%DPo$pq ll6#5#56 #( &+# %qs(C cUR$r`rrs rget_input_embeddingsRwkvModel.get_input_embeddingss rcXlgr`rrnew_embeddingss rset_input_embeddingsRwkvModel.set_input_embeddingss(rN input_idsattention_mask inputs_embedsrIrroutput_hidden_states return_dictreturnc UbUOURRnUbUOURRnUbUO(UR(dURROSnUbUOURR nUb[ RS5 URUR:XaUR5 UbUb [S5eUcUc [S5eUcURU5nU(aUcURS5URRURR4n [S5V s/sHCn [ R""XS::a UR$O[ R&UR(S .6PME nn US ==S -ss'UR*(a/UR(aU(a[ RS 5 SnUn U(aS OSn U(aS OSn[-UR.5H{unnU"XXVS9upnUR(a?URR0S:a%US-URR0-S:XaU S- n U(aX4-nU(dMuU U4-n M} UR3U 5n U(aX4-nU(d[5SXX455$[7U UUU S9$s sn f)a( input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values.get_seq_length()` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only `input_ids` that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) state (tuple of five `torch.FloatTensor` of shape `(batch_size, hidden_size, num_hidden_layers)`, *optional*): If passed along, the model uses the previous state in all the blocks (which will give the output for the `input_ids` provided as if the model add `state_input_ids + input_ids` as context). use_cache (`bool`, *optional*): If set to `True`, the last state is returned and can be used to quickly generate the next logits. NFz<`attention_mask` was passed, but it is unused in this model.zDYou cannot specify both input_ids and inputs_embeds at the same timez5You have to specify either input_ids or inputs_embedsrrrrgꌠ9Y>)FzZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...r^)rIrrr(c3.# UH ocMUv M g7fr`r^)r|xs rr~$RwkvModel.forward..`st$bq$bs )rrIrr)rrr$trainingrr%r warning_oncer_rescale_layersr,rr+rMrrir0r7r&r8r'r enumerater rescale_everyrrr)rr!r"r#rIrrr$r%kwargsrrrall_self_attentionsall_hidden_statesrblockrs rr?RwkvModel.forwards@2C1N-TXT_T_TqTq$8$D $++JjJj "+!6IZ^ZgZgT[[=R=Rmr %0%cd d  =#8TU U   OOI6M "''*DKK,C,CT[[EbEbcE q "A a-"5"5U]][h[o[o"  !H H  & &4==##p" % $5b4"6BD#DKK0JC/4i0 ,M* ((KK--11W 9 99Q> - 1 #$58H$H!  &9ZM&I#!1$ M2  14D D t];L$btt t++*   [s A Kc URUR(+:XagURRS:Ga[R "5 [ UR5GHupUR(aURRRRS[XRR-5-5 URRRRS[XRR-5-5 M[URRRS5(aURRRR R#S[XRR-5-5 URRRR R#S[XRR-5-5 GM[URRRS5(aOUR%URRU5 UR%URRU5 GM!URRRR#S[XRR-5-5 URRRR#S[XRR-5-5 GM SSS5 UR(+Ulg!,(df  N%=f)Nrr(SCB quant_state)rr,rr0r0rr/rrrNrmul_intrrHhasattrr7div_ _bnb_4bit_dequantize_and_rescale)rblock_idr4s rr.RwkvModel._rescale_layersis?  # #DMM(9 :  ;; $ $q ('0'=OH}}..55::1HP[P[PiPiDi@j;jk**0077<>qCT_T_TmTmHmDn?no!..44;;@@c(VaVaVoVoJoFpApq(>!"(,}}#4 #!s KM Mc[5(d [S5eSSKnURR UR R UR R5nURS[X RR-5-5 URRURS5SS9RUR5n[!USU5 g) z Perform the dequantization and rescaling of the weights of a given layer. After that operation the layer will be quantized again. z/Please install bitsandbytes to use this method.rNr(cpuF) requires_gradr)r r bitsandbytes functionaldequantize_4bitrdatar8r<r:rr0r Params4bitrCr'setattr)r target_layerr>bnbdequant_weights quant_weights rr=*RwkvModel._bnb_4bit_dequantize_and_rescales )**OP P"..889L9L9Q9QS_SfSfSrSrsQ#h++2K2K&K"LLMvv((););E)BRW(X[[\k\r\rs  h 5r)rrrrr)NNNNNNNN)rarbrcrdrrrr r0 LongTensorrrboolrrr?r.r=rfrrs@rr r s ).2262604!%)-,0#'h ##d*h ((4/h ((4/ h E%%&- h $; h  $;h #Tkh D[h   h h T5066rr z The RWKV Model transformer with a language modeling head on top (linear layer with weights tied to the input embeddings). cT^\rSrSrSS0rU4SjrSrSr\SS\ RS-S \ RS-S \ RS-S \ \ RS-S \ RS-S \ S-S\ S-S\ S-S\ S-S\\ R-S\\-4Sjj5rSrU=r$)RwkvForCausalLMiz head.weightzrwkv.embeddings.weightc>[TU]U5 [U5Ul[R "UR URSS9UlUR5 g)NFr) rrr rrrrMrheadr)rrrs rrRwkvForCausalLM.__init__sH  f% IIf00&2C2C%P  rcUR$r`rSrs rget_output_embeddings%RwkvForCausalLM.get_output_embeddingss yyrcXlgr`rVrs rset_output_embeddings%RwkvForCausalLM.set_output_embeddingss" rNr!r"r#rIlabelsrrr$r%logits_to_keepr&c U bU OURRn URUUUUUUU S9n U Sn [U [5(a [ U *S5OU nUR U SS2USS245nSnUb)UR"SXURRS.U D6nU (dU4U SS-nUbU4U-$U$[UUU RU RU RS9$)a input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values.get_seq_length()` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only `input_ids` that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) state (tuple of five `torch.FloatTensor` of shape `(batch_size, hidden_size, num_hidden_layers)`, *optional*): If passed along, the model uses the previous state in all the blocks (which will give the output for the `input_ids` provided as if the model add `state_input_ids + input_ids` as context). labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for language modeling. Note that the labels **are shifted** inside the model, i.e. you can set `labels = input_ids` Indices are selected in `[-100, 0, ..., config.vocab_size]` All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` use_cache (`bool`, *optional*): If set to `True`, the last state is returned and can be used to quickly generate the next logits. N)r#rIrrr$r%r)r r\rr)r r rIrrr^) rr%rrr:slicerS loss_functionrrrIrr)rr!r"r#rIr\rrr$r%r]r1 rwkv_outputsr slice_indicesr r rNs rr?RwkvForCausalLM.forwardsL&1%SV8W8W~ot4]k =M1)<=>  %%pVt{{OeOepiopDYab!11F)-)9TGf$ Ev E!$$&44#..   r)rSr) NNNNNNNNNr)rarbrcrd_tied_weights_keysrrWrZr r0rNrrrOr:Tensorrrr?rfrrs@rrQrQs(()AB#.2262604*.!%)-,0#'-.D ##d*D ((4/D ((4/ D E%%&- D   4' D $;D  $;D #TkD D[D ell*D  # #D D rrQ)rQr rr_)-rr dataclassesrr0rrr generationrmodeling_layersrmodeling_utilsr utilsr r r r rrrconfiguration_rwkvr get_loggerrarrrautogradFunctionrryrrrrrrrrr rQ__all__r^rrrqs} ! &)9-+   H %5g %..11g T)XbC5 C5L#)bii#)L*DR%/R%R%j   < <  <  <<  <$k6#k6k6\ V )?V  V r Br