Zj SrSSKrSSKrSSKrSSKJr SSKJrJrJr SSK J r SSK J r SSKJrJrJr SS KJr SS KJrJr SS KJr SS KJrJrJrJrJrJrJ r SS K!J"r" SSK#J$r$J%r%J&r&J'r'J(r( SSK)J*r* \'RV"\,5r-"SS\R\5r/"SS\R\5r0"SS\R\5r1"SS\R\5r2"SS\R\5r3"SS\R\5r4"SS\R\5r5"SS \5r6"S!S"\R\5r7\&"S#S$\"55r8"S%S&\85r9\&"S'S(\855r:\&"S)S*9"S+S,\8\55r;\&"S-S.\855r<\&"S/S*9"S0S1\855r=\&"S2S3\855r>\&"S4S5\855r?/S6Qr@g)7zPyTorch T5 model.N)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)ACT2FN)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)create_bidirectional_maskcreate_causal_mask)GradientCheckpointingLayer)BaseModelOutput)BaseModelOutputWithPastAndCrossAttentionsSeq2SeqLMOutputSeq2SeqModelOutput#Seq2SeqQuestionAnsweringModelOutputSeq2SeqSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel) DUMMY_INPUTS DUMMY_MASKauto_docstringloggingtorch_compilable_check)T5Configc2^\rSrSrSU4SjjrSrSrU=r$) T5LayerNorm.c>[TU]5 [R"[R "U55UlX lg)zS Construct a layernorm module in the T5 style. No bias and no subtraction of mean. N)super__init__r Parametertorchonesweightvariance_epsilon)self hidden_sizeeps __class__s s/root/GenerationalWealth/GenerationalWealth/venv/lib/python3.13/site-packages/transformers/models/t5/modeling_t5.pyr%T5LayerNorm.__init__/s/ ll5::k#:; #cUR[R5RS5R SSS9nU[R "X R -5-nURR[R[R4;a%URURR5nURU-$)NT)keepdim) tor'float32powmeanrsqrtr*r)dtypefloat16bfloat16)r+ hidden_statesvariances r/forwardT5LayerNorm.forward7s !##EMM266q9>>r4>P% H?T?T4T(UU  ;;   ? ?),,T[[->->?M{{]**r1)r*r))gư>)__name__ __module__ __qualname____firstlineno__r%r@__static_attributes__ __classcell__r.s@r/r!r!.s$ + +r1r!c6^\rSrSrS\4U4SjjrSrSrU=r$)T5DenseActDenseGconfigcX>[TU]5 [R"URUR SS9Ul[R"UR URSS9Ul[R"UR5Ul [URUl gNFbias)r$r%rLineard_modeld_ffwiwoDropout dropout_ratedropoutr dense_act_fnactr+rLr.s r/r%T5DenseActDense.__init__Hsn ))FNNFKKeD))FKKeDzz&"5"56 &--.r1cURU5nURU5nURU5n[URR [ R5(aURURR R:waaURR R[ R:wa/URURR R5nUR U5nU$N) rTrZrX isinstancerUr)r'Tensorr;int8r6r+r>s r/r@T5DenseActDense.forwardOs . /  ]3 tww~~u|| 4 4##tww~~';';;$$ 2),,TWW^^-A-ABM . r1)rZrXrTrU rBrCrDrErr%r@rFrGrHs@r/rJrJGs/x/  r1rJc6^\rSrSrS\4U4SjjrSrSrU=r$)T5DenseGatedActDense]rLc>[TU]5 [R"URUR SS9Ul[R"URUR SS9Ul[R"UR URSS9Ul[R"UR5Ul [URUlgrN)r$r%rrQrRrSwi_0wi_1rUrVrWrXr rYrZr[s r/r%T5DenseGatedActDense.__init__^s IIfnnfkkF IIfnnfkkF ))FKKeDzz&"5"56 &--.r1c8URURU55nURU5nX#-nURU5n[ UR R [R5(aURUR R R:waaUR R R[R:wa/URUR R R5nUR U5nU$r^) rZrirjrXr_rUr)r'r`r;rar6)r+r> hidden_gelu hidden_linears r/r@T5DenseGatedActDense.forwardfshhtyy78  -0 #3  ]3 tww~~u|| 4 4##tww~~';';;$$ 2),,TWW^^-A-ABM . r1)rZrXrirjrUrdrHs@r/rfrf]s/x/r1rfc6^\rSrSrS\4U4SjjrSrSrU=r$) T5LayerFFzrLc>[TU]5 UR(a[U5UlO[ U5Ul[ URURS9Ul [R"UR5Ul g)Nr-)r$r% is_gated_actrfDenseReluDenserJr!rRlayer_norm_epsilon layer_normrrVrWrXr[s r/r%T5LayerFF.__init__{s_    "6v">D "1&"9D %fnn&:S:STzz&"5"56 r1cpURU5nURU5nXRU5-nU$r^)rxrvrX)r+r>forwarded_statess r/r@T5LayerFF.forwards;??=9../?@% 5E(FF r1)rvrXrxrdrHs@r/rqrqzs7x7r1rqct^\rSrSrS S\S\S-4U4Sjjjr\S Sj5rS Sjr S Sjr S r U=r $) T5AttentionNrL layer_idxc>[TU]5 URUlX lURUlUR UlUR UlURUlURUl URUl URUR-Ul X0lUc>UR(a-[R!SUR"R$S35 [&R("UR URSS9Ul[&R("UR URSS9Ul[&R("UR URSS9Ul[&R("URUR SS9UlUR(a0[&R2"URUR5UlSUlg)NzInstantiating a decoder z without passing `layer_idx` is not recommended and will to errors during the forward call, if caching is used. Please make sure to provide a `layer_idx` when creating this class.FrO)r$r% is_decoderhas_relative_attention_biasrelative_attention_num_bucketsrelative_attention_max_distancerRd_kvkey_value_proj_dim num_headsn_headsrWrX inner_dimrlogger warning_oncer.rBrrQqkvo Embeddingrelative_attention_biasgradient_checkpointingr+rLrrr.s r/r%T5Attention.__init__se  +++F(.4.S.S+/5/U/U,~~ "(++'' ** (?(??"     *4>>+B+B*CD,,  4<<eD4<<eD4<<eD4>>4<=max_distance map to the same bucket. All relative positions <=-max_distance map to the same bucket. This should allow for more graceful generalization to longer sequences than the model has been trained on Args: relative_position: an int32 Tensor bidirectional: a boolean - whether the attention is bidirectional num_buckets: an integer max_distance: an integer Returns: a Tensor with the same shape as relative_position, containing int32 values in the range [0, num_buckets) rr3r) r6r'longabsmin zeros_likelogfloatmath full_likewhere)relative_position bidirectional num_buckets max_distancerelative_buckets max_exactis_smallrelative_position_if_larges r/_relative_position_bucket%T5Attention._relative_position_buckets,  A K Q!6 : :5:: F T T  % *; < !&+N>NO`>a!b b  1$ $0&/ II'--/); <hh|/0 1& ( "UZZ. &" &+YY &8RbcTc(d& " EKKE_``r1cUc URRRn[R"U[R US9SS2S4U-n[R"U[R US9SSS24nXe- nUR UUR(+URURS9nURU5n U R/SQ5RS5n U $)z%Compute binned relative position biasN)r;device)rrr)r3rrr) rr)rr'arangerrrrrpermute unsqueeze) r+ query_length key_lengthrpast_seen_tokenscontext_positionmemory_positionrrelative_position_bucketvaluess r/ compute_biasT5Attention.compute_biass >1188??F << EJJvVWXZ^W^_brr,,zFSTXZ[T[\+>#'#A#A #.;;== $B$  --.FG *44Q7 r1c :URSSn/UQSPURP7n UbURUR5OSn [ U [ R 5(aU R5OU n USLn URU5RU 5RSS5n Sn [ U[5(aFURRUR5n U (a URnOURnOUnU (aUOUnU (aQUbNU (aGUR URR"nUR URR$nO/URSSQSPURP7nUR'U5RU5RSS5nUR)U5RU5RSS5nUbUUR+UUUR5unnU (a.[ U[5(aSURUR'[ R,"U URSS55nUcURS nUR.(dh[ R0"SU RSUSU4UR2UR4S 9nUR6(aUR8(aSUlOUR=USUUR2U S 9nUb#USS2SS2SS2SURS 24nUU-nUnUU- n[>R@RCURE5SS 9RGU5n[>R@RIUURHUR8S 9n[ R,"UU5nURSS5RK5nURL"/UQSP76nUROU5nUU4nU(aUU4-nU$)zp Self-attention (if key_value_states is None) or attention over source sentence (provided by key_value_states). Nr4rrr3FTr)rr;)rrdim)ptraining)(shaperget_seq_lengthrr_r'r`clonerview transposer is_updatedgetcross_attention_cacheself_attention_cachelayerskeysrrrupdatematmulrzerosrr;rr requires_gradrr functionalsoftmaxrtype_asrX contiguousreshaper)r+r>maskkey_value_states position_biaspast_key_valuesoutput_attentionskwargs input_shape hidden_shaperis_cross_attention query_statesrcurr_past_key_valuescurrent_states key_states value_stateskv_shapescoresr causal_maskposition_bias_masked attn_weights attn_outputoutputss r/r@T5Attention.forwards$))#2. BBbB$*A*AB M\Mh?99$..Ino7ABRTYT`T`7a7a+113gw.T9vvm,11,?II!QO  o': ; ;(3377GJ!'6'L'L$'6'K'K$#2 -?)] /"=*-44T^^DIIJ/66t~~FMMLP--cr2PBP8O8OPH/44X>HHANJ66.166x@JJ1aPL*+?+F+FzS_aeaoao+p( L%*_FY*Z*ZAEO..t~~>lJ,@,@A,FG  #))"-J33 %  **1-{1~zJSYS`S`hnhtht! ..4==26M/ $ 1 1NJv}}Wg!2! "1a,Bj.>.>r.B,B#BC - ; ,&&}},,V\\^,DLLVT }},,\T\\TXTaTa,b ll<> !++Aq1<<> !));;;; ff[)  . /Gr1)rRrXrrrrrrrrrrrrrrFN)T )Nr)NNNNF) rBrCrDrErintr% staticmethodrrr@rFrGrHs@r/r~r~sc%* $ , ,: , ,D- - ^([[r1r~cN^\rSrSrSS\S-4U4SjjjrSSjrSrU=r$) T5LayerSelfAttentioniNNrc>[TU]5 [XUS9Ul[ UR UR S9Ul[R"UR5Ul g)Nrrrt) r$r%r~ SelfAttentionr!rRrwrxrrVrWrXrs r/r%T5LayerSelfAttention.__init__OsQ ( W` &fnn&:S:STzz&"5"56 r1c URU5nURUUUUUUS9n XRU S5-nU4U SS-n U $)N)rrr use_cacherrr)rxrrX) r+r>attention_maskrrrrrnormed_hidden_statesattention_outputrs r/r@T5LayerSelfAttention.forwardWsn $}=-- '+/ . & 5Ea5H(II  "%5ab%99r1)rrXrxr)NNNFF rBrCrDrErr%r@rFrGrHs@r/rrNs47SSWZ77r1rcL^\rSrSrSS\S-4U4SjjjrSSjrSrU=r$) T5LayerCrossAttentionioNrc>[TU]5 [USUS9Ul[ UR UR S9Ul[R"UR5Ul g)NFrrt) r$r%r~EncDecAttentionr!rRrwrxrrVrWrX)r+rLrr.s r/r%T5LayerCrossAttention.__init__psN *6u`ij%fnn&:S:STzz&"5"56 r1c URU5nURUUUUUUS9n XRU S5-n U 4U SS-n U $)N)rrrrrrr)rxrrX) r+r>rrrrrrrr layer_outputrs r/r@T5LayerCrossAttention.forwardvsm $}=// -'+/ 0 %||4DQ4G'HH /$4QR$88r1)rrXrxr^)NNNFrrHs@r/rros07#*77r1rcV^\rSrSrSS\S-4U4SjjjrSSjrSrU=r$) T5BlockiNrcl>[TU]5 URUl[R"5UlUR R [XUS95 UR(a"UR R [XS95 UR R [U55 g)Nr)r) r$r%rr ModuleListlayerappendrrrqrs r/r%T5Block.__init__s}  ++]]_   lu v  ?? JJ  3FP Q )F+,r1c URS"UUUUUU S9n U SnU SSn UR[R:Xa[R"[R "U5R 5[R"UR5RS- [R"UR5R5n[R"X*US9nUR=(a USLnU(aURS"UUUUUU S9nUSnUR[R:Xa[R"[R "U5R 5[R"UR5RS- [R"UR5R5n[R"X*US9nU USS-n URS"U5nUR[R:Xa[R"[R "U5R 5[R"UR5RS- [R"UR5R5n[R"X*US9nU4nUU -$)Nr)rrrrrri)rmax)rrrrrr4) rr;r'r<risinfanyfinfor clampr)r+r>rrencoder_hidden_statesencoder_attention_maskencoder_decoder_position_biasrrr return_dictrself_attention_outputsattention_outputs clamp_valuedo_cross_attentioncross_attention_outputsrs r/r@T5Block.forwardsM"&A )'+/ " /q1 2126   %-- /++ M*..0 M//044t; M//044K "KK <[YM!__R1Fd1R &*jjm!65; /"3 ' #4A6M""emm3#kkKK .224KK 3 34884?KK 3 3488 !& M|Q\ ] !24KAB4O O  2}5    %-- /++ M*..0 M//044t; M//044K "KK <[YM " ' ' r1)rrr) NNNNNNFFTrrHs@r/rrsB -SSWZ - -"#&*J J r1rcn^\rSrSrSrS\4U4SjjrS\RS\R4Sjr Sr U=r $) T5ClassificationHeadiz-Head for sentence-level classification tasks.rLc(>[TU]5 [R"URUR5Ul[R "URS9Ul[R"URUR5Ul g)N)r) r$r%rrQrRdenserVclassifier_dropoutrX num_labelsout_projr[s r/r%T5ClassificationHead.__init__sZ YYv~~v~~> zzF$=$=>  &..&2C2CD r1r>returncURU5nURU5n[R"U5nURU5nUR U5nU$r^)rXrr'tanhrrbs r/r@T5ClassificationHead.forwardsN ]3  =1  =1  ]3  m4 r1)rrXr) rBrCrDrE__doc__rr%r'r`r@rFrGrHs@r/rrs47ExE U\\ellr1rc|\rSrSr%\\S'SrSrSrS/r S/r \ S5r \ R"5S5rS rS rg ) T5PreTrainedModelirL transformerTrrUcz[R"[5n[R"[5nUUUS.nU$)N)decoder_input_ids input_idsdecoder_attention_mask)r'tensorrr)r+r+ input_mask dummy_inputss r/r/T5PreTrainedModel.dummy_inputss6LL. \\*- !*"&0 r1cURRn[U[5(a%[R "UR US-5 g [U[[[[45(Ga[R"URR SUS-S9 [US5(aHURR(d-[R"URR SUS-S9 [US5(an[R"UR R SX RR"S--S9 [R$"UR R&5 g g [U[(5(aj[US5(aX[R"UR*R SUS-S9 [R$"UR*R&5 g g [U[,5(GaA[R"UR.R SX RR"S--S9 [UR.S5(aAUR.R&b*[R$"UR.R&5 [R"UR0R SX RR"S--S9 [UR0S5(aCUR0R&b+[R$"UR0R&5 g g g [U[25(GaA[R"UR4R SX RR"S--S9 [UR4S5(aAUR4R&b*[R$"UR4R&5 [R"UR6R SX RR8S--S9 [UR6S5(aCUR6R&b+[R$"UR6R&5 g g g [U[:5(Ga[R"UR<R SX RR"S--S9 [UR<S5(aAUR<R&b*[R$"UR<R&5 [R"UR>R SX RR"S--S9 [UR>S5(aAUR>R&b*[R$"UR>R&5 [R"UR6R SX RR8S--S9 [UR6S5(aCUR6R&b+[R$"UR6R&5 g g g [U[@5(GaFURR"nURRBnURRDn[R"URFR SX#U-S--S9 [R"URHR SX#S--S9 [R"URJR SX#S--S9 [R"URLR SX%U-S--S9 URN(a0[R"URPR SX#S--S9 g g g ) zInitialize the weightsg?g)r9stdlm_head qa_outputs classifierrPN))rLinitializer_factorr_r!init constant_r)T5ModelT5ForConditionalGenerationT5EncoderModelT5ForQuestionAnsweringnormal_sharedhasattrtie_word_embeddingsr3r4rRzeros_rPT5ForTokenClassificationr6rrrrJrTrUrSrfrirjr~rrrrrrrr)r+modulefactorrRrrs r/ _init_weightsT5PreTrainedModel._init_weightssO// fk * * NN6==&3, 7   0.BX Y   LL--CVc\ Jvy))$++2Q2Q V^^22&3,Ov|,, V..55CVP[P[PcPchlOlEmn F--223- 8 9 9v|,, V..55CVc\R F--223- 4 5 5 LL,,3F{{GZGZ_cFc%>%@#qr'"$:&!  PQ Q&&'8D'@,O  r1N)rBrCrDrEr__annotations__base_model_prefixsupports_gradient_checkpointing_can_compile_fullgraph_no_split_modules_keep_in_fp32_modulespropertyr/r'no_gradrFrPrFrRr1r/r'r's^ %&*#!" !F  ]]_4p4pl!r1r'cL^\rSrSrU4SjrSrSSjrSrU=r$)T5Stacki\c >[TU]U5 [R"URUR 5UlURUl[R"[UR5Vs/sHn[U[US:H5US9PM sn5Ul [UR URS9Ul[R""UR$5UlUR)5 SUlgs snf)NrrrtF)r$r%rr vocab_sizerR embed_tokensrrrange num_layersrboolblockr!rwfinal_layer_normrVrWrX post_initr)r+rLir.s r/r%T5Stack.__init__]s  LL):):FNNK ++]]]bcictct]u v]uXYWVa1fQR S]u v  !,FNN@Y@Y Zzz&"5"56  &+# ws9!DcXlgr^)r_r+new_embeddingss r/set_input_embeddingsT5Stack.set_input_embeddingsms*r1c  UbUOURRnUbUOURRnU bU OURRn U bU OURRn Ub*Ub'UR (aSOSn [ SU SU S35eUb&UR5n URSU S5nO>UbUR5SSn O'UR (aSOSn [ SU SU S 35eUR(a/UR(aU(a[RS 5 S nUc)URc [ S 5eURU5nU upUS La UR (d[ SUS35eUR (ahU(a`Uc]URR(a/[[!URS9[!URS95nO'[!URS9nOUR (dSnURR (a8[#URUU[%U[5(a UR&OUS9nO[)URUUS9nSnUR (aUb[)URUUUS9nU (aSOSnU(aSOSnU(aUR (aSOSnSnSnUR+U5nUR,HynU (aUU4-nU"UUUUUUUUUU S9 nUSnUSnUR (aUbUU(aSOSnU(dMTUUS4-nUR (dMpUUS4-nM{ UR/U5nUR+U5nU (aUU4-nU (d[1SUUUUU455$[3UUUUUS9$)Ndecoder_zYou cannot specify both zinput_ids and zinputs_embeds at the same timer4zYou have to specify either z input_ids or inputs_embedszZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...Fz"T5Stack.forward..s" As )last_hidden_staterr> attentionscross_attentions)rLrroutput_hidden_statesrrrLsizerrrrrr_is_encoder_decoderr r rr_rrrXrcrdtupler)r+r+rrrrprrrryrrerr_msg_prefixr batch_size seq_lengthencoder_extended_attention_maskall_hidden_statesall_attentionsall_cross_attentionsrrr> layer_module layer_outputss r/r@T5Stack.forwardps"+!6IDKK+/??ZN*>*:.HXXvw  "#..*K!r;r?;I  &',,.s3K+/??ZN:>:J-XfWggtuv v  & &4==##p"    ( !_`` --i8M!,  ?? #LTFRg!hii ??_4;;11&9$DKK8,dkk:Z'O'3$++&FO#O ;; ! !/{{+-o/BCC!0 D D$ N7{{+-N +/' ??4@.G{{+5&; / +#7BD0d&7DOOrRV (,% ]3  JJL#$58H$H!(%/- /#"3' M*!,M *!,M#8#D0=CTaZ[0\-  !/=3C2E!E???+?=QRCSBU+U(;'>--m<  ]3  1]4D D  "#%"(    9+++%1   r1)rcrXr_rdrr) NNNNNNNNNN) rBrCrDrEr%rkr@rFrGrHs@r/r\r\\s9, + "#![ [ r1r\c^\rSrSrS/rSSS.rS\4U4SjjrSrSr \ SS \ RS -S \ RS -S \ RS -S \ RS -S\\\ RS -S\S -S\ R"S -S\ R"S -S\S -S\S -S\S -S\S -S\\ R\-4Sjj5rSrU=r$)r:iFdecoder.block.0.layer.1.EncDecAttention.relative_attention_bias.weight shared.weightencoder.embed_tokens.weightdecoder.embed_tokens.weightrLc>[TU]U5 [R"URUR 5Ul[R"U5nSUl SUl [U5Ul [R"U5nSUl URUl[U5UlUR!5 gNFT)r$r%rrr^rRr?copydeepcopyrrr\encodernum_decoder_layersradecoderrer+rLencoder_configdecoder_configr.s r/r%T5Model.__init__s  ll6#4#4fnnE v.$)!#( ~. v.$(!$*$=$=!~.  r1cUR$r^r?r+s r/get_input_embeddingsT5Model.get_input_embeddings) {{r1c|XlURRU5 URRU5 gr^r?rrkrris r/rkT5Model.set_input_embeddings,+$  )).9 )).9r1Nr+rr*r,encoder_outputsrrpdecoder_inputs_embedsrrryrr!c RU bU OURRn U bU OURRn UcURUUUU U U S9nORU (aK[ U[ 5(d6[ US[ U5S:aUSOS[ U5S:aUSOSS9nUSnURUUUUUUU U U U S9 nU (dX-$[URURURURURURURURS9$) a> input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. T5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [T5 Training](./t5#training). decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) T5 uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). To know more on how to prepare `decoder_input_ids` for pretraining take a look at [T5 Training](./t5#training). decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. Example: ```python >>> from transformers import AutoTokenizer, T5Model >>> tokenizer = AutoTokenizer.from_pretrained("google-t5/t5-small") >>> model = T5Model.from_pretrained("google-t5/t5-small") >>> input_ids = tokenizer( ... "Studies have been shown that owning a dog is good for you", return_tensors="pt" ... ).input_ids # Batch size 1 >>> decoder_input_ids = tokenizer("Studies show that", return_tensors="pt").input_ids # Batch size 1 >>> # preprocess: Prepend decoder_input_ids with start token which is pad token for T5Model. >>> # This is not needed for torch's T5ForConditionalGeneration as it does this internally using labels arg. >>> decoder_input_ids = model._shift_right(decoder_input_ids) >>> # forward pass >>> outputs = model(input_ids=input_ids, decoder_input_ids=decoder_input_ids) >>> last_hidden_states = outputs.last_hidden_state ```Nr+rrprryrrrr3rvr>rw r+rrprrrrrryr)rvrdecoder_hidden_statesdecoder_attentionsrxencoder_last_hidden_staterencoder_attentions)rLrrrr_rlenrrrvrr>rwrx)r+r+rr*r,rrrprrrryrrr>decoder_outputss r/r@T5Model.forward1sPD"+!6IDKKBF(,-159!%)-,0#'q ##d*q ))D0q !++d2 q !& 0 04 7 q uU%6%6784? q q ||d*q  %||d2q $;q  $;q #Tkq D[q  u  !$6 6q q r1r:z: T5 Model with a `language modeling` head on top. ) custom_introc^\rSrSrS/rSSSS.rS\4U4SjjrSrSr \ SS \ RS -S \ RS -S \ RS -S \ RS -S\\\ R S -S\S -S\ RS -S\ RS -S\ RS -S\S -S\S -S\S -S\S -S\\ R\-4Sjj5rS\ R 4SjrSrU=r$)r;irr)zlm_head.weightrrrLc>[TU]U5 URUl[R "UR UR5Ul[R"U5nSUl SUl [U5Ul [R"U5nSUl URUl[U5Ul[R""URUR SS9UlUR'5 g)NFTrO)r$r%rR model_dimrrr^r?rrrrr\rrrarrQr3rers r/r%#T5ForConditionalGeneration.__init__s  ll6#4#4fnnE v.$)!#( ~. v.$(!$*$=$=!~. yy1B1BO  r1cUR$r^rrs r/r/T5ForConditionalGeneration.get_input_embeddingsrr1c|XlURRU5 URRU5 gr^rris r/rk/T5ForConditionalGeneration.set_input_embeddingsrr1Nr+rr*r,rrrprlabelsrrryrr!c U bU OURRn U bU OURRn UcURUUUU U U S9nORU (aK[ U[ 5(d6[ US[ U5S:aUSOS[ U5S:aUSOSS9nUSnU bUcUcURU 5nURUUUUUUU U U U S9 nUSnURR(aUURS--nURU5nSnU b[[S S 9nU RUR5n U"URS UR!S 55U RS 55nU (dU4USS-U-nUbU4U-$U$[#UUUR$UR&UR(UR*UR,UR&UR(S 9 $) a input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. T5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [T5 Training](./t5#training). decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) T5 uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). To know more on how to prepare `decoder_input_ids` for pretraining take a look at [T5 Training](./t5#training). decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[-100, 0, ..., config.vocab_size - 1]`. All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` Examples: ```python >>> from transformers import AutoTokenizer, T5ForConditionalGeneration >>> tokenizer = AutoTokenizer.from_pretrained("google-t5/t5-small") >>> model = T5ForConditionalGeneration.from_pretrained("google-t5/t5-small") >>> # training >>> input_ids = tokenizer("The walks in park", return_tensors="pt").input_ids >>> labels = tokenizer(" cute dog the ", return_tensors="pt").input_ids >>> outputs = model(input_ids=input_ids, labels=labels) >>> loss = outputs.loss >>> logits = outputs.logits >>> # inference >>> input_ids = tokenizer( ... "summarize: studies have shown that owning a dog is good for you", return_tensors="pt" ... ).input_ids # Batch size 1 >>> outputs = model.generate(input_ids) >>> print(tokenizer.decode(outputs[0], skip_special_tokens=True)) >>> # studies have shown that owning a dog is good for you. ```Nrrrr3rrr5rI ignore_indexr4 losslogitsrrrrxrrr)rLrrrr_rrrPrscale_decoder_outputsrr3rr6rrrzrrr>rwrxrv)r+r+rr*r,rrrprrrrryrrr>rsequence_output lm_logitsrloss_fctoutputs r/r@"T5ForConditionalGeneration.forwards R"+!6IDKK=A(,26:>*.!%)-,0#'M ##d*M ))D0M !++d2 M !& 0 04 7 M uU\\23d: M M ((4/M  %0047M   4'M $;M  $;M #TkM D[M u  !O 3!M M ^)ELL))r1r;c ^\rSrSrSS0rS/rS\4U4SjjrSrSr \ SS \ RS -S \ RS -S \ RS -S \S -S\S -S\S -S\\ R\-4Sjj5rSrU=r$)r<ifrrrrLc>[TU]U5 [R"URUR 5UlUnSUlSUl[U5Ul UR5 g)NF) r$r%rrr^rRr?rr{r\rre)r+rLrr.s r/r%T5EncoderModel.__init__ksY  ll6#4#4fnnE #( ,1)~.  r1cUR$r^rrs r/r#T5EncoderModel.get_input_embeddingswrr1cFXlURRU5 gr^)r?rrkris r/rk#T5EncoderModel.set_input_embeddingszs$  )).9r1Nr+rrprryrr!c dUbUOURRnURUUUUUUS9nU$)a input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. T5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. To know more on how to prepare `input_ids` for pretraining take a look a [T5 Training](./t5#training). Example: ```python >>> from transformers import AutoTokenizer, T5EncoderModel >>> tokenizer = AutoTokenizer.from_pretrained("google-t5/t5-small") >>> model = T5EncoderModel.from_pretrained("google-t5/t5-small") >>> input_ids = tokenizer( ... "Studies have been shown that owning a dog is good for you", return_tensors="pt" ... ).input_ids # Batch size 1 >>> outputs = model(input_ids=input_ids) >>> last_hidden_states = outputs.last_hidden_state ```r)rLrr) r+r+rrprryrrrs r/r@T5EncoderModel.forward~sHD&1%[TU]U5 [U5Ul[ U5UlUR 5 gr^)r$r%r:r(rclassification_headrer[s r/r%$T5ForSequenceClassification.__init__s5  "6?#7#?  r1Nr+rr*r,rrprrrrryrr!c  4U bU OURRn UbSn Uc%Ub"[SURR35eUc"UcUc [ S5eUR U5nURUUUUUUUU U U U S9 nUSnURURR5RUR5n[[R"URS55R!5S:HS5 UR"unnnUUSS24R%US U5SS2S SS24nUR'U5nSnUGbURUR5nURR(cURR*S:XaS URlOyURR*S:aNUR,[R.:XdUR,[R0:XaS URlOS URlURR(S :XaT[35nURR*S:Xa&U"UR55UR555nOU"UU5nOURR(S :XaG[75nU"UR%S URR*5UR%S 55nO-URR(S :Xa[95nU"UU5nU (dU4USS-nUbU4U-$U$[;UUUR<UR>UR@URBURDURFURHS 9 $)a input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. T5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [T5 Training](./t5#training). decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) T5 uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). To know more on how to prepare `decoder_input_ids` for pretraining take a look at [T5 Training](./t5#training). decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). NFz8Passing input embeddings is currently not supported for If no `decoder_input_ids` or `decoder_inputs_embeds` are passed, `input_ids` cannot be `None`. Please pass either `input_ids` or `decoder_input_ids` or `decoder_inputs_embeds`.) rr*r,rrprrrryrrrz7All examples must have the same number of tokens.r4 regressionsingle_label_classificationmulti_label_classificationr)%rLrNotImplementedErrorr.rBrLrPr(eq eos_token_idr6rrr'unique_consecutivesumnumelrrr problem_typerr;rrrsqueezerrrrrrrxrrr)r+r+rr*r,rrprrrrryrrrreos_maskr~_r,sentence_representationrrrrs r/r@#T5ForSequenceClassification.forwardsN`&1%>KbKbJcd   $)>)F  U !% 1 1) < "" )/#9+'"7/!5##  "!*<< 8 89<<_=S=ST  $ $X\\!_ 5 ; ; = B E &5%:%:" A{"1(A+">"C"CJPRT_"`abdfhiai"j))*AB  YYv}}-F{{''/;;))Q./;DKK,[[++a/V\\UZZ5OSYS_S_chclclSl/LDKK,/KDKK,{{''<7"9;;))Q.#FNN$4fnn6FGD#FF3D))-JJ+- B 0F0F GUWY))-II,./Y,F)-)9TGf$ Ev E.#33")"?"?&99$55&-&G&G")"?"?&99  r1)rr(r)rBrCrDrErrr%rr'rr`listrrbr|rr@rFrGrHs@r/rrsU+s)s&x.2.259:>:>26:>*.!%)-,0#'A ##d*A  t+A !++d2 A !& 0 04 7 A e//047 A ((4/A  %0047A   4'A $;A  $;A #TkA D[A  0 0A A r1rc^\rSrSrS\4U4Sjjr\SS\RS-S\RS-S\RS-S\RS-S \ S-S \ S-S \ S-S \ \R\ -4S jj5r Sr U=r$)rCiDrLc0>[TU]U5 URUl[U5Ul[ R "UR5Ul[ R"URUR5Ul UR5 gr^) r$r%rr<r(rrVrrXrQr,r6rer[s r/r%!T5ForTokenClassification.__init__Fsj   ++)&1zz&";";< ))F$6$68I8IJ r1Nr+rrprrryrr!c UbUOURRnURUUUUUUS9n U Sn URU 5n UR U 5n Sn Ub<[ 5n U "U R SUR5UR S55n U(dXSS4nU bU 4U-$U$[U U U RU RS9$)a input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. T5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [T5 Training](./t5#training). labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. N)rrprryrrr4r3)rrr>rw) rLrr(rXr6rrrrr>rw)r+r+rrprrryrrrr>rrrrs r/r@ T5ForTokenClassification.forwardQs4&1%[TU]U5 URUl[R "UR UR5Ul[R"U5nSUl SUl [U5Ul [R"U5nSUl URUl[U5UlUR"Ul[R$"UR&UR"5UlUR+5 gr)r$r%rRrrrr^r?rrrrr\rrrarrrQr,r4rers r/r%T5ForQuestionAnswering.__init__s  ll6#4#4fnnE v.$)!#( ~. v.$(!$*$=$=!~.  ++))F$6$68I8IJ r1cUR$r^rrs r/r+T5ForQuestionAnswering.get_input_embeddingsrr1c|XlURRU5 URRU5 gr^rris r/rk+T5ForQuestionAnswering.set_input_embeddingsrr1Nr+rr*r,rstart_positions end_positionsrprrrryrr!c U bU OURRn U bU OURRn UbUbSn Uc"U cUc [S5eUR U5nU bU OURRn U bU OURRn UcUR UUUU U U S9nORU (aK[ U[5(d6[US[U5S:aUSOS[U5S:aUSOSS9nUSnURUUU SUUU U U U S 9 nUSnURU5nURSS S 9unnURS 5R5nURS 5R5nSnUbUb[UR55S:a*URS 5RUR 5n[UR55S:a*URS 5RUR 5nURS5nUR#SU5nUR#SU5n[%US 9nU"UU5nU"UU5nUU-S- nU (dUU4USS-U-nUbU4U-$U$['UUUUR(UR*UR,UR.UR0UR*UR,S 9 $)a input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. T5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [T5 Training](./t5#training). decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) T5 uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). To know more on how to prepare `decoder_input_ids` for pretraining take a look at [T5 Training](./t5#training). decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. NFrrrrr3rrr4rr) r start_logits end_logitsrrrrxrrr)rLrrrLrPrr_rrrr4splitrrrzr6rrrrrr>rwrxrv)r+r+rr*r,rrrrprrrryrrr>rrrrr total_loss ignored_indexr start_lossend_lossrs r/r@T5ForQuestionAnswering.forwards3\&1%)F  U !% 1 1) < !*!6IDKK=A371526:>!%)-,0#'I ##d*I ))D0I !++d2 I !& 0 04 7 I uU\\23d: I ))D0I ''$.I ((4/I  %0047I $;I  $;I #TkI D[I u  !$G G!I I r1r=)r<r;r:r'r=rrC)Ar%rrr'rtorch.nnrrrrorr8 activationsr cache_utilsr r r generationr masking_utilsrrmodeling_layersrmodeling_outputsrrrrrrrmodeling_utilsrutilsrrrrrconfiguration_t5r get_loggerrBrModuler!rJrfrqr~rrrrr'r\r:r;r<rrCr=__all__rRr1r/rs AA&!CC)J9.^^&   H %+"))+2bii,299: $"))D299BBII>W (W t299$^!^!^!Bo o dT T T n x)!2Ox)  x)vD&DDN M "3M  M `C 0C C Lo .o o d r1