Zj~!SrSSKrSSKrSSKrSSKJr SSKJrJrJr SSK J r SSK J r SSKJrJrJr SS KJr SS KJr SS KJr SS KJrJrJrJrJrJrJr SS K J!r! SSK"J#r#J$r$J%r%J&r&J'r'J(r( SSK)J*r* \'RV"\,5r-"SS\R\5r/"SS\R\5r0"SS\R\5r1"SS\R\5r2"SS\R\5r3"SS\R\5r4"SS\R\5r5"SS \5r6"S!S"\R\5r7\%"S#S$\!55r8"S%S&\85r9\%"S'S(\855r:\%"S)S*9"S+S,\8\55r;\%"S-S.\855r<\%"S/S*9"S0S1\855r=\%"S2S3\855r>\%"S4S5\855r?/S6Qr@g)7zPyTorch UMT5 model.N)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)ACT2FN)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)create_causal_mask)GradientCheckpointingLayer)BaseModelOutput)BaseModelOutputWithPastAndCrossAttentionsSeq2SeqLMOutputSeq2SeqModelOutput#Seq2SeqQuestionAnsweringModelOutputSeq2SeqSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel) DUMMY_INPUTS DUMMY_MASKauto_docstringis_torchdynamo_compilingloggingtorch_compilable_check) UMT5Configc2^\rSrSrSU4SjjrSrSrU=r$) UMT5LayerNorm6c>[TU]5 [R"[R "U55UlX lg)zU Construct a layernorm module in the UMT5 style. No bias and no subtraction of mean. N)super__init__r Parametertorchonesweightvariance_epsilon)self hidden_sizeeps __class__s w/root/GenerationalWealth/GenerationalWealth/venv/lib/python3.13/site-packages/transformers/models/umt5/modeling_umt5.pyr%UMT5LayerNorm.__init__7s/ ll5::k#:; #cUR[R5RS5R SSS9nU[R "X R -5-nURR[R[R4;a%URURR5nURU-$)NT)keepdim) tor'float32powmeanrsqrtr*r)dtypefloat16bfloat16)r+ hidden_statesvariances r/forwardUMT5LayerNorm.forward?s !##EMM266q9>>r4>P% H?T?T4T(UU  ;;   ? ?),,T[[->->?M{{]**r1)r*r))gư>)__name__ __module__ __qualname____firstlineno__r%r@__static_attributes__ __classcell__r.s@r/r!r!6s$ + +r1r!c6^\rSrSrS\4U4SjjrSrSrU=r$)UMT5DenseActDensePconfigcX>[TU]5 [R"URUR SS9Ul[R"UR URSS9Ul[R"UR5Ul [URUl gNFbias)r$r%rLineard_modeld_ffwiwoDropout dropout_ratedropoutr dense_act_fnactr+rLr.s r/r%UMT5DenseActDense.__init__Qsn ))FNNFKKeD))FKKeDzz&"5"56 &--.r1cURU5nURU5nURU5n[URR [ R5(aURURR R:waaURR R[ R:wa/URURR R5nUR U5nU$N) rTrZrX isinstancerUr)r'Tensorr;int8r6r+r>s r/r@UMT5DenseActDense.forwardXs . /  ]3 tww~~u|| 4 4##tww~~';';;$$ 2),,TWW^^-A-ABM . r1)rZrXrTrU rBrCrDrErr%r@rFrGrHs@r/rJrJPs/z/  r1rJc6^\rSrSrS\4U4SjjrSrSrU=r$)UMT5DenseGatedActDensegrLc>[TU]5 [R"URUR SS9Ul[R"URUR SS9Ul[R"UR URSS9Ul[R"UR5Ul [URUlgrN)r$r%rrQrRrSwi_0wi_1rUrVrWrXr rYrZr[s r/r%UMT5DenseGatedActDense.__init__hs IIfnnfkkF IIfnnfkkF ))FKKeDzz&"5"56 &--.r1c8URURU55nURU5nX#-nURU5n[ UR R [R5(aURUR R R:waaUR R R[R:wa/URUR R R5nUR U5nU$r^) rZrirjrXr_rUr)r'r`r;rar6)r+r> hidden_gelu hidden_linears r/r@UMT5DenseGatedActDense.forwardpshhtyy78  -0 #3  ]3 tww~~u|| 4 4##tww~~';';;$$ 2),,TWW^^-A-ABM . r1)rZrXrirjrUrdrHs@r/rfrfgs/z/r1rfc6^\rSrSrS\4U4SjjrSrSrU=r$) UMT5LayerFFrLc>[TU]5 UR(a[U5UlO[ U5Ul[ URURS9Ul [R"UR5Ul g)Nr-)r$r% is_gated_actrfDenseReluDenserJr!rRlayer_norm_epsilon layer_normrrVrWrXr[s r/r%UMT5LayerFF.__init__s_    "8"@D "3F";D 'Fforwarded_statess r/r@UMT5LayerFF.forwards;??=9../?@% 5E(FF r1)rvrXrxrdrHs@r/rqrqs7z7r1rqc ^\rSrSrSrSS\S-4U4SjjjrS\RS\R4Sjr S r SS jr SS \RS \RS-S \ S-S\RS-4Sjjr SrU=r$) UMT5Attentionz/ T5's attention using relative_attention_bias. N layer_idxc>[TU]5 URUlX lURUlUR UlUR UlURUlURUl URUl URUR-Ul X0lUc>UR(a-[R!SUR"R$S35 [&R("UR URSS9Ul[&R("UR URSS9Ul[&R("UR URSS9Ul[&R("URUR SS9UlUR(a1[&R2"URUR5Ulgg)NzInstantiating a decoder z without passing `layer_idx` is not recommended and will to errors during the forward call, if caching is used. Please make sure to provide a `layer_idx` when creating this class.FrO)r$r% is_decoderhas_relative_attention_biasrelative_attention_num_bucketsrelative_attention_max_distancerRd_kvkey_value_proj_dim num_headsn_headsrWrX inner_dimrlogger warning_oncer.rBrrQqkvo Embeddingrelative_attention_bias)r+rLrrr.s r/r%UMT5Attention.__init__s`  +++F(.4.S.S+/5/U/U,~~ "(++'' ** (?(??"     *4>>+B+B*CD,,  4<<eD4<<eD4<<eD4>>4<FFq!QPQRr1cSnURnURnUR(dCUS-nX!S:R[R 5U-- n[R "U5nO,[R"U[R"U55*nUS-nX:n[R"UR5U- 5[R"XE- 5- nXsU- -nXWR[R 5-n[R"U[R"XS- 55nU[R"XaU5- nU$)aR Adapted from Mesh Tensorflow: https://github.com/tensorflow/mesh/blob/0cb87fe07da627bf0b7e60475d59f95ed6b5be3d/mesh_tensorflow/transformer/transformer_layers.py#L593 Translate relative position to a bucket number for relative attention. The relative position is defined as memory_position - query_position, i.e. the distance in tokens from the attending position to the attended-to position. If bidirectional=False, then positive relative positions are invalid. We use smaller buckets for small absolute relative_position and larger buckets for larger absolute relative_positions. All relative positions >=max_distance map to the same bucket. All relative positions <=-max_distance map to the same bucket. This should allow for more graceful generalization to longer sequences than the model has been trained on Args: relative_position: an int32 Tensor bidirectional: a boolean - whether the attention is bidirectional num_buckets: an integer max_distance: an integer Returns: a Tensor with the same shape as relative_position, containing int32 values in the range [0, num_buckets) rr3r)rrrr6r'longabsmin zeros_likelogfloatmath full_likewhere) r+relative_positionrelative_buckets num_buckets max_distance max_exactis_small log_ratiorelative_position_if_larges r/_relative_position_bucket'UMT5Attention._relative_position_buckets/*99 ;;  A K Q!6 : :5:: F T T  % *; < !&+N>NO`>a!b b  1$ $0II/557)CDtxxP\PhGii y!89 %.ejj1I%I"%*YY &8RbcTc(d& " EKKE_``r1cUc URRRn[R"U[R US9SS2S4U-n[R"U[R US9SSS24nXe- nUR U5nURU5n U R/SQ5RS5n U $)z%Compute binned relative position biasN)r;device)r3rrr) rr)rr'arangerrr unsqueeze) r+ query_length key_lengthrpast_seen_tokenscontext_positionmemory_positionrrelative_position_bucketvaluess r/ compute_biasUMT5Attention.compute_biass >1188??F << EJJvVWXZ^W^_brr,,zFSTXZ[T[\+>#'#A#ABS#T --.FG *44Q7 r1r>encoder_hidden_statespast_key_valuesattention_maskc URSSupgUbURUR5OSn[U[R 5(aUR 5OUnUSLn URU5n U RUSURUR5RSS5n Sn Ub[[U[5(aFURRUR5n U (a URn OUR n OUn U (aUOUn U (aQUbNU (aGU R"URR$nU R"URR&nOUR)U 5nUR+U 5nURUSURUR5RSS5nURUSURUR5RSS5nUbSU R-XUR5upU (a.[U[5(aSURUR'[R."XRSS55nURSnUR0(d9[R2"SURUU4UR4UR6S 9nOUR9UUUR4US 9nUbUU-nUnUU- n[:R<R?URA5SS 9RCU5n[:R<REUURDURFS 9n[R."UU5nURSS5RI5nURXgS5nURKU5nUU4$) Nr3rr4rFTr)rr;)rrdim)ptraining)&shapeget_seq_lengthrr_r'r`clonerrrr transposer is_updatedgetcross_attention_cacheself_attention_cachelayerskeysrrrupdatematmulrzerosrr;rr functionalsoftmaxrtype_asrXr contiguousr)r+r>rrrkwargs batch_size seq_lengthris_cross_attention query_statesrcurr_past_key_valuescurrent_states key_states value_statesscoresr position_biasposition_bias_masked attn_weights attn_outputs r/r@UMT5Attention.forwardsU"/!4!4Ra!8 M\Mh?99$..Ino7ABRTYT`T`7a7a+113gw3$>vvm, #((RtG^G^_iijkmno   &:oGZ+[+[(3377GJ!'6'L'L$'6'K'K$#2 2D.- /"=*-44T^^DIIJ/66t~~FMML/J66.1L#RtG^G^_iijkmnoJ',,ZT\\4KbKbcmmnoqrsL*+?+F+Fzaeaoao+p( %*_FY*Z*ZAEO..t~~>l,@,@A,FG%%b) //!KKDLL*j9&--W]WcWcM!--Jv}}O_.M  %)N:M,&&}},,V\\^,DLLVT }},,\T\\TXTaTa,b ll<> !++Aq1<<> !&&zrB ff[) L((r1)rRrXrrrrrrrrrrrrr)FN)NrNNN)rBrCrDrE__doc__intr%r'r`rrrr r@rFrGrHs@r/r~r~skSSWZkk6%,, - ^ 6:(,.2 N)||N) %||d2N) N)  t+ N)N)r1r~cH^\rSrSrSS\S-4U4SjjjrSSjrSrU=r$) UMT5LayerSelfAttentioniINrc>[TU]5 [USUS9Ul[ UR UR S9Ul[R"UR5Ul g)NTrrrt) r$r%r~ SelfAttentionr!rRrwrxrrVrWrXr+rLrr.s r/r%UMT5LayerSelfAttention.__init__JsN *6t_hi'Frrrnormed_hidden_statesattention_outputoutputss r/r@UMT5LayerSelfAttention.forwardPse $}=-- )+.  & 5Ea5H(II  "%5ab%99r1)rrXrxr^)NN rBrCrDrErr%r@rFrGrHs@r/rrIs*7#*77 r1rcJ^\rSrSrSS\S-4U4SjjjrSSjrSrU=r$) UMT5LayerCrossAttentionibNrc>[TU]5 [USUS9Ul[ UR UR S9Ul[R"UR5Ul g)NFrrt) r$r%r~EncDecAttentionr!rRrwrxrrVrWrXrs r/r% UMT5LayerCrossAttention.__init__csO ,VQVbkl'Frrrrrr layer_outputrs r/r@UMT5LayerCrossAttention.forwardisg $}=// "7)+ 0  %||4DQ4G'HH /$4QR$88r1)rrXrxr^rrrHs@r/rrbs-7#*77# r1rcP^\rSrSrSS\S-4U4SjjjrSSjrSrU=r$) UMT5Blocki}Nrcj>[TU]5 URUl[R"5UlUR R [XS95 UR(a"UR R [XS95 UR R [U55 g)Nr) r$r%rr ModuleListlayerappendrrrqrs r/r%UMT5Block.__init__~sv  ++]]_  0MN ?? JJ  5fR S +f-.r1c URS"UUUS9upUR[R:Xa}[R"UR5R n [R "[R"U5R5U S- U 5n [R"X*U S9nSn UR=(a USLn U (aURS"UUUUS9upUR[R:Xa}[R"UR5R n [R "[R"U5R5U S- U 5n [R"X*U S9nURS"U5nUR[R:Xa}[R"UR5R n [R "[R"U5R5U S- U 5n [R"X*U S9nU4nU(aXU 4- nU$)Nrri)rmaxrrr4) rr;r'r<finfor risinfanyclampr)r+r>rrencoder_attention_maskr use_cacheoutput_attentionsrself_attn_weights max_dtype clamp_valuecross_attn_weightsdo_cross_attentionrs r/r@UMT5Block.forwards,0::a= )+, (    %-- / M$7$78< >Gr1)rrr^)NNNNFFrrHs@r/rr}s6/#*//"#55r1rcn^\rSrSrSrS\4U4SjjrS\RS\R4Sjr Sr U=r $) UMT5ClassificationHeadiz-Head for sentence-level classification tasks.rLc(>[TU]5 [R"URUR5Ul[R "URS9Ul[R"URUR5Ul g)N)r) r$r%rrQrRdenserVclassifier_dropoutrX num_labelsout_projr[s r/r%UMT5ClassificationHead.__init__sZ YYv~~v~~> zzF$=$=>  &..&2C2CD r1r>rcURU5nURU5n[R"U5nURU5nUR U5nU$r^)rXrr'tanhrrbs r/r@UMT5ClassificationHead.forwardsN ]3  =1  =1  ]3  m4 r1)rrXr) rBrCrDrErrr%r'r`r@rFrGrHs@r/rrs47EzE U\\ellr1rc|\rSrSr%\\S'SrSrSrS/r S/r \ S5r \ R"5S5rS rS rg ) UMT5PreTrainedModelirL transformerTrrUcz[R"[5n[R"[5nUUUS.nU$)N)decoder_input_ids input_idsdecoder_attention_mask)r'tensorrr)r+r) input_mask dummy_inputss r/r- UMT5PreTrainedModel.dummy_inputss6LL. \\*- !*"&0 r1cURRn[U[5(a%[R "UR US-5 g [U[[[[45(Ga[R"URR SUS-S9 [US5(aHURR(d-[R"URR SUS-S9 [US5(an[R"UR R SX RR"S--S9 [R$"UR R&5 g g [U[(5(aj[US5(aX[R"UR*R SUS-S9 [R$"UR*R&5 g g [U[,5(GaA[R"UR.R SX RR"S--S9 [UR.S5(aAUR.R&b*[R$"UR.R&5 [R"UR0R SX RR"S--S9 [UR0S5(aCUR0R&b+[R$"UR0R&5 g g g [U[25(GaA[R"UR4R SX RR"S--S9 [UR4S5(aAUR4R&b*[R$"UR4R&5 [R"UR6R SX RR8S--S9 [UR6S5(aCUR6R&b+[R$"UR6R&5 g g g [U[:5(Ga[R"UR<R SX RR"S--S9 [UR<S5(aAUR<R&b*[R$"UR<R&5 [R"UR>R SX RR"S--S9 [UR>S5(aAUR>R&b*[R$"UR>R&5 [R"UR6R SX RR8S--S9 [UR6S5(aCUR6R&b+[R$"UR6R&5 g g g [U[@5(GaFURR"nURRBnURRDn[R"URFR SX#U-S--S9 [R"URHR SX#S--S9 [R"URJR SX#S--S9 [R"URLR SX%U-S--S9 URN(a0[R"URPR SX#S--S9 g g g ) zInitialize the weights?g)r9stdlm_head qa_outputs classifierrPN))rLinitializer_factorr_r!init constant_r) UMT5ModelUMT5ForConditionalGenerationUMT5EncoderModelUMT5ForQuestionAnsweringnormal_sharedhasattrtie_word_embeddingsr2r3rRzeros_rPUMT5ForTokenClassificationr5rrrrJrTrUrSrfrirjr~rrrrrrrr)r+modulefactorrRrrs r/ _init_weights!UMT5PreTrainedModel._init_weightssW// fm , , NN6==&3, 7  , (     LL--CVc\ Jvy))$++2Q2Q V^^22&3,Ov|,, V..55CVP[P[PcPchlOlEmn F--223- : ; ;v|,, V..55CVc\R F--223- 6 7 7 LL,,3F{{GZGZ_cFc%>%@#qr'"$:&!  PQ Q&&'8D'@,O  r1N)rBrCrDrEr__annotations__base_model_prefixsupports_gradient_checkpointing_can_compile_fullgraph_no_split_modules_keep_in_fp32_modulespropertyr-r'no_gradrErOrFrQr1r/r%r%s` %&*#!$ !F  ]]_@p@pD!r1r%cL^\rSrSrU4SjrSrSSjrSrU=r$) UMT5StackiAc >[TU]U5 [R"URUR 5UlURUl[R"[UR5Vs/sH n[XS9PM sn5Ul [UR URS9Ul[R "UR"5UlSUlUR)5 gs snf)NrrtF)r$r%rr vocab_sizerR embed_tokensrrrange num_layersrblockr!rwfinal_layer_normrVrWrXgradient_checkpointing post_init)r+rLir.s r/r%UMT5Stack.__init__Bs  LL):):FNNK ++]]ERXRcRcLd#eLdqIf$BLd#ef -fnn&B[B[ \zz&"5"56 ',#  $fs9C8cXlgr^)r^r+new_embeddingss r/set_input_embeddingsUMT5Stack.set_input_embeddingsNs*r1c UbUOURRnUbUOURRnU bU OURRn U bU OURRn Ub*Ub'UR (aSOSn [ SU SU S35eUb&UR5n URSU S5nO>UbUR5SSn O'UR (aSOSn [ SU SU S 35eUR(a/UR(aU(a[RS 5 S nUc)URc [ S 5eURU5nU upUS La UR (d[ SUS35eUR (ahU(a`Uc]URR(a/[[!URS9[!URS95nO'[!URS9nOUR (dSnUbUR#5OSnUc4[%5(d%UU-n[&R("UUUR*S9nUR (a[-URUUUS9nO\UbWUSS2SSSS24nUR/UR0S9nSU- [&R2"UR05R4-nOSnUR (aOUbLUR5unnnUU4nUc[&R("UUR*S9nUR7U5nOSnU (aSOSnU(aSOSnU(aUR (aSOSnUR9U5nR<5HRunnU (aUU4-nU"UUUUUUUS9nUSnU(dM-UUS4- nUR (dMIUUS4- nMT UR?U5nUR9U5nU (aUU4-nU (d[ASUUUUU455$[CUUUUUS9$)Ndecoder_zYou cannot specify both zinput_ids and zinputs_embeds at the same timer4zYou have to specify either z input_ids or inputs_embedszZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...Fz$UMT5Stack.forward..s" As )last_hidden_staterr> attentionscross_attentions)"rLrroutput_hidden_states return_dictrrKrrrcrrrr^is_encoder_decoderr r rrr'r(rrr6r;r rinvert_attention_maskrX enumeraterarbtupler)r+r)rrrrorrrrwrxrerr_msg_prefix input_shaperrpast_key_values_lengthmask_seq_length causal_maskencoder_batch_sizeencoder_sequence_length_encoder_hidden_shapeencoder_extended_attention_maskall_hidden_statesall_attentionsall_cross_attentionsr>re layer_module layer_outputss r/r@UMT5Stack.forwardQsY"+!6IDKK+/??ZN*>*:.HXXvw  "#..*K!r;r?;I  &',,.s3K+/??ZN:>:J-XfWggtuv v  & &4==##p"    ( !_`` --i8M!,  ?? #LTFRg!hii ??_4;;11&9$DKK8,dkk:Z'O'3$++&FO#OETE`!?!?!Afg  !*B*D*D4zAO"ZZ OML`L`aN ??,{{+- / K  '(D$)9:K%..}/B/B.CK, M>> from transformers import UMT5Model, AutoTokenizer >>> model = UMT5Model.from_pretrained("google/umt5-small") >>> tokenizer = AutoTokenizer.from_pretrained("google/umt5-small") >>> noisy_text = "UN Offizier sagt, dass weiter werden muss in Syrien." >>> label = " verhandelt" >>> inputs = tokenizer(inputs, return_tensors="pt") >>> labels = tokenizer(label=label, return_tensors="pt") >>> outputs = model(input_ids=inputs["input_ids"], decoder_input_ids=labels["input_ids"]) >>> hidden_states = outputs.last_hidden_state ```umt5rL shared.weightencoder.embed_tokens.weightdecoder.embed_tokens.weightc>[TU]U5 [R"URUR 5Ul[R"U5nSUl SUl [U5Ul [R"U5nSUl URUl[U5UlUR!5 gNFT)r$r%rrr]rRr>copydeepcopyrrr[encodernum_decoder_layersr`decoderrdr+rLencoder_configdecoder_configr.s r/r%UMT5Model.__init__s  ll6#4#4fnnE v.$)!#(  0 v.$(!$*$=$=! 0  r1cUR$r^r>r+s r/get_input_embeddingsUMT5Model.get_input_embeddings {{r1c|XlURRU5 URRU5 gr^r>rrjrrhs r/rjUMT5Model.set_input_embeddings+$  )).9 )).9r1Nr)rr(r*encoder_outputsrrodecoder_inputs_embedsrrrwrxrc RU bU OURRn U bU OURRn UcURUUUU U U S9nORU (aK[ U[ 5(d6[ US[ U5S:aUSOS[ U5S:aUSOSS9nUSnURUUUUUUU U U U S9 nU (dX-$[URURURURURURURURS9$) aP input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. UMT5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [UMT5 Training](./umt5#training). decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) UMT5 uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). To know more on how to prepare `decoder_input_ids` for pretraining take a look at [UMT5 Training](./umt5#training). decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. Example: ```python >>> from transformers import AutoTokenizer, UMT5Model >>> tokenizer = AutoTokenizer.from_pretrained("google/umt5-small") >>> model = UMT5Model.from_pretrained("google/umt5-small") >>> input_ids = tokenizer( ... "Studies have been shown that owning a dog is good for you", return_tensors="pt" ... ).input_ids # Batch size 1 >>> decoder_input_ids = tokenizer("Studies show that", return_tensors="pt").input_ids # Batch size 1 >>> # preprocess: Prepend decoder_input_ids with start token which is pad token for UMT5Model. >>> # This is not needed for torch's UMT5ForConditionalGeneration as it does this internally using labels arg. >>> decoder_input_ids = model._shift_right(decoder_input_ids) >>> # forward pass >>> outputs = model(input_ids=input_ids, decoder_input_ids=decoder_input_ids) >>> last_hidden_states = outputs.last_hidden_state ```Nr)rrorrwrxrrr3rtr>ru r)rrorrrrrrwrx)rtrdecoder_hidden_statesdecoder_attentionsrvencoder_last_hidden_staterencoder_attentions)rLrrxrr_rlenrrrtrr>rurv)r+r)rr(r*rrrorrrrwrxrr>decoder_outputss r/r@UMT5Model.forwardsPD"+!6IDKK NNNNNNNNNNNN)rBrCrDrEr model_typerrR_tied_weights_keysr%rrjrr' LongTensor FloatTensor BoolTensorr|r r`boolrr@rFrGrHs@r/r9r9ss "J '6'6 $: .23759:>BF(,-159!%)-,0#'q ##d*q ))D0q !++d2 q !& 0 04 7 q uU%6%6784? q q ||d*q  %||d2q $;q  $;q #Tkq D[q  u  !$6 6q q r1r9z< UMT5 Model with a `language modeling` head on top. ) custom_introc^\rSrSrSrSrSSSS.rU4SjrSrSr \ SS \ RS -S \ RS -S \ RS -S \ RS -S\\\ R S -S\S -S\ RS -S\ RS -S\ RS -S\S -S\S -S\S -S\S -S\\ R\-4Sjj5rS\ R 4SjrSrU=r$)r:ia Examples: ```python >>> from transformers import UMT5ForConditionalGeneration, AutoTokenizer >>> model = UMT5ForConditionalGeneration.from_pretrained("google/umt5-small") >>> tokenizer = AutoTokenizer.from_pretrained("google/umt5-small") >>> article = "UN Offizier sagt, dass weiter verhandelt werden muss in Syrien." >>> summary = "Weiter Verhandlung in Syrien." >>> inputs = tokenizer(article, text_target=summary, return_tensors="pt") >>> outputs = model(**inputs) >>> loss = outputs.loss ```rr)rrzlm_head.weightc>[TU]U5 URUl[R "UR UR5Ul[R"U5nSUl SUl [U5Ul [R"U5nSUl URUl[U5Ul[R""URUR SS9UlUR'5 g)NFTrO)r$r%rR model_dimrrr]r>rrrrr[rrr`rrQr2rdrs r/r%%UMT5ForConditionalGeneration.__init__s  ll6#4#4fnnE v.$)!#(  0 v.$(!$*$=$=! 0 yy1B1BO  r1cUR$r^rrs r/r1UMT5ForConditionalGeneration.get_input_embeddingsrr1c|XlURRU5 URRU5 gr^rrhs r/rj1UMT5ForConditionalGeneration.set_input_embeddingsrr1Nr)rr(r*rrrorlabelsrrrwrxrc U bU OURRn U bU OURRn UcURUUUU U U S9nORU (aK[ U[ 5(d6[ US[ U5S:aUSOS[ U5S:aUSOSS9nUSnU bUcUcURU 5nURUUUUUUU U U U S9 nUSnURR(aUURS--nURU5nSnU b[[S S 9nU RUR5n U"URS UR!S 55U RS 55nU (dU4USS-U-nUbU4U-$U$[#UUUR$UR&UR(UR*UR,UR&UR(S 9 $) aP input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. UMT5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [UMT5 Training](./umt5#training). decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) UMT5 uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). To know more on how to prepare `decoder_input_ids` for pretraining take a look at [UMT5 Training](./umt5#training). decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[-100, 0, ..., config.vocab_size - 1]`. All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` Examples: ```python >>> from transformers import AutoTokenizer, UMT5ForConditionalGeneration >>> tokenizer = AutoTokenizer.from_pretrained("google/umt5-small") >>> model = UMT5ForConditionalGeneration.from_pretrained("google/umt5-small") >>> # training >>> input_ids = tokenizer("The walks in park", return_tensors="pt").input_ids >>> labels = tokenizer(" cute dog the ", return_tensors="pt").input_ids >>> outputs = model(input_ids=input_ids, labels=labels) >>> loss = outputs.loss >>> logits = outputs.logits >>> # inference >>> input_ids = tokenizer("Studies have shown that good for you", return_tensors="pt").input_ids >>> outputs = model.generate(input_ids) >>> tokenizer.decode(outputs[0], skip_special_tokens=True) ```Nrrrr3rrr4rH ignore_indexr4 losslogitsrrrrvrrr)rLrrxrr_rrrOrr@rr2rr6rrrrrr>rurvrt)r+r)rr(r*rrrorrrrrwrxrr>rsequence_output lm_logitsrloss_fctoutputs r/r@$UMT5ForConditionalGeneration.forwards"L"+!6IDKK NNNNNNNNNNNNN)rBrCrDrErrrr%rrjrr'rrrr|r`r rrr@rrFrGrHs@r/r:r:s  J'6'6) ,: .23759:>=A(,26:>*.!%)-,0#'L ##d*L ))D0L !++d2 L !& 0 04 7 L uU\\23d: L L ((4/L  %0047L   4'L $;L  $;L #TkL D[L u  !O 3!L L ^)ELL))r1r:c^\rSrSrSrSrSS0rU4SjrSrSr \ SS \ RS -S \ RS -S \ RS -S \S -S\S -S\S -S\\ R\-4Sjj5rSrU=r$)r;ida Examples: ```python >>> from transformers import UMT5EncoderModel, AutoTokenizer >>> model = UMT5EncoderModel.from_pretrained("google/umt5-small") >>> tokenizer = AutoTokenizer.from_pretrained("google/umt5-small") >>> article = "UN Offizier sagt, dass weiter verhandelt werden muss in Syrien." >>> input_ids = tokenizer(article, return_tensors="pt").input_ids >>> outputs = model(input_ids) >>> hidden_state = outputs.last_hidden_state ```rrrc >[TU]U5 [R"URUR 5Ul[R"U5nSUl SUl [U5Ul UR5 g)NF)r$r%rrr]rRr>rrrryr[rrd)r+rLrr.s r/r%UMT5EncoderModel.__init__zsb  ll6#4#4fnnE v.#( ,1) 0  r1cUR$r^rrs r/r%UMT5EncoderModel.get_input_embeddingsrr1cFXlURRU5 gr^)r>rrjrhs r/rj%UMT5EncoderModel.set_input_embeddingss$  )).9r1Nr)rrorrwrxrc dUbUOURRnURUUUUUUS9nU$)a input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. UMT5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. To know more on how to prepare `input_ids` for pretraining take a look a [UMT5 Training](./umt5#training). Example: ```python >>> from transformers import AutoTokenizer, UMT5EncoderModel >>> tokenizer = AutoTokenizer.from_pretrained("google/umt5-small") >>> model = UMT5EncoderModel.from_pretrained("google/umt5-small") >>> input_ids = tokenizer( ... "Studies have been shown that owning a dog is good for you", return_tensors="pt" ... ).input_ids # Batch size 1 >>> outputs = model(input_ids=input_ids) >>> last_hidden_states = outputs.last_hidden_state ```r)rLrxr) r+r)rrorrwrxrrs r/r@UMT5EncoderModel.forwardsHF&1%)NNNNNN)rBrCrDrErrrr%rrjrr'rrrr|rr@rFrGrHs@r/r;r;ds  J & :.23726)-,0#',##d*,))D0,((4/ ,  $; , #Tk ,D[, u  !O 3,,r1r;z UMT5 model with a sequence classification/head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. c^\rSrSrS/rS\4U4Sjjr\SS\RS-S\RS-S\RS-S \RS-S \ \RS-S \RS-S \RS-S \RS-S\ S-S\ S-S\ S-S\ S-S\\-4Sjj5rSrU=r$)UMT5ForSequenceClassificationiFdecoder.block.0.layer.1.EncDecAttention.relative_attention_bias.weightrLc>[TU]U5 [U5Ul[ U5UlUR 5 gr^)r$r%r9r&rclassification_headrdr[s r/r%&UMT5ForSequenceClassification.__init__s6  $V,#9&#A  r1Nr)rr(r*rrorrrrrwrxrc  4U bU OURRn UbSn Uc%Ub"[SURR35eUc"UcUc [ S5eUR U5nURUUUUUUUU U U U S9 nUSnURURR5RUR5n[[R"URS55R!5S:HS5 UR"unnnUUSS24R%US U5SS2S SS24nUR'U5nSnUGbURUR5nURR(cURR*S:XaS URlOyURR*S:aNUR,[R.:XdUR,[R0:XaS URlOS URlURR(S :XaT[35nURR*S:Xa&U"UR55UR555nOU"UU5nOURR(S :XaG[75nU"UR%S URR*5UR%S 55nO-URR(S :Xa[95nU"UU5nU (dU4USS-nUbU4U-$U$[;UUUR<UR>UR@URBURDURFURHS 9 $)a input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. UMT5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [UMT5 Training](./umt5#training). decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) UMT5 uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). To know more on how to prepare `decoder_input_ids` for pretraining take a look at [UMT5 Training](./umt5#training). decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). NFz8Passing input embeddings is currently not supported for If no `decoder_input_ids` or `decoder_inputs_embeds` are passed, `input_ids` cannot be `None`. Please pass either `input_ids` or `decoder_input_ids` or `decoder_inputs_embeds`.) rr(r*rrorrrrwrxrrz7All examples must have the same number of tokens.r4 regressionsingle_label_classificationmulti_label_classificationr)%rLrxNotImplementedErrorr.rBrKrOr&eq eos_token_idr6rrr'unique_consecutivesumnumelrrr problem_typerr;rrrsqueezerrrrrrrvrrr)r+r)rr(r*rrorrrrrwrxrrreos_maskrrr,sentence_representationrrrrs r/r@%UMT5ForSequenceClassification.forwardsN`&1%>KbKbJcd   $)>)F  U !% 1 1) < "" )/#9+'"7/!5##  "!*<< 8 89<<_=S=ST  $ $X\\!_ 5 ; ; = B E &5%:%:" A{"1(A+">"C"CJPRT_"`abdfhiai"j))*AB  YYv}}-F{{''/;;))Q./;DKK,[[++a/V\\UZZ5OSYS_S_chclclSl/LDKK,/KDKK,{{''<7"9;;))Q.#FNN$4fnn6FGD#FF3D))-JJ+- B 0F0F GUWY))-II,./Y,F)-)9TGf$ Ev E.#33")"?"?&99$55&-&G&G")"?"?&99  r1)rr&r)rBrCrDrE"_keys_to_ignore_on_load_unexpectedrr%rr'rr`listrrr|rr@rFrGrHs@r/rrsU+s)s&z.2.259:>:>26:>*.!%)-,0#'A ##d*A  t+A !++d2 A !& 0 04 7 A e//047 A ((4/A  %0047A   4'A $;A  $;A #TkA D[A  0 0A A r1rc^\rSrSrS/rS\4U4Sjjr\SS\RS-S\RS-S\RS-S \RS-S \ S-S \ S-S \ S-S \ \R\ -4Sjj5r SrU=r$)rBiWrrLc0>[TU]U5 URUl[U5Ul[ R "UR5Ul[ R"URUR5Ul UR5 gr^) r$r%rr;r&rrVrrXrQr,r5rdr[s r/r%#UMT5ForTokenClassification.__init__\sj   +++F3zz&";";< ))F$6$68I8IJ r1Nr)rrorrrwrxrc UbUOURRnURUUUUUUS9n U Sn URU 5n UR U 5n Sn Ub<[ 5n U "U R SUR5UR S55n U(dXSS4nU bU 4U-$U$[U U U RU RS9$)a input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. UMT5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [UMT5 Training](./umt5#training). labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. N)rrorrwrxrr4r3)rrr>ru) rLrxr&rXr5rrrrr>ru)r+r)rrorrrwrxrrr>rrrrs r/r@"UMT5ForTokenClassification.forwardgs6&1%[TU]U5 URUl[R "UR UR5Ul[R"U5nSUl SUl [U5Ul [R"U5nSUl URUl[U5UlUR"Ul[R$"URUR"5UlUR)5 gr)r$r%rRrrrr]r>rrrrr[rrr`rrrQr3rdrs r/r%!UMT5ForQuestionAnswering.__init__s  ll6#4#4fnnE v.$)!#(  0 v.$(!$*$=$=! 0  ++))FNNF4E4EF r1cUR$r^rrs r/r-UMT5ForQuestionAnswering.get_input_embeddingsrr1c|XlURRU5 URRU5 gr^rrhs r/rj-UMT5ForQuestionAnswering.set_input_embeddingsrr1Nr)rr(r*rstart_positions end_positionsrorrrrwrxrc U bU OURRn U bU OURRn UbUbSn Uc"U cUc [S5eUR U5nU bU OURRn U bU OURRn UcUR UUUU U U S9nORU (aK[ U[5(d6[US[U5S:aUSOS[U5S:aUSOSS9nUSnURUUU SUUU U U U S 9 nUSnURU5nURSS S 9unnURS 5R5nURS 5R5nSnUbUb[UR55S:a*URS 5RUR 5n[UR55S:a*URS 5RUR 5nURS5nUR#SU5nUR#SU5n[%US 9nU"UU5nU"UU5nUU-S- nU (dUU4USS-U-nUbU4U-$U$['UUUUR(UR*UR,UR.UR0UR*UR,S 9 $)a input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. UMT5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [UMT5 Training](./umt5#training). decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) UMT5 uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). To know more on how to prepare `decoder_input_ids` for pretraining take a look at [UMT5 Training](./umt5#training). decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. NFrrrrr3rrr4rr) r start_logits end_logitsrrrrvrrr)rLrxrrKrOrr_rrrr3splitrrrr6rrrrrr>rurvrt)r+r)rr(r*rrrrorrrrwrxrr>rrrrr total_loss ignored_indexr start_lossend_lossrs r/r@ UMT5ForQuestionAnswering.forwards3\&1%)F  U !% 1 1) < !*!6IDKKr)rBrCrDrErr%rrjrr'rrrr|r`rrr@rFrGrHs@r/r<r<s(7'6 .: .23759:>=A371526:>!%)-,0#'I ##d*I ))D0I !++d2 I !& 0 04 7 I uU\\23d: I ))D0I ''$.I ((4/I  %0047I $;I  $;I #TkI D[I u  !$G G!I I r1r<)r;r:r<rrBr9r%)Arrrr'rtorch.nnrrrrnrr7 activationsr cache_utilsr r r generationr masking_utilsrmodeling_layersrmodeling_outputsrrrrrrrmodeling_utilsrutilsrrrrrrconfiguration_umt5r get_loggerrBrModuler!rJrfrqr~rrrrr%r[r9r:r;rrBr<__all__rQr1r/rs AA&!CC)/9.+   H %+BII+4 .RYY<"))$o)BIIo)dRYY2bii6@*@HRYY$j!/j!j!Zf #f Rf #f f R H)#6H)  H)VX*XXv N $7N  N bG !4G G Tp 2p p f r1