ZjSSKJr SSKrSSKJr SSKJrJrJr SSKJ r SSK J r J r SSKJrJrJr SSKJr SS KJrJr SS KJr SS KJrJrJrJrJrJrJ r J!r! SS K"J#r#J$r$ SS K%J&r& SSK'J(r( SSK)J*r*J+r+J,r, SSK-J.r.J/r/ SSK0J1r1 SSK2J3r3 \,Rh"\55r6"SS\Rn5r8SJS\RnS\RrS\RrS\RrS\RrS-S\:S-S\:S\&\*4Sjjr;"SS \Rn5r<"S!S"\Rn5r="S#S$\Rn5r>"S%S&\Rn5r?"S'S(\Rn5r@"S)S*\Rn5rA"S+S,\5rB\+"S-S.\$55rC"S/S0\Rn5rD"S1S2\Rn5rE\+"S3S49"S5S6\C55rF\+"S7S49"S8S9\C\55rG\+"S:S;\C55rH"S<S=\Rn5rI\+"S>S49"S?S@\C55rJ\+"SASB\C55rK\+"SCSD\C55rL"SESF\Rn5rM\+"SGSH\C55rN/SIQrOg)K)CallableN)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)ACT2FNgelu)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)create_bidirectional_maskcreate_causal_mask)GradientCheckpointingLayer))BaseModelOutputWithPastAndCrossAttentions,BaseModelOutputWithPoolingAndCrossAttentions!CausalLMOutputWithCrossAttentionsMaskedLMOutputMultipleChoiceModelOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)apply_chunking_to_forward)TransformersKwargsauto_docstringlogging)can_return_tuplemerge_with_config_defaults)capture_outputs) RobertaConfigc ^\rSrSrSrU4SjrSS\RS-S\RS-S\RS-S\RS-S \ S \R4 S jjr \ S 5r \ SS j5rSrU=r$)RobertaEmbeddings8zGConstruct the embeddings from word, position and token_type embeddings.c>>[TU]5 [R"URUR UR S9Ul[R"URUR 5Ul [R"UR URS9Ul [R"UR5UlURS[ R""UR$5R'S5SS9 URS[ R("UR*R-5[ R.S9SS9 UR Ul[R"UR$UR UR0S9Ulg) N) padding_idxeps position_idsr$F) persistenttoken_type_ids)dtype)super__init__nn Embedding vocab_size hidden_size pad_token_idword_embeddingstype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutregister_buffertorcharangemax_position_embeddingsexpandzerosr-sizelongr*position_embeddingsselfconfig __class__s }/root/GenerationalWealth/GenerationalWealth/venv/lib/python3.13/site-packages/transformers/models/roberta/modeling_roberta.pyr4RobertaEmbeddings.__init__;s4 !||F,=,=v?Q?Q_e_r_rs%'\\&2H2H&J\J\%]"f&8&8f>S>STzz&"<"<=   ELL)G)GHOOPWXej    ekk$*;*;*@*@*B%**Ubg  "..#%<<  * *F,>,>DL\L\$  N input_idsr1r- inputs_embedspast_key_values_lengthreturncUc;UbURXRU5nOURX@R5nUbUR5nOUR5SSnUupxUc[ US5(aQUR R URSS5n [R"U SUS9n U R Xx5nO8[R"U[RURRS9nUcURU5nURU5n XJ-n UR!U5n X-n UR#U 5n UR%U 5n U $)Nr/r1rr$)dimindexr2device)"create_position_ids_from_input_idsr*&create_position_ids_from_inputs_embedsrHhasattrr1rFshaperCgatherrGrIr-rZr:r<rJr=rA) rLrRr1r-rSrT input_shape batch_size seq_lengthbuffered_token_type_idsr< embeddingsrJs rOforwardRobertaEmbeddings.forwardOsb  $#FF//1G   $JJ=ZjZjk  #..*K',,.s3K!,  !t-..*.*=*=*D*D\EWEWXYEZ\^*_'*/,,7NTU]i*j'!8!?!? !W!&[ SWSdSdSkSk!l   00;M $ : :> J": "66|D5 ^^J/ \\*- rQcUR5SSnUSn[R"US-X1-S-[RURS9nUR S5R U5$)z We are provided embeddings directly. We cannot infer which are padded so just generate sequential position ids. Args: inputs_embeds: torch.Tensor Returns: torch.Tensor Nr/r$rYr)rHrCrDrIrZ unsqueezerF)rSr*r`sequence_lengthr-s rOr\8RobertaEmbeddings.create_position_ids_from_inputs_embedssn$((*3B/ %a.|| !O_:Q>ejjYfYmYm %%a(// <L==((6??([L,,|3K''1-88:K  $$rQc ^\rSrSrS U4SjjrS S\R S\RS-S\S-S\ \ S\ \R 4 S jjr S r U=r$) RobertaSelfAttentionNcN>[TU]5 URUR-S:wa7[ US5(d&[ SURSURS35eXlURUl[URUR- 5UlURUR-Ul URS-Ul [R"URUR5Ul [R"URUR5Ul[R"URUR5Ul[R "UR"5UlUR&UlX lX0lgNrembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()r)r3r4r8num_attention_headsr] ValueErrorrMrnattention_head_size all_head_sizerr5Linearrrrr?attention_probs_dropout_probrA is_decoder is_causal layer_idxrLrMrrrNs rOr4RobertaSelfAttention.__init__sM     : : :a ?PVXhHiHi#F$6$6#78 445Q8  #)#=#= #&v'9'9F??*9*N*N'&=%C%CI\`\j\j%k "I(?(M(M KK , ,.E) %8      % $}}C$,,..LL %  % ! "));;;;FFH ((rQ) rrrMrArrrrrrrrFN)NNrtrurvrwr4rCr{rzr rrtuplerer}r~rs@rOrrsl#648(, ')||')))D0') ') +, ') u||  ')')rQrc^\rSrSrS U4SjjrS S\R S\RS-S\RS-S\S-S\ \ S \ \R 4 S jjr S r U=r$)RobertaCrossAttentioniNc,>[TU]5 URUR-S:wa7[ US5(d&[ SURSURS35eXlURUl[URUR- 5UlURUR-Ul URS-Ul [R"URUR5Ul [R"URUR5Ul[R"URUR5Ul[R "UR"5UlX lX0lgr)r3r4r8rr]rrMrnrrrr5rrrrr?rrArrrs rOr4RobertaCrossAttention.__init__s@     : : :a ?PVXhHiHi#F$6$6#78 445Q8  #)#=#= #&v'9'9FB**4>>:(?(M(M KK , ,.E) %8      % $}}C$,,..LL %  % ! "));;;;FFH ((rQ) rrrMrArrrrrrrr)NNN)rtrurvrwr4rCr{rzr rrrrer}r~rs@rOrrs#4;?376: 1)||1) %00471)))D0 1) -t3 1) +, 1) u|| 1)1)rQrcz^\rSrSrU4SjrS\R S\R S\R 4SjrSrU=r $)RobertaSelfOutputiMc(>[TU]5 [R"URUR5Ul[R "URURS9Ul[R"UR5Ul gNr+) r3r4r5rr8denser=r>r?r@rArKs rOr4RobertaSelfOutput.__init__Ns` YYv1163E3EF f&8&8f>S>STzz&"<"<= rQr input_tensorrUcpURU5nURU5nURX-5nU$NrrAr=rLrrs rOreRobertaSelfOutput.forwardT5 =1  ]3 }'CD rQr=rrA rtrurvrwr4rCr{rer}r~rs@rOrrM6> U\\RWR^R^rQrc^\rSrSrS U4SjjrSS\R S\RS-S\RS-S\RS-S\S-S \ \ S \ \R 4S jjr S r U=r$)RobertaAttentioni[Nc>[TU]5 X@lU(a[O[nU"XUS9Ul[ U5Ulg)Nrr)r3r4is_cross_attentionrrrLroutput)rLrMrrrattention_classrNs rOr4RobertaAttention.__init__\s9 "43E/K_#F9U '/ rQrrrencoder_attention_maskrrrUc UR(dUOUnUR"U4UUUS.UD6upxURXq5nXx4$)N)rrr)rrLr) rLrrrrrrattention_outputrs rOreRobertaAttention.forwardcs\04/F/FLb)- * "7)+ *  * & ;;'7G--rQ)rrrL)FNFNNNNrrs@rOrr[s048:>;?(, .||.))D0. %0047 . !& 1 1D 8 .  .+,. u|| ..rQrcb^\rSrSrU4SjrS\R S\R 4SjrSrU=r $)RobertaIntermediateixc>[TU]5 [R"URUR 5Ul[UR[5(a[URUl gURUl gr) r3r4r5rr8intermediate_sizerr hidden_actstrr intermediate_act_fnrKs rOr4RobertaIntermediate.__init__ys` YYv1163K3KL f'' - -'-f.?.?'@D $'-'8'8D $rQrrUcJURU5nURU5nU$rrr)rLrs rOreRobertaIntermediate.forwards& =1 00? rQrrrs@rOrrxs(9U\\ellrQrcz^\rSrSrU4SjrS\R S\R S\R 4SjrSrU=r $) RobertaOutputic(>[TU]5 [R"URUR 5Ul[R"UR URS9Ul[R"UR5Ul gr) r3r4r5rrr8rr=r>r?r@rArKs rOr4RobertaOutput.__init__s` YYv779K9KL f&8&8f>S>STzz&"<"<= rQrrrUcpURU5nURU5nURX-5nU$rrrs rOreRobertaOutput.forwardrrQrrrs@rOrrrrQrc^\rSrSrSU4SjjrSS\R S\RS-S\RS-S\RS-S\S-S \ \ S \R 4S jjr S r S r U=r$) RobertaLayeriNc>[TU]5 URUlSUl[ XR US9UlUR UlURUlUR(a0UR (d[US35e[ USUSS9Ul [U5Ul [U5Ul g)Nr$rz> should be used as a decoder model if cross attention is addedFT)rrr)r3r4chunk_size_feed_forward seq_len_dimrr attentionadd_cross_attentionrcrossattentionr intermediaterr)rLrMrrNs rOr4RobertaLayer.__init__s '-'E'E$)&>  $ ,$  $   1 ??4@4!122 =dVDDD )-(;(;%%& ) !0 )  ) % " 6 0  # #T%A%A4CSCSUe rQcJURU5nURX!5nU$r)rr)rLrintermediate_outputrs rOrRobertaLayer.feed_forward_chunks)"//0@A{{#6I rQ)rrrrrrrrrr)rtrurvrwr4rCr{rzr rrrerr}r~rs@rOrrs,,48:>;?(, %||%))D0% %0047 % !& 1 1D 8 %  %+,% %NrQrcx^\rSrSr\rSrSrSrSr Sr Sr \ \ \S.r\R""5U4Sj5rSrU=r$)RobertaPreTrainedModelirobertaT)r attentionscross_attentionsc>[TU]U5 [U[5(a![R "UR 5 g[U[5(a|[R"UR[R"URRS5RS55 [R "UR5 gg)zInitialize the weightsr/r.N)r3 _init_weightsr RobertaLMHeadinitzeros_biasr'copy_r-rCrDr^rFr1)rLrrNs rOr$RobertaPreTrainedModel._init_weightss f% fm , , KK $  1 2 2 JJv**ELL9L9L9R9RSU9V,W,^,^_f,g h KK-- .3rQ)rtrurvrwr% config_classbase_model_prefixsupports_gradient_checkpointing_supports_flash_attn_supports_sdpa_supports_flex_attn_supports_attention_backendrrr_can_record_outputsrCno_gradrr}r~rs@rOr r sV L!&*#N"&%*1  ]]_//rQr c^\rSrSrU4SjrSS\R S\RS-S\RS-S\RS-S\S-S \ S-S \ \ S \ \R \ -4S jjrS rU=r$)RobertaEncoderic >[TU]5 Xl[R"[ UR 5Vs/sH n[XS9PM sn5Ulgs snf)N)r) r3r4rMr5 ModuleListrangenum_hidden_layersrlayer)rLrMirNs rOr4RobertaEncoder.__init__sH  ]]uU[UmUmOn#oOn!L$EOn#op #osANrrrrr use_cacherrUc [UR5HupU "UUU4UUS.UD6nM [UU(aUS9$SS9$)N)rr)last_hidden_stater) enumerater)r) rLrrrrrr,rr* layer_modules rOreRobertaEncoder.forwardsg )4OA(%(> /   M 59+/8O  >B  rQ)rMr))NNNNN)rtrurvrwr4rCr{rzr boolrrrrrer}r~rs@rOr$r$sq48:>;?(,!% || ))D0  %0047  !& 1 1D 8    $; +,  u|| H H  rQr$cb^\rSrSrU4SjrS\R S\R 4SjrSrU=r $) RobertaPooleric>[TU]5 [R"URUR5Ul[R "5Ulgr)r3r4r5rr8rTanh activationrKs rOr4RobertaPooler.__init__s9 YYv1163E3EF '')rQrrUc\USS2S4nURU5nURU5nU$Nr)rr7)rLrfirst_token_tensor pooled_outputs rOreRobertaPooler.forwards6+1a40 #56  6 rQ)r7rrrs@rOr4r4s($ U\\ellrQr4a  The model can behave as an encoder (with only self-attention) as well as a decoder, in which case a layer of cross-attention is added between the self-attention layers, following the architecture described in [Attention is all you need](https://huggingface.co/papers/1706.03762) by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin. To behave as an decoder the model needs to be initialized with the `is_decoder` argument of the configuration set to `True`. To be used in a Seq2Seq model, the model needs to initialized with both `is_decoder` argument and `add_cross_attention` set to `True`; an `encoder_hidden_states` is then expected as an input to the forward pass. ) custom_introc^\rSrSrSS/rSU4SjjrSrSr\\ \ SS\ RS-S \ RS-S \ RS-S \ RS-S \ RS-S \ RS-S\ RS-S\ S-S\S-S\\S\\ R\-4Sjj555rSrSrU=r$) RobertaModelir'rc>[TU]U5 XlSUl[ U5Ul[ U5UlU(a [U5OSUl UR5 g)z^ add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer FN) r3r4rMgradient_checkpointingr'rdr$encoderr4pooler post_init)rLrMadd_pooling_layerrNs rOr4RobertaModel.__init__.sS   &+#+F3%f- /@mF+d  rQc.URR$rrdr:rLs rOget_input_embeddings!RobertaModel.get_input_embeddings?s...rQc$XRlgrrI)rLrs rOset_input_embeddings!RobertaModel.set_input_embeddingsBs */'rQNrRrr1r-rSrrrr,rrUc USLUSL- (a [S5eURR(aU bU OURRn OSn U (abUc_UcURR(a.[ [ URS9[ URS95O[ URS9nUbUR5OSn URUUUUU S9n URUUU UUS9up'UR"U 4UUUUU US.U D6n U RnURbURU5OSn[UUU RS9$) Nz:You must specify exactly one of input_ids or inputs_embedsF)rMr)rRr-r1rSrT)rrembedding_outputrr)rrrrr,r-)r. pooler_outputr)rrMrr,is_encoder_decoderr r get_seq_lengthrd_create_attention_masksrCr.rDrr)rLrRrr1r-rSrrrr,rrTrQencoder_outputssequence_outputr<s rOreRobertaModel.forwardEsw  -t"; <YZ Z ;; ! !%.%:  @U@UII 0)4 8V8V$L $DlZ^ZeZeFfg!5  FUE`!?!?!Afg??%)'#9 + 261M1M)#9-"7+ 2N2 .,,   )"7#9+%    *;;8< 8O O4UY ;-'+;;  rQcURR(a[URUUUS9nO[URUUS9nUb[URUUUS9nX4$)N)rMrSrr)rMrSr)rMrSrr)rMrrr)rLrrrQrrs rOrU$RobertaModel._create_attention_maskssr ;; ! !/{{.- / N7{{.-N " -%>{{.5&; & "55rQ)rMrdrCrBrD)T) NNNNNNNNN)rtrurvrw_no_split_modulesr4rKrNr"r#rrCr{r r2rrrrrerUr}r~rs@rOr@r@s3-n="/0 *..2.2,0-1596:(,!%? <<$&?  t+?  t+ ? llT) ? ||d* ?  %||d2? !& t 3? ? $;? +,?  u|| K K?  ? B66rQr@zS RoBERTa Model with a `language modeling` head on top for CLM fine-tuning. c^\rSrSrSSS.rU4SjrSrSr\\ SS \ RS-S \ RS-S \ RS-S \ RS-S \ RS-S\ RS-S\ RS-S\ RS-S\ \ \ RS-S\S-S\\ R -S\\S\ \ R \-4Sjj55rSrU=r$)RobertaForCausalLMi)roberta.embeddings.word_embeddings.weight lm_head.biaszlm_head.decoder.weightzlm_head.decoder.biasc>[TU]U5 UR(d[R S5 [ USS9Ul[U5UlUR5 g)NzOIf you want to use `RobertaLMHeadModel` as a standalone, add `is_decoder=True.`FrF r3r4rloggerwarningr@rrlm_headrErKs rOr4RobertaForCausalLM.__init__sL    NNl m#FeD $V,  rQc.URR$rrfdecoderrJs rOget_output_embeddings(RobertaForCausalLM.get_output_embeddings||###rQc$XRlgrrirLnew_embeddingss rOset_output_embeddings(RobertaForCausalLM.set_output_embeddings - rQNrRrr1r-rSrrlabelsrr,logits_to_keeprrUc UbSn UR"U4UUUUUUU U SS. U D6n U Rn[U [5(a [ U *S5OU nUR USS2USS245nSnUb)UR "SUXRRS.U D6n[UUU RU RU RU RS9$)a token_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the left-to-right language modeling loss (next word prediction). Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` Example: ```python >>> from transformers import AutoTokenizer, RobertaForCausalLM, AutoConfig >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("FacebookAI/roberta-base") >>> config = AutoConfig.from_pretrained("FacebookAI/roberta-base") >>> config.is_decoder = True >>> model = RobertaForCausalLM.from_pretrained("FacebookAI/roberta-base", config=config) >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> prediction_logits = outputs.logits ```NFT) rr1r-rSrrrr, return_dict)logitsrtr7)lossrxrrrrr)rr.rrnslicerf loss_functionrMr7rrrrr)rLrRrr1r-rSrrrtrr,ruroutputsr slice_indicesrxrys rOreRobertaForCausalLM.forwards`  I@D  A ))%'"7#9+ A  A  11 8B>SV8W8W~ot4]k mA}a,?@A  %%pVF{{OeOepiopD0#33!//))$55   rQrfr) NNNNNNNNNNr)rtrurvrw_tied_weights_keysr4rkrqr!rrCryrzrr2rnr{rrrrer}r~rs@rOr]r]s#N . $..237260426:>;?*.BF!%-.O ##d*O ))D0O ((4/ O &&- O ((4/ O  %0047O !& 1 1D 8O   4'O uU%6%6784?O $;O ell*O +,O  u|| @ @O O rQr]c^\rSrSrSSS.rU4SjrSrSr\\ SS \ RS-S \ RS-S \ RS-S \ RS-S \ RS-S\ RS-S\ RS-S\ RS-S\ \S\\ R \-4Sjj55rSrU=r$)RobertaForMaskedLMir^r_r`c>[TU]U5 UR(a[R S5 [ USS9Ul[U5UlUR5 g)NznIf you want to use `RobertaForMaskedLM` make sure `config.is_decoder=False` for bi-directional self-attention.FrbrcrKs rOr4RobertaForMaskedLM.__init__"sR     NN1  $FeD $V,  rQc.URR$rrirJs rOrk(RobertaForMaskedLM.get_output_embeddings1rmrQc$XRlgrriros rOrq(RobertaForMaskedLM.set_output_embeddings4rsrQNrRrr1r-rSrrrtrrUc pUR"U4UUUUUUSS.U D6n U Sn URU 5n Sn UbaURU R5n[ 5nU"U R SUR R5UR S55n [U U U RU RS9$)a token_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` T)rr1r-rSrrrwrNr/ryrxrr) rrftorZrrrMr7rrr)rLrRrr1r-rSrrrtrr|rWprediction_scoresmasked_lm_lossloss_fcts rOreRobertaForMaskedLM.forward7s:,,   ))%'"7#9    "!* LL9  YY0778F')H%&7&<&;?*.5 ##d*5 ))D05 ((4/ 5 &&- 5 ((4/ 5  %00475 !& 1 1D 85   4'5 +,5  u|| ~ -5 5 rQrc2^\rSrSrSrU4SjrSrSrU=r$)riqz*Roberta Head for masked language modeling.c>[TU]5 [R"URUR5Ul[R "URURS9Ul[R"URUR5Ul [R"[R"UR55Ulgr)r3r4r5rr8rr=r> layer_normr7rj ParameterrCrGrrKs rOr4RobertaLMHead.__init__ts YYv1163E3EF ,,v'9'9v?T?TUyy!3!3V5F5FG LLV->->!?@ rQc URU5n[U5nURU5nURU5nU$r)rr rrjrLfeaturesrxs rOreRobertaLMHead.forward|s; JJx  G OOA  LLOrQ)rrjrr rtrurvrwrxr4rer}r~rs@rOrrqs4ArQrz RoBERTa Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. c8^\rSrSrU4Sjr\\SS\RS-S\RS-S\RS-S\RS-S\RS-S \RS-S \ \ S \ \R\-4S jj55rS rU=r$) RobertaForSequenceClassificationic>[TU]U5 URUlXl[ USS9Ul[ U5UlUR5 gNFrb) r3r4 num_labelsrMr@rRobertaClassificationHead classifierrErKs rOr4)RobertaForSequenceClassification.__init__sH   ++ #FeD 3F; rQNrRrr1r-rSrtrrUc UR"U4UUUUSS.UD6nUSn URU 5n Sn UGbURU R5nURR cUR S:XaSURlOoUR S:aNUR[R:XdUR[R:XaSURlOSURlURR S:XaI[5n UR S:Xa&U "U R5UR55n OU "X5n OURR S:Xa=[5n U "U RS UR 5URS 55n O,URR S:Xa[5n U "X5n [!U U UR"UR$S 9$) a token_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Trr1r-rSrwrNr$ regressionsingle_label_classificationmulti_label_classificationr/r)rrrrZrM problem_typerr2rCrIrnrsqueezerrrrrr rLrRrr1r-rSrtrr|rWrxryrs rOre(RobertaForSequenceClassification.forwards6,,  ))%'   "!*1  YYv}}-F{{''/??a'/;DKK,__q(fllejj.HFLL\a\e\eLe/LDKK,/KDKK,{{''<7"9??a'#FNN$4fnn6FGD#F3D))-JJ+- B @&++b/R))-II,./'!//))   rQ)rrMrrNNNNNN)rtrurvrwr4r!rrCryrzrrrr{rrer}r~rs@rOrrs .237260426*.C ##d*C ))D0C ((4/ C &&- C ((4/ C   4'C +,C  u|| 7 7C C rQrc8^\rSrSrU4Sjr\\SS\RS-S\RS-S\RS-S\RS-S\RS-S \RS-S \ \ S \ \R\-4S jj55rS rU=r$)RobertaForMultipleChoiceic>[TU]U5 [U5Ul[R "UR 5Ul[R"URS5Ul UR5 g)Nr$) r3r4r@rr5r?r@rArr8rrErKs rOr4!RobertaForMultipleChoice.__init__sV  #F+ zz&"<"<= ))F$6$6: rQNrRr1rrtr-rSrrUc *UbURSOURSnUb!URSURS55OSn Ub!URSURS55OSn Ub!URSURS55OSn Ub!URSURS55OSn Ub1URSURS5URS55OSn UR"U 4U U U U SS.UD6nUSnUR U5nUR U5nURSU5nSnUb.UR UR5n[5nU"UU5n[UUURURS9$)aO input_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) token_type_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) position_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) inputs_embeds (`torch.FloatTensor` of shape `(batch_size, num_choices, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. Nr$r/T)r-r1rrSrwr) r^rrHrrArrrZrrrr)rLrRr1rrtr-rSr num_choicesflat_input_idsflat_position_idsflat_token_type_idsflat_attention_maskflat_inputs_embedsr|r<rxreshaped_logitsryrs rOre RobertaForMultipleChoice.forwardsV-6,Aiooa(}GZGZ[\G] CLCXINN2,>?^bLXLdL--b,2C2CB2GHjnR`Rln11"n6I6I"6MNrvR`Rln11"n6I6I"6MNrv(   r=#5#5b#9=;M;Mb;Q R  ,,  *..,      ]3 / ++b+6  YY556F')HOV4D("!//))   rQ)rrArr)rtrurvrwr4r!rrCryrzrrrr{rrer}r~rs@rOrrs.22637*.0426P ##d*P ((4/P ))D0 P   4' P &&- P ((4/P +,P  u|| 8 8P P rQrc8^\rSrSrU4Sjr\\SS\RS-S\RS-S\RS-S\RS-S\RS-S \RS-S \ \ S \ \R\-4S jj55rS rU=r$)RobertaForTokenClassificationiBcd>[TU]U5 URUl[USS9UlUR b UR O UR n[R"U5Ul [R"URUR5Ul UR5 gr)r3r4rr@rclassifier_dropoutr@r5r?rArr8rrErLrMrrNs rOr4&RobertaForTokenClassification.__init__Ds   ++#FeD )/)B)B)NF % %TZTnTn zz"45 ))F$6$68I8IJ rQNrRrr1r-rSrtrrUc zUR"U4UUUUSS.UD6nUSn URU 5n URU 5n Sn UbWURU R5n[ 5n U "U R SUR5UR S55n [U U URURS9$)a token_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. TrrNr/r) rrArrrZrrrrrrrs rOre%RobertaForTokenClassification.forwardRs2,,  ))%'   "!*,,71  YYv}}-F')HFKKDOO[TU]5 [R"URUR5UlUR b UR O URn[R"U5Ul [R"URUR5Ul gr) r3r4r5rr8rrr@r?rArout_projrs rOr4"RobertaClassificationHead.__init__s YYv1163E3EF )/)B)B)NF % %TZTnTn zz"45  &"4"4f6G6GH rQc USS2SSS24nURU5nURU5n[R"U5nURU5nUR U5nU$r:)rArrCtanhrrs rOre!RobertaClassificationHead.forwardsY Q1W  LLO JJqM JJqM LLO MM! rQ)rrArrrs@rOrrs7IrQrcX^\rSrSrU4Sjr\\SS\RS-S\RS-S\RS-S\RS-S\RS-S \RS-S \RS-S \ \ S \ \R\-4S jj55rSrU=r$)RobertaForQuestionAnsweringic>[TU]U5 URUl[USS9Ul[ R "URUR5UlUR5 gr) r3r4rr@rr5rr8 qa_outputsrErKs rOr4$RobertaForQuestionAnswering.__init__sU   ++#FeD ))F$6$68I8IJ rQNrRrr1r-rSstart_positions end_positionsrrUc UR"U4UUUUSS.UD6n U Sn URU 5n U RSSS9upU RS5R 5n U RS5R 5n SnUbUb[ UR 55S:aURS5n[ UR 55S:aURS5nU R S5nURSU5nURSU5n[US9nU"X5nU"X5nUU-S - n[UU U U RU RS 9$) a token_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) Trrr$r/rlN) ignore_indexr)ry start_logits end_logitsrr) rrsplitrrlenrHclamprrrr)rLrRrr1r-rSrrrr|rWrxrr total_loss ignored_indexr start_lossend_losss rOre#RobertaForQuestionAnswering.forwardsx0,,  ))%'   "!*1#)<<r<#: #++B/::< ''+668   &=+D?'')*Q."1"9"9""==%%'(1, - 5 5b 9 (--a0M-33A}EO)//=AM']CH!,@J :H$x/14J+%!!//))   rQ)rrr)NNNNNNN)rtrurvrwr4r!rrCryrzrrrr{rrer}r~rs@rOrrs.2372604263715> ##d*> ))D0> ((4/ > &&- > ((4/ > ))D0> ''$.> +,>  u|| ; ;> > rQr)r]rrrrrr@r )Nr)Pcollections.abcrrCtorch.nnr5rrrrr activationsr r cache_utilsr r r generationr masking_utilsrrmodeling_layersrmodeling_outputsrrrrrrrrmodeling_utilsrrprocessing_utilsr pytorch_utilsrutilsrrr utils.genericr!r"utils.output_capturingr#configuration_robertar% get_loggerrtrdModuler'r{floatrrrrrrrrr r$r4r@r]rrrrrrr__all__rrQrOrs,% AA&'CC)J9   G&6@@I50   H %g8 g8`!% II% <<% % << % LL4' % T\ %%'(%8@)299@)FI)BIII)X  .ryy.: "))  BII >->B/_//2 RYY @ BII   |6)|6 |6~ i /i  i XR /R R jBII, Q '=Q  Q h] 5] ] @C $:C C L ,K "8K K \ rQ