Zj4SrSSKJr SSKJr SSKrSSKJr SSKJrJ r J r SSK J r SS KJr SS KJr SS KJrJrJrJrJrJrJr SS KJrJr SS KJr SSKJ r SSK!J"r"J#r#J$r$J%r% SSK&J'r'J(r( SSK)J*r* SSK+J,r, \%RZ"\.5r/"SS\R`5r1SBS\R`S\RdS\RdS\RdS\RdS-S\3S-S\3S\\#4Sjjr4"SS \R`5r5"S!S"\R`5r6"S#S$\R`5r7"S%S&\R`5r8\$"S'S(\55r9\$"S)S*9\"S+S,\"555r:\$"S-S.\955r;\$"S/S*9"S0S1\955r<"S2S3\R`5r="S4S5\R`5r>\$"S6S7\955r?\$"S8S*9"S9S:\955r@\$"S;S<\955rA\$"S=S>\955rB\$"S?S@\955rC/SAQrDg)CzPyTorch ALBERT model.)Callable) dataclassN)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)ACT2FN)create_bidirectional_mask)BaseModelOutputBaseModelOutputWithPoolingMaskedLMOutputMultipleChoiceModelOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)apply_chunking_to_forward) ModelOutputTransformersKwargsauto_docstringlogging)can_return_tuplemerge_with_config_defaults)capture_outputs) AlbertConfigc ^\rSrSrSrS\4U4SjjrS S\RS-S\RS-S\RS-S \RS-S \R4 S jjr S r U=r $)AlbertEmbeddings1zI Construct the embeddings from word, position and token_type embeddings. configc >[TU]5 [R"URUR UR S9Ul[R"URUR 5Ul [R"URUR 5Ul [R"UR URS9Ul [R"UR5UlUR#S[$R&"UR5R)S5SS9 UR#S[$R*"UR,R/5[$R0S9SS9 g) N) padding_idxeps position_idsrF) persistenttoken_type_ids)dtype)super__init__r Embedding vocab_sizeembedding_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingstype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutregister_buffertorcharangeexpandzerosr)sizelongselfr$ __class__s {/root/GenerationalWealth/GenerationalWealth/venv/lib/python3.13/site-packages/transformers/models/albert/modeling_albert.pyr0AlbertEmbeddings.__init__6s !||F,=,=v?T?Tbhbubuv#%<<0N0NPVPePe#f %'\\&2H2H&J_J_%`"f&;&;AVAVWzz&"<"<=   ELL)G)GHOOPWXej    ekk$*;*;*@*@*B%**Ubg  N input_idsr-r) inputs_embedsreturncUbUR5nOUR5SSnUupgUcURSS2SU24nUc[US5(aQURR UR SS5n[ R"USUS9nUR Xg5nO8[ R"U[ RURRS9nUcURU5nURU5n XI-n URU5n X-n URU 5n URU 5n U $)Nr+r-rr)dimindex)r.device)rDr)hasattrr-rBshaper@gatherrCrErRr5r9r7r:r>) rGrLr-r)rM input_shape batch_size seq_lengthbuffered_token_type_idsr9 embeddingsr7s rIforwardAlbertEmbeddings.forwardGs@  #..*K',,.s3K!,  ,,Q  ^ J": "66|D5 ^^J/ \\*- rK)r:r>r7r9r5)NNNN)__name__ __module__ __qualname____firstlineno____doc__r r0r@ LongTensor FloatTensorTensorr[__static_attributes__ __classcell__rHs@rIr"r"1s | &.2260426 '##d*'((4/'&&- ' ((4/ '  ''rKr"modulequerykeyvalueattention_maskscalingr>kwargsc UcURS5S-n[R"XRSS55U-nUbX-n[R R USS9n[R RXURS9n[R"X5n U RSS5R5n X4$)Nr+r rP)ptrainingr) rDr@matmul transposer functionalsoftmaxr>rt contiguous) rhrirjrkrlrmr>rn attn_weights attn_outputs rIeager_attention_forwardr|rs**R.D(<<}}Q':;gEL!#4 ==((2(>L==((6??([L,,|3K''1-88:K  $$rKc ^\rSrSrS\4U4SjjrS S\RS\RS-S\ \ S\ \R\R44S jjr S r U=r$) AlbertAttentionr$c6>[TU]5 URUR-S:wa6[ US5(d%[ SURSUR35eXlURUlURUlURUR-UlURUR-UlURS-Ul [R"UR5Ul [R"UR5Ul[R "URUR5Ul[R "URUR5Ul[R "URUR5Ul[R "URUR5Ul[R*"URUR,S9UlSUlg)Nrr3zThe hidden size (z6) is not a multiple of the number of attention heads (rpr'F)r/r0 hidden_sizenum_attention_headsrS ValueErrorr$attention_head_size all_head_sizermrr<attention_probs_dropout_probattention_dropoutr=output_dropoutLinearrirjrkdenser:r; is_causalrFs rIr0AlbertAttention.__init__s     : : :a ?PVXhHiHi#F$6$6#78 4457  #)#=#= !--#)#5#59S9S#S !558P8PP//5 !#F,O,O!P jj)C)CDYYv1143E3EF 99V//1C1CDYYv1143E3EF YYv1163E3EF f&8&8f>S>STrKN hidden_statesrlrnrNc URSSn/UQSPURP7nURU5R"U6R SS5nUR U5R"U6R SS5nUR U5R"U6R SS5n[R"URR[5n U "UUUUU4UR(dSOURRURS.UD6upU R "/UQSP76R#5n UR%U 5n UR'U 5n UR)X-5n X4$)Nr+rrq)r>rm)rTrriviewrvrjrkr get_interfacer$_attn_implementationr|rtrrsrmreshaperyrrr:) rGrrlrnrV hidden_shape query_layer key_layer value_layerattention_interfacer{rzs rIr[AlbertAttention.forwardso $))#2. CCbC$*B*BC jj/44lCMMaQRS HH]+00,?II!QO jj/44lCMMaQRS (?(M(M KK , ,.E) %8      % $}}C$2H2H2J2JLL %  % ! "));;;;FFH jj- ))+6 nn]%@A ((rK)r:rrrr$rrrrjrrrirmrkNr]r^r_r`r r0r@rdrcrrtupler[rerfrgs@rIr~r~sk|<48")||")))D0")+, ") u||U\\) * ")")rKr~c ^\rSrSrS\4U4SjjrS S\RS\RS-S\ \ S\ \R\R44S jjr S \RS\R4S jr S rU=r$) AlbertLayerr$c>[TU]5 XlURUlSUl[ R "URURS9Ul [U5Ul [ R"URUR5Ul[ R"URUR5Ul[ UR"Ul[ R&"UR(5Ulg)Nrr')r/r0r$chunk_size_feed_forward seq_len_dimrr:rr;full_layer_layer_normr~ attentionrintermediate_sizeffn ffn_outputr hidden_act activationr<r=r>rFs rIr0AlbertLayer.__init__s  '-'E'E$%'\\&2D2D&J_J_%`"(099V//1I1IJ))F$<$P>PQ !2!23zz&"<"<= rKNrrlrnrNc UR"X40UD6upE[URURURU5nUR Xd-5nU$r)rrff_chunkrrr)rGrrlrnattention_output_rs rIr[AlbertLayer.forwards\ #nn]UfU. MM  ( (      22:3PQ rKrclURU5nURU5nURU5nU$r)rrr)rGrrs rIrAlbertLayer.ff_chunks3XX./ __Z0 __Z0 rK) rrrr$r>rrrrr)r]r^r_r`r r0r@rdrcrrrr[rrerfrgs@rIrrs >| > 48||))D0+,  u||U\\) * "%,,rKrc ^\rSrSrS\4U4SjjrS S\RS\RS-S\ \ S\ \R\ \R-S 44S jjr S r U=r$) AlbertLayerGroupr$c>[TU]5 [R"[ UR 5Vs/sHn[ U5PM sn5Ulgs snfr)r/r0r ModuleListrangeinner_group_numr albert_layersrGr$rrHs rIr0AlbertLayerGroup.__init__sC ]]vOeOeIf+gIfAK,?If+gh+gsANrrlrnrN.c T[UR5HupEU"X40UD6nM U$r) enumerater)rGrrlrn layer_index albert_layers rIr[AlbertLayerGroup.forwards2 *343E3E)F %K(Q&QM*GrK)rrrrgs@rIrrswi|i48||))D0+,  u||eELL1136 7 rKrc ^\rSrSrS\4U4SjjrS S\RS\RS-S\ \ S\ \ -4S jjr S rU=r$) AlbertTransformerir$c(>[TU]5 Xl[R"UR UR 5Ul[R"[UR5Vs/sHn[U5PM sn5Ul gs snfr) r/r0r$rrr3rembedding_hidden_mapping_inrrnum_hidden_groupsralbert_layer_groupsrs rIr0AlbertTransformer.__init__sj  +-99V5J5JFL^L^+_(#%==TYZ`ZrZrTs1tTsq2B62JTs1t#u 1ts-BNrrlrnrNc URU5n[URR5HRn[ X@RRURR - - 5nUR U"UU40UD6nMT [US9$)N)last_hidden_state)rrr$num_hidden_layersintrrr )rGrrlrni group_idxs rIr[AlbertTransformer.forwards 88G t{{445AA!>!>A^A^!^_`I 44Y?M 6??rK)rr$rr)r]r^r_r`r r0r@rdrcrrr rr[rerfrgs@rIrrsfv|v48@||@))D0@+, @ 5 @@rKrcd\rSrSr\rSrSrSrSr Sr \ \ S.r \R"5S5rSrg)AlbertPreTrainedModeli#albertT)r attentionsc[U[R5(ac[R"UR SUR RS9 URb![R"UR5 gg[U[R5(a[R"UR SUR RS9 URbK[UR SS5(d.[R"UR UR5 ggg[U[R5(aA[R"UR5 [R"UR 5 g[U[5(a![R"UR5 g[U[ 5(a|[R""UR$[&R("UR$R*S5R-S55 [R"UR.5 gg)zInitialize the weights.r)meanstdN_is_hf_initializedFr+r*) isinstancerrinitnormal_weightr$initializer_rangebiaszeros_r1r&getattrr:ones_ AlbertMLMHeadr"copy_r)r@rArTrBr-)rGrhs rI _init_weights#AlbertPreTrainedModel._init_weights0ss fbii ( ( LLSdkk6S6S T{{& FKK('  - - LLSdkk6S6S T!!-gfmmMach6i6i FMM&*<*<=>7j-  - - KK $ JJv}} %  . . KK $  0 1 1 JJv**ELL9L9L9R9RSU9V,W,^,^_f,g h KK-- .2rKN)r]r^r_r`r config_classbase_model_prefix_supports_flash_attn_supports_sdpa_supports_flex_attn_supports_attention_backendrr~_can_record_outputsr@no_gradrrerrKrIrr#sLL N"&$%  ]]_//rKrz2 Output type of [`AlbertForPreTraining`]. ) custom_introc\rSrSr%SrSr\RS-\S'Sr \RS-\S'Sr \RS-\S'Sr \ \RS-\S'Sr \ \RS-\S'S rg) AlbertForPreTrainingOutputiFae loss (*optional*, returned when `labels` is provided, `torch.FloatTensor` of shape `(1,)`): Total loss as the sum of the masked language modeling loss and the next sequence prediction (classification) loss. prediction_logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). sop_logits (`torch.FloatTensor` of shape `(batch_size, 2)`): Prediction scores of the next sequence prediction (classification) head (scores of True/False continuation before SoftMax). Nlossprediction_logits sop_logitsrrr)r]r^r_r`rarr@rc__annotations__rrrrrrerrKrIrrFs} &*D%  d ")26u((4/6+/J!!D(/59M5**+d2926Je''(4/6rKrch^\rSrSr\rSrSS\S\4U4SjjjrS\ R4Sjr S\ RSS 4S jr \ \\SS \R"S -S \R$S -S \R"S -S\R"S -S\R$S -S\\S\\-4Sjj555rSrU=r$) AlbertModeli_rr$add_pooling_layercr>[TU]U5 Xl[U5Ul[ U5UlU(aK[R"URUR5Ul [R"5Ul OSUl SUl URUlUR5 g)z^ add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer N)r/r0r$r"rZrencoderrrrpoolerTanhpooler_activationrattn_implementation post_init)rGr$rrHs rIr0AlbertModel.__init__ds   *62(0 ))F$6$68J8JKDK%'WWYD "DK%)D "#)#>#>  rKrNc.URR$rrZr5rGs rIget_input_embeddings AlbertModel.get_input_embeddingszs...rKrkNc$XRlgrr)rGrks rIset_input_embeddings AlbertModel.set_input_embeddings}s */'rKrLrlr-r)rMrnc 4USLUSL- (a [S5eURXX5S9n[URUUS9nUR"UU4SU0UD6nUSn UR b'UR UR U SS2S455OSn [U U S9$)Nz:You must specify exactly one of input_ids or inputs_embeds)r)r-rM)r$rMrlr)r)r pooler_output)rrZr r$rrrr) rGrLrlr-r)rMrnembedding_outputencoder_outputssequence_output pooled_outputs rIr[AlbertModel.forwards  -t"; <YZ Z?? + 3;;*)  ,,   &   *!,VZVaVaVm..t{{?1a4;P/QRsw )-'  rK)rr$rZrrr)T)NNNNN)r]r^r_r`r rrboolr0rr1rrrrrr@rbrcrrrrr[rerfrgs@rIrr_sL |,/bll/0",,040 .237260426 $ ##d*$ ))D0$ ((4/ $ &&- $ ((4/ $ +,$  $e +$  $ rKrz Albert Model with two heads on top as done during the pretraining: a `masked language modeling` head and a `sentence order prediction (classification)` head. c^\rSrSrSSS.rS\4U4SjjrS\R4Sjr S \RSS 4S jr S\R4S jr \ \SS \R S -S\R"S -S\R S -S\R S -S\R"S -S\R S -S\R S -S\\S\\-4Sjj55rSrU=r$)AlbertForPreTrainingi(albert.embeddings.word_embeddings.weightpredictions.biaszpredictions.decoder.weightzpredictions.decoder.biasr$c>[TU]U5 [U5Ul[ U5Ul[ U5UlUR5 gr) r/r0rrr predictions AlbertSOPHeadsop_classifierrrFs rIr0AlbertForPreTraining.__init__sB  !&) (0+F3 rKrNc.URR$rrdecoderrs rIget_output_embeddings*AlbertForPreTraining.get_output_embeddings'''rKnew_embeddingsNc$XRlgrrrGrs rIset_output_embeddings*AlbertForPreTraining.set_output_embeddingss#1 rKcBURRR$rrrZr5rs rIr)AlbertForPreTraining.get_input_embeddings{{%%555rKrLrlr-r)rMlabelssentence_order_labelrnc UR"U4UUUUSS.UD6n U SSupURU 5n URU 5n SnUbvUbs[5nU"U R SUR R 5UR S55nU"U R SS5UR S55nUU-n[UU U U RU RS9$)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` sentence_order_label (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the next sequence prediction (classification) loss. Input should be a sequence pair (see `input_ids` docstring) Indices should be in `[0, 1]`. `0` indicates original order (sequence A, then sequence B), `1` indicates switched order (sequence B, then sequence A). Example: ```python >>> from transformers import AutoTokenizer, AlbertForPreTraining >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("albert/albert-base-v2") >>> model = AlbertForPreTraining.from_pretrained("albert/albert-base-v2") >>> input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True)).unsqueeze(0) >>> # Batch size 1 >>> outputs = model(input_ids) >>> prediction_logits = outputs.prediction_logits >>> sop_logits = outputs.sop_logits ```Trlr-r)rM return_dictNrqr+)rrrrr) rrrrrr$r2rrr)rGrLrlr-r)rMr(r)rnoutputsr r prediction_scores sop_scores total_lossloss_fctmasked_lm_losssentence_order_losss rIr[AlbertForPreTraining.forwardsN++  ))%'   *1!& ,,_=((7   "6"B')H%&7&<&[TU]5 [R"URUR S9Ul[R "[R"UR55Ul [R"URUR5Ul [R"URUR5Ul[UR Ulg)Nr')r/r0rr:r3r; Parameterr@rCr2rrrrrr rrrFs rIr0AlbertMLMHead.__init__s f&;&;AVAVWLLV->->!?@ YYv1163H3HI yy!6!68I8IJ  !2!23rKrrNcURU5nURU5nURU5nURU5nUnU$r)rrr:r)rGrr.s rIr[AlbertMLMHead.forwardsF =1  6 }5  ]3 )  rK)r:rrrr r]r^r_r`r r0r@rdr[rerfrgs@rIrrs/4|4!U\\!ell!!rKrcj^\rSrSrS\4U4SjjrS\RS\R4SjrSr U=r $)ri$r$c>[TU]5 [R"UR5Ul[R "URUR5Ul gr) r/r0rr<classifier_dropout_probr>rr num_labels classifierrFs rIr0AlbertSOPHead.__init__%sB zz&"@"@A ))F$6$68I8IJrKr rNcJURU5nURU5nU$r)r>rB)rGr dropout_pooled_outputlogitss rIr[AlbertSOPHead.forward+s% $ ] ;!67 rK)rBr>r=rgs@rIrr$s1K|K U\\ellrKrc^\rSrSrSSS.rU4SjrS\R4SjrS\RSS 4S jr S\R4S jr \ \ SS \RS -S \R S -S\RS -S\RS -S\R S -S\RS -S\\S\\-4Sjj55rSrU=r$)AlbertForMaskedLMi1rrrc>[TU]U5 [USS9Ul[ U5UlUR 5 gNF)r)r/r0rrrrrrFs rIr0AlbertForMaskedLM.__init__8s7  !&EB (0 rKrNc.URR$rrrs rIr'AlbertForMaskedLM.get_output_embeddingsArrKrNcZXRlURURlgr)rrrr!s rIr"'AlbertForMaskedLM.set_output_embeddingsDs"#1 . 3 3rKcBURRR$rr%rs rIr&AlbertForMaskedLM.get_input_embeddingsHr'rKrLrlr-r)rMr(rnc 6UR"SUUUUUSS.UD6nUSn URU 5n Sn UbF[5n U "U RSURR 5URS55n [ U U URURS9$)a$ labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` Example: ```python >>> import torch >>> from transformers import AutoTokenizer, AlbertForMaskedLM >>> tokenizer = AutoTokenizer.from_pretrained("albert/albert-base-v2") >>> model = AlbertForMaskedLM.from_pretrained("albert/albert-base-v2") >>> # add mask_token >>> inputs = tokenizer("The capital of [MASK] is Paris.", return_tensors="pt") >>> with torch.no_grad(): ... logits = model(**inputs).logits >>> # retrieve index of [MASK] >>> mask_token_index = (inputs.input_ids == tokenizer.mask_token_id)[0].nonzero(as_tuple=True)[0] >>> predicted_token_id = logits[0, mask_token_index].argmax(axis=-1) >>> tokenizer.decode(predicted_token_id) 'france' ``` ```python >>> labels = tokenizer("The capital of France is Paris.", return_tensors="pt")["input_ids"] >>> labels = torch.where(inputs.input_ids == tokenizer.mask_token_id, labels, -100) >>> outputs = model(**inputs, labels=labels) >>> round(outputs.loss.item(), 2) 0.81 ``` TrLrlr-r)rMr,rNr+rrFrrr) rrrrr$r2rrr) rGrLrlr-r)rMr(rnr-sequence_outputsr.r2r1s rIr[AlbertForMaskedLM.forwardKs^++ ))%'   #1: ,,-=>  ')H%&7&<&[TU]U5 URUlXl[ U5Ul[ R"UR5Ul [ R"URURR5Ul UR5 gr)r/r0rAr$rrrr<r@r>rrrBrrFs rIr0(AlbertForSequenceClassification.__init__sr   ++ !&) zz&"@"@A ))F$6$6 8N8NO rKNrLrlr-r)rMr(rnrNc UR"S UUUUUSS.UD6nUSn URU 5n URU 5n Sn UGbURRcUR S:XaSURlOoUR S:aNUR [R:XdUR [R:XaSURlOSURlURRS:XaI[5n UR S:Xa&U "U R5UR55n OU "X5n OURRS:Xa=[5n U "U RSUR 5URS55n O,URRS:Xa[5n U "X5n [U U UR UR"S 9$) ae labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). TrTrN regressionsingle_label_classificationmulti_label_classificationr+rUr)rr>rBr$ problem_typerAr.r@rErrsqueezerrrrrr) rGrLrlr-r)rMr(rnr-r rFrr1s rIr['AlbertForSequenceClassification.forwards$++ ))%'      ]3 /  {{''/??a'/;DKK,__q(fllejj.HFLL\a\e\eLe/LDKK,/KDKK,{{''<7"9??a'#FNN$4fnn6FGD#F3D))-JJ+- B @&++b/R))-II,./'!//))   rK)rrBr$r>rArX)r]r^r_r`r r0rrr@rbrcrrrrr[rerfrgs@rIrZrZs | .237260426*.; ##d*; ))D0; ((4/ ; &&- ; ((4/ ;   4'; +,;  "E ); ; rKrZc&^\rSrSrS\4U4Sjjr\\SS\RS-S\RS-S\RS-S\RS-S \RS-S \RS-S \ \ S \ \-4S jj55rSrU=r$)AlbertForTokenClassificationir$cx>[TU]U5 URUl[USS9UlUR b UR O UR n[R"U5Ul [R"URURR5Ul UR5 grK)r/r0rArrr@r=rr<r>rrr$rBr)rGr$r@rHs rIr0%AlbertForTokenClassification.__init__s   ++!&EB --9  * *++ zz"9: ))F$6$6 8N8NO rKNrLrlr-r)rMr(rnrNc DUR"U4UUUUSS.UD6nUSn URU 5n URU 5n Sn Ub<[5n U "U R SUR 5UR S55n [ U U URURS9$)z labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. Tr+rNr+rU) rr>rBrrrArrr) rGrLrlr-r)rMr(rnr-r rFrr1s rIr[$AlbertForTokenClassification.forwards ++  ))%'   "!*,,71  ')HFKKDOOrArX)r]r^r_r`r r0rrr@rbrcrrrrr[rerfrgs@rIreres| .237260426*.' ##d*' ))D0' ((4/ ' &&- ' ((4/ '   4'' +,'   &' ' rKrecF^\rSrSrS\4U4Sjjr\\SS\RS-S\RS-S\RS-S\RS-S \RS-S \RS-S \RS-S \ \ S \ \-4Sjj55rSrU=r$)AlbertForQuestionAnsweringi%r$c>[TU]U5 URUl[USS9Ul[ R "URUR5UlUR5 grK) r/r0rArrrrr qa_outputsrrFs rIr0#AlbertForQuestionAnswering.__init__'sU   ++!&EB ))F$6$68I8IJ rKNrLrlr-r)rMstart_positions end_positionsrnrNc UR"S UUUUUSS.UD6n U Sn URU 5n U RSSS9upU RS5R 5n U RS5R 5n SnUbUb[ UR 55S:aURS5n[ UR 55S:aURS5nU R S5nURSU5nURSU5n[US9nU"X5nU"X5nUU-S- n[UU U U RU RS 9$) NTrTrrr+rr) ignore_indexrq)r start_logits end_logitsrrr) rrmsplitrbrylenrDclamprrrr)rGrLrlr-r)rMrorprnr-r rFrsrtr0 ignored_indexr1 start_lossend_losss rIr["AlbertForQuestionAnswering.forward1sx++ ))%'   "!*#?#)<<r<#: #++B/::< ''+668   &=+D?'')*Q."1"9"9""==%%'(1, - 5 5b 9 (--a0M-33A}EO)//=AM']CH!,@J :H$x/14J+%!!//))   rK)rrArmr5r]r^r_r`r r0rrr@rbrcrrrrr[rerfrgs@rIrkrk%s|.23726042637153 ##d*3 ))D03 ((4/ 3 &&- 3 ((4/ 3 ))D03 ''$.3 +,3  $e +3 3 rKrkc&^\rSrSrS\4U4Sjjr\\SS\RS-S\RS-S\RS-S\RS-S \RS-S \RS-S \ \ S \ \-4S jj55rSrU=r$)AlbertForMultipleChoiceiir$c>[TU]U5 [U5Ul[R "UR 5Ul[R"URS5Ul UR5 g)Nr) r/r0rrrr<r@r>rrrBrrFs rIr0 AlbertForMultipleChoice.__init__ksV  !&) zz&"@"@A ))F$6$6: rKNrLrlr-r)rMr(rnrNc UbURSOURSnUb!URSURS55OSnUb!URSURS55OSnUb!URSURS55OSnUb!URSURS55OSnUb1URSURS5URS55OSnUR"U4UUUUSS.UD6n U Sn UR U 5n UR U 5n U RSU5n Sn Ub[ 5nU"X5n [U U U RU RS9$)a input_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.__call__`] and [`PreTrainedTokenizer.encode`] for details. [What are input IDs?](../glossary#input-ids) token_type_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) position_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) inputs_embeds (`torch.FloatTensor` of shape `(batch_size, num_choices, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where *num_choices* is the size of the second dimension of the input tensors. (see *input_ids* above) Nrr+Tr+rU) rTrrDrr>rBrrrr)rGrLrlr-r)rMr(rn num_choicesr-r rFreshaped_logitsrr1s rIr[AlbertForMultipleChoice.forwardusT-6,Aiooa(}GZGZ[\G] >G>SINN2y~~b'9:Y] M[Mg,,R1D1DR1HImqM[Mg,,R1D1DR1HImqGSG_|((\->->r-BCei (   r=#5#5b#9=;M;Mb;Q R  ++  ))%'      ]3 #}= ++b+6  ')HO4D("!//))   rK)rrBr>rXr|rgs@rIr~r~is|.237260426*.M ##d*M ))D0M ((4/ M &&- M ((4/ M   4'M +,M  $e +M M rKr~)rrrrIrZrerkr~)Nr)Eracollections.abcr dataclassesrr@rtorch.nnrrrr r activationsr masking_utilsr modeling_outputsr rrrrrrmodeling_utilsrrprocessing_utilsr pytorch_utilsrutilsrrrr utils.genericrrutils.output_capturingrconfiguration_albertr get_loggerr]loggerModuler"rdfloatr|r~rrrrrrrrrrIrZrerkr~__all__rrKrIrs$! AA&!6G&NMI5.   H %=ryy=N!% II% <<% % << % LL4' % T\ %%'(%8>)bii>)B#"))#Lryy"@ @:/O//D  77  7&G 'G G T \ 0\  \ ~!BII!* BII _ -_ _ D J &;J  J Z: #8: : z@ !6@ @ FZ 3Z Z z rK