ZjPSrSSKJr SSKJr SSKrSSKJr SSKJrJ r J r SSK J r SS KJrJr \"5(aSS KJr SS KJr SS KJr SS KJrJrJrJrJrJrJrJrJ r SSK!J"r" SSK#J$r$ SSKJ%r% SSK&J'r' \%RP"\)5r*Sr+Sr,Sr-"SS\R\5r/"SS\R\5r0"SS\R\5r1"SS\R\5r2"SS\R\5r3"S S!\R\5r4"S"S#\5r5"S$S%\R\5r6"S&S'\R\5r7"S(S)\R\5r8"S*S+\R\5r9"S,S-\R\5r:"S.S/\R\5r;"S0S1\R\5r<\"S2S3\"55r=\"S4S59\"S6S7\555r>\"S8S9\=55r?\"S:S59"S;S<\=55r@\"S=S>\=55rA\"S?S59"S@SA\=55rB\"SBS59"SCSD\=55rC\"SESF\=55rD\"SGSH\=55rE\"SISJ\=55rF/SKQrGg)LzPyTorch FNet model.) dataclass)partialN)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)auto_docstringis_scipy_available)linalg)ACT2FN)GradientCheckpointingLayer) BaseModelOutputBaseModelOutputWithPoolingMaskedLMOutput ModelOutputMultipleChoiceModelOutputNextSentencePredictorOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel)apply_chunking_to_forward)logging) FNetConfigcURSnUSU2SU24nUR[R5n[R"SXU5$)z4Applies 2D matrix multiplication to 3D input arrays.rNzbij,jk,ni->bnk)shapetypetorch complex64einsum)xmatrix_dim_onematrix_dim_two seq_lengths w/root/GenerationalWealth/GenerationalWealth/venv/lib/python3.13/site-packages/transformers/models/fnet/modeling_fnet.py_two_dim_matmulr)5sLJ#KZK*$<=N uA <<(!^ LLc[XU5$N)r))r$r%r&s r(two_dim_matmulr->s 1n ==r*cUn[[UR5SS5H n[RR XS9nM" U$)z Applies n-dimensional Fast Fourier Transform (FFT) to input array. Args: x: Input n-dimensional array. Returns: n-dimensional Fourier transform of input n-dimensional array. rN)axis)reversedrangendimr!fft)r$outr/s r(fftnr5Cs@ Cqvvqr*+iimmCm+, Jr*c6^\rSrSrSrU4SjrSSjrSrU=r$)FNetEmbeddingsSzGConstruct the embeddings from word, position and token_type embeddings.cj>[TU]5 [R"URUR UR S9Ul[R"URUR 5Ul [R"URUR 5Ul [R"UR URS9Ul [R"UR UR 5Ul[R "UR"5UlUR'S[(R*"UR5R-S5SS9 UR'S[(R."UR0R35[(R4S9SS9 g) N) padding_idxeps position_idsrF) persistenttoken_type_idsdtype)super__init__r Embedding vocab_size hidden_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingstype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsLinear projectionDropouthidden_dropout_probdropoutregister_bufferr!arangeexpandzerosr=sizelongselfconfig __class__s r(rEFNetEmbeddings.__init__Vs: !||F,=,=v?Q?Q_e_r_rs#%<<0N0NPVPbPb#c %'\\&2H2H&J\J\%]"f&8&8f>S>ST))F$6$68J8JKzz&"<"<=   ELL)G)GHOOPWXej    ekk$*;*;*@*@*B%**Ubg  r*cbUbUR5nOUR5SSnUSnUcURSS2SU24nUcv[US5(a-URSS2SU24nUR USU5nUnO8[ R "U[ RURRS9nUcURU5nURU5n XI-n URU5n X- n URU 5n URU 5n URU 5n U $)Nr?rrArrCdevice)rZr=hasattrrArXr!rYr[rcrJrNrLrOrRrU) r] input_idsrAr= inputs_embeds input_shaper'buffered_token_type_ids buffered_token_type_ids_expandedrN embeddingsrLs r(forwardFNetEmbeddings.forwardjs:  #..*K',,.s3K ^  ,,Q  ^ J": "66|D) ^^J/ __Z0 \\*- r*)rOrUrLrRrNrJ)NNNN) __name__ __module__ __qualname____firstlineno____doc__rErk__static_attributes__ __classcell__r_s@r(r7r7SsQ (!!r*r7c4^\rSrSrU4SjrSrSrSrU=r$)FNetBasicFourierTransformcD>[TU]5 URU5 gr,)rDrE_init_fourier_transformr\s r(rE"FNetBasicFourierTransform.__init__s  $$V,r*cUR(d([[RRSS9UlgUR S::a[5(aURS[R"[R"UR5[RS95 URS[R"[R"UR5[RS95 [[UR UR"S9Ulg[$R&"S5 [Ulg[Ulg) N)rdimdft_mat_hiddenrB dft_mat_seq)r%r&zpSciPy is needed for DFT matrix calculation and is not found. Using TPU optimized fast fourier transform instead.)use_tpu_fourier_optimizationsrr!r3r5fourier_transformrKr rVtensorr dftrHr"tpu_short_seq_lengthr-rrrwarning)r]r^s r(ry1FNetBasicFourierTransform._init_fourier_transforms33%,UYY^^%HD "  + +t 3!##$$$ell6::f>P>P3QY^YhYh&i$$!5<< 6;V;V0W_d_n_n#o*1"43C3CTXTgTg*&**.&%)D "r*c>URU5RnU4$r,)rreal)r] hidden_statesoutputss r(rk!FNetBasicFourierTransform.forwards" ((7<<zr*)r) rmrnrorprEryrkrrrsrts@r(rvrvs-*.r*rvc.^\rSrSrU4SjrSrSrU=r$)FNetBasicOutputc~>[TU]5 [R"URUR S9UlgNr;)rDrErrOrHrPr\s r(rEFNetBasicOutput.__init__s, f&8&8f>S>STr*c,URX!-5nU$r,rOr]r input_tensors r(rkFNetBasicOutput.forwards|'CD r*rrmrnrorprErkrrrsrts@r(rrsUr*rc.^\rSrSrU4SjrSrSrU=r$)FNetFourierTransformcb>[TU]5 [U5Ul[ U5Ulgr,)rDrErvr]routputr\s r(rEFNetFourierTransform.__init__s& -f5 %f- r*cXURU5nURUSU5nU4nU$Nr)r]r)r]r self_outputsfourier_outputrs r(rkFNetFourierTransform.forwards1yy/ \!_mD!#r*)rr]rrts@r(rrs. r*rcb^\rSrSrU4SjrS\R S\R 4SjrSrU=r $)FNetIntermediatec>[TU]5 [R"URUR 5Ul[UR[5(a[URUl gURUl gr,) rDrErrQrHintermediate_sizedense isinstance hidden_actstrrintermediate_act_fnr\s r(rEFNetIntermediate.__init__s` YYv1163K3KL f'' - -'-f.?.?'@D $'-'8'8D $r*rreturncJURU5nURU5nU$r,rrr]rs r(rkFNetIntermediate.forwards& =1 00? r*r rmrnrorprEr!Tensorrkrrrsrts@r(rrs(9U\\ellr*rcz^\rSrSrU4SjrS\R S\R S\R 4SjrSrU=r $) FNetOutputc(>[TU]5 [R"URUR 5Ul[R"UR URS9Ul[R"UR5Ul gr) rDrErrQrrHrrOrPrSrTrUr\s r(rEFNetOutput.__init__s` YYv779K9KL f&8&8f>S>STzz&"<"<= r*rrrcpURU5nURU5nURX-5nU$r,)rrUrOrs r(rkFNetOutput.forwards5 =1  ]3 }'CD r*)rOrrUrrts@r(rrs6> U\\RWR^R^r*rc4^\rSrSrU4SjrSrSrSrU=r$) FNetLayerc>[TU]5 URUlSUl[ U5Ul[ U5Ul[U5Ul gNr) rDrEchunk_size_feed_forward seq_len_dimrfourierr intermediaterrr\s r(rEFNetLayer.__init__sI '-'E'E$+F3 ,V4 ( r*cURU5nUSn[URURURU5nU4nU$r)rrfeed_forward_chunkrr)r]rself_fourier_outputsr layer_outputrs r(rkFNetLayer.forwardsO#||M:-a00  # #T%A%A4CSCSUc  /r*cJURU5nURX!5nU$r,)rr)r]rintermediate_outputrs r(rFNetLayer.feed_forward_chunks("//?{{#6G r*)rrrrr) rmrnrorprErkrrrrsrts@r(rrs) r*rc2^\rSrSrU4SjrSSjrSrU=r$) FNetEncoderic>[TU]5 Xl[R"[ UR 5Vs/sHn[U5PM sn5UlSUl gs snf)NF) rDrEr^r ModuleListr1num_hidden_layersrlayergradient_checkpointing)r]r^_r_s r(rEFNetEncoder.__init__sR  ]]uVE]E]?^#_?^!If$5?^#_` &+#$`sA&cU(aSOSn[UR5HupVU(aXA4-nU"U5nUSnM U(aXA4-nU(d[SX455$[XS9$)Nrc3.# UH ocMUv M g7fr,r).0vs r( &FNetEncoder.forward..sX$Fq$Fs )last_hidden_stater) enumeratertupler)r]routput_hidden_states return_dictall_hidden_statesi layer_module layer_outputss r(rkFNetEncoder.forward sw"6BD(4OA#$58H$H!(7M)!,M 5 14D D X]$FXX X``r*)r^rr)FTrrts@r(rrs, aar*rcb^\rSrSrU4SjrS\R S\R 4SjrSrU=r $) FNetPooleri!c>[TU]5 [R"URUR5Ul[R "5Ulgr,)rDrErrQrHrTanh activationr\s r(rEFNetPooler.__init__"s9 YYv1163E3EF '')r*rrc\USS2S4nURU5nURU5nU$r)rr)r]rfirst_token_tensor pooled_outputs r(rkFNetPooler.forward's6+1a40 #56  6 r*)rrrrts@r(rr!s($ U\\ellr*rcb^\rSrSrU4SjrS\R S\R 4SjrSrU=r $)FNetPredictionHeadTransformi1cp>[TU]5 [R"URUR5Ul[ UR[5(a[URUl OURUl [R"URURS9Ul gr) rDrErrQrHrrrrrtransform_act_fnrOrPr\s r(rE$FNetPredictionHeadTransform.__init__2s~ YYv1163E3EF f'' - -$*6+<+<$=D !$*$5$5D !f&8&8f>S>STr*rrclURU5nURU5nURU5nU$r,)rrrOrs r(rk#FNetPredictionHeadTransform.forward;s4 =1 --m< }5 r*)rOrrrrts@r(rr1s)UU\\ellr*rc.^\rSrSrU4SjrSrSrU=r$)FNetLMPredictionHeadiBc>[TU]5 [U5Ul[R "UR UR5Ul[R"[R"UR55Ul gr,) rDrEr transformrrQrHrGdecoder Parameterr!rYbiasr\s r(rEFNetLMPredictionHead.__init__CsW 4V<yy!3!3V5F5FG LLV->->!?@ r*cJURU5nURU5nU$r,)rrrs r(rkFNetLMPredictionHead.forwardIs$}5  ]3 r*)rrrrrts@r(rrBsA r*rc.^\rSrSrU4SjrSrSrU=r$)FNetOnlyMLMHeadiOcB>[TU]5 [U5Ulgr,)rDrEr predictionsr\s r(rEFNetOnlyMLMHead.__init__Ps /7r*c(URU5nU$r,r)r]sequence_outputprediction_scoress r(rkFNetOnlyMLMHead.forwardTs ,,_=  r*r rrts@r(rrOs8!!r*rc.^\rSrSrU4SjrSrSrU=r$)FNetOnlyNSPHeadiZcn>[TU]5 [R"URS5UlgNr|)rDrErrQrHseq_relationshipr\s r(rEFNetOnlyNSPHead.__init__[s'  " &*<*[TU]5 [U5Ul[R "UR S5Ulgr)rDrErrrrQrHrr\s r(rEFNetPreTrainingHeads.__init__fs4 /7 " &*<*[TU]U5 [U[5(a|[R "UR [R"UR RS5RS55 [R"UR5 gg)Nr?r>) rD _init_weightsrr7initcopy_r=r!rWrrXzeros_rA)r]moduler_s r(r$!FNetPreTrainedModel._init_weightswso f% fn - - JJv**ELL9L9L9R9RSU9V,W,^,^_f,g h KK-- . .r*r) rmrnrorpr__annotations__base_model_prefixsupports_gradient_checkpointingr$rrrsrts@r(r!r!qs &*#//r*r!z0 Output type of [`FNetForPreTraining`]. ) custom_introc\rSrSr%SrSr\RS-\S'Sr \RS-\S'Sr \RS-\S'Sr \ \RS-\S'Sr g) FNetForPreTrainingOutputi~ar loss (*optional*, returned when `labels` is provided, `torch.FloatTensor` of shape `(1,)`): Total loss as the sum of the masked language modeling loss and the next sequence prediction (classification) loss. prediction_logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). seq_relationship_logits (`torch.FloatTensor` of shape `(batch_size, 2)`): Prediction scores of the next sequence prediction (classification) head (scores of True/False continuation before SoftMax). Nlossprediction_logitsseq_relationship_logitsrr)rmrnrorprqr0r! FloatTensorr*r1r2rrrrrr*r(r/r/~sd &*D%  d ")26u((4/68<U..5<59M5**+d29r*r/c^\rSrSrSrSU4SjjrSrSr\SS\ RS-S\ RS-S \ RS-S \ RS-S \ S-S \ S-S \ \-4Sjj5rSrU=r$) FNetModeliz The model can behave as an encoder, following the architecture described in [FNet: Mixing Tokens with Fourier Transforms](https://huggingface.co/papers/2105.03824) by James Lee-Thorp, Joshua Ainslie, Ilya Eckstein, Santiago Ontanon. c>[TU]U5 Xl[U5Ul[ U5UlU(a [U5OSUlUR5 g)z^ add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer N) rDrEr^r7rjrencoderrpooler post_init)r]r^add_pooling_layerr_s r(rEFNetModel.__init__sK   (0"6* ,=j(4  r*c.URR$r,rjrJr]s r(get_input_embeddingsFNetModel.get_input_embeddingss...r*c$XRlgr,r=)r]values r(set_input_embeddingsFNetModel.set_input_embeddingss */'r*NrerAr=rfrrrc UbUOURRnUbUOURRnUbUb [S5eUbUR 5nUupO&UbUR 5SSnUupO [S5eURR (a+U S::a%URR U :wa [S5eUb URO URn Ucr[URS5(a3URRSS2SU 24n U RX5n U nO$[R"U[RU S9nURUUUUS9nURUUUS 9nUS nUR bUR!U5OSnU(d UU4US S-$[#UUUR$S 9$) NzDYou cannot specify both input_ids and inputs_embeds at the same timer?z5You have to specify either input_ids or inputs_embedsrzThe `tpu_short_seq_length` in FNetConfig should be set equal to the sequence length being passed to the model when using TPU optimizations.rArb)rer=rArf)rrrr)r pooler_outputr)r^rr ValueErrorrZrrrcrdrjrArXr!rYr[r7r8rr)r]rerAr=rfrrkwargsrg batch_sizer'rcrhriembedding_outputencoder_outputsr rFs r(rkFNetModel.forwards%9$D $++JjJj &1%cd d  "#..*K%0 "J  &',,.s3K%0 "J TU U KK 5 5d" 00J>;  &/%:!! @T@T  !t(899*.//*H*HKZK*X'3J3Q3QR\3i0!A!&[ SY!Z??%)' +  ,, !5#'  *!,8< 8O O4UY #]3oab6II I)-')77  r*)r^rjr7r8)T)NNNNNN)rmrnrorprqrEr?rCr r! LongTensorr3boolrrrkrrrsrts@r(r5r5s /0.2260426,0#'D ##d*D ((4/D &&- D ((4/ D #Tk D D[D   D D r*r5z FNet Model with two heads on top as done during the pretraining: a `masked language modeling` head and a `next sentence prediction (classification)` head. c8^\rSrSrSSS.rU4SjrSrSr\SS \ RS-S \ RS-S \ RS-S \ RS-S \ RS-S\ RS-S\ S-S\ S-S\ \ -4Sjj5rSrU=r$)FNetForPreTrainingicls.predictions.bias&fnet.embeddings.word_embeddings.weightzcls.predictions.decoder.biaszcls.predictions.decoder.weightc>[TU]U5 [U5Ul[ U5UlUR 5 gr,)rDrEr5r"rclsr9r\s r(rEFNetForPreTraining.__init__ s4  f% '/ r*cBURRR$r,rUrrr>s r(get_output_embeddings(FNetForPreTraining.get_output_embeddingsxx##+++r*cXRRlURURRlgr,rUrrrr]new_embeddingss r(set_output_embeddings(FNetForPreTraining.set_output_embeddings*'5$$2$7$7!r*NrerAr=rflabelsnext_sentence_labelrrrc UbUOURRnURUUUUUUS9n U SSupURX5upSnUbvUbs[ 5nU"U R SURR 5UR S55nU"UR SS5UR S55nUU-nU(dX4U SS-nUbU4U-$U$[UU UU RS9$)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` next_sentence_label (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the next sequence prediction (classification) loss. Input should be a sequence pair (see `input_ids` docstring) Indices should be in `[0, 1]`: - 0 indicates sequence B is a continuation of sequence A, - 1 indicates sequence B is a random sequence. Example: ```python >>> from transformers import AutoTokenizer, FNetForPreTraining >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("google/fnet-base") >>> model = FNetForPreTraining.from_pretrained("google/fnet-base") >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> prediction_logits = outputs.prediction_logits >>> seq_relationship_logits = outputs.seq_relationship_logits ```NrAr=rfrrr|r?)r0r1r2r) r^rr"rUrviewrGr/r)r]rerAr=rfrcrdrrrHrr rr r total_lossloss_fctmasked_lm_lossnext_sentence_lossrs r(rkFNetForPreTraining.forwards+L&1%[TU]U5 [U5Ul[ U5UlUR 5 gr,)rDrEr5r"rrUr9r\s r(rEFNetForMaskedLM.__init__g4  f% "6* r*cBURRR$r,rXr>s r(rY%FNetForMaskedLM.get_output_embeddingspr[r*cXRRlURURRlgr,r]r^s r(r`%FNetForMaskedLM.set_output_embeddingssrbr*NrerAr=rfrcrrrc UbUOURRnURUUUUUUS9n U Sn URU 5n Sn UbF[ 5n U "U R SURR 5UR S55n U(dU 4U SS-nU bU 4U-$U$[XU RS9$)a{ labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Nrfrr?r|r0logitsr) r^rr"rUrrgrGrr)r]rerAr=rfrcrrrHrr r rjrirs r(rkFNetForMaskedLM.forwardws$&1%[b[p[pqqr*rmNNNNNNN)rmrnrorprorErYr`r r!rrNrrrkrrrsrts@r(rqrq`s)?*R ,8*..2,0-1&*,0#'(r<<$&(r t+(rllT) (r ||d* (r  t# (r#Tk(rD[(r  (r(rr*rqzT FNet Model with a `next sentence prediction (classification)` head on top. c^\rSrSrU4Sjr\SS\RS-S\RS-S\RS-S\RS-S\RS-S \S-S \S-S \ \ -4S jj5r S r U=r $)FNetForNextSentencePredictionic>[TU]U5 [U5Ul[ U5UlUR 5 gr,)rDrEr5r"rrUr9r\s r(rE&FNetForNextSentencePrediction.__init__rtr*NrerAr=rfrcrrrc ZUbUOURRnURUUUUUUS9n U Sn URU 5n Sn Ub2[ 5n U "U R SS5UR S55n U(dU 4U SS-nU bU 4U-$U$[ U U U RS9$)a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the next sequence prediction (classification) loss. Input should be a sequence pair (see `input_ids` docstring). Indices should be in `[0, 1]`: - 0 indicates sequence B is a continuation of sequence A, - 1 indicates sequence B is a random sequence. Example: ```python >>> from transformers import AutoTokenizer, FNetForNextSentencePrediction >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("google/fnet-base") >>> model = FNetForNextSentencePrediction.from_pretrained("google/fnet-base") >>> prompt = "In Italy, pizza served in formal settings, such as at a restaurant, is presented unsliced." >>> next_sentence = "The sky is blue due to the shorter wavelength of blue light." >>> encoding = tokenizer(prompt, next_sentence, return_tensors="pt") >>> outputs = model(**encoding, labels=torch.LongTensor([1])) >>> logits = outputs.logits >>> assert logits[0, 0] < logits[0, 1] # next sentence was random ```Nrfrr?r|rz)r^rr"rUrrgrr)r]rerAr=rfrcrrrHrrseq_relationship_scoresrkrirs r(rk%FNetForNextSentencePrediction.forwardsH&1%[TU]U5 URUl[U5Ul[ R "UR5Ul[ R"URUR5Ul UR5 gr, rDrE num_labelsr5r"rrSrTrUrQrH classifierr9r\s r(rE&FNetForSequenceClassification.__init__si   ++f% zz&"<"<= ))F$6$68I8IJ r*NrerAr=rfrcrrrc 4UbUOURRnURUUUUUUS9n U Sn URU 5n UR U 5n Sn UGbURR cUR S:XaSURlOoUR S:aNUR[R:XdUR[R:XaSURlOSURlURR S:XaI[5n UR S:Xa&U "U R5UR55n OU "X5n OURR S:Xa=[5n U "U RSUR 5URS55n O,URR S:Xa[5n U "X5n U(dU 4U SS-nU bU 4U-$U$[!XU R"S 9$) ae labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Nrfr regressionsingle_label_classificationmulti_label_classificationr?r|rz)r^rr"rUr problem_typerrCr!r[intrsqueezerrgrrr)r]rerAr=rfrcrrrHrrr{r0rirs r(rk%FNetForSequenceClassification.forwards$&1%[TU]U5 [U5Ul[R "UR 5Ul[R"URS5Ul UR5 gr) rDrEr5r"rrSrTrUrQrHrr9r\s r(rEFNetForMultipleChoice.__init__GsV  f% zz&"<"<= ))F$6$6: r*NrerAr=rfrcrrrc UbUOURRnUbURSOURSn Ub!URSUR S55OSnUb!URSUR S55OSnUb!URSUR S55OSnUb1URSUR S5UR S55OSnUR UUUUUUS9n U Sn UR U 5n URU 5n U RSU 5n SnUb[5nU"X5nU(dU 4U SS-nUbU4U-$U$[XU RS9$)a input_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) token_type_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) position_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) inputs_embeds (`torch.FloatTensor` of shape `(batch_size, num_choices, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert *input_ids* indices into associated vectors than the model's internal embedding lookup matrix. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) Nrr?rfr|rz) r^rrrgrZr"rUrrrr)r]rerAr=rfrcrrrH num_choicesrrr{reshaped_logitsr0rirs r(rkFNetForMultipleChoice.forwardQsT&1%G>SINN2y~~b'9:Y] M[Mg,,R1D1DR1HImqGSG_|((\->->r-BCei (   r=#5#5b#9=;M;Mb;Q R  )) )%'!5#     ]3 / ++b+6  ')HO4D%''!"+5F)-)9TGf$ Ev E(dZaZoZoppr*)rrUr"r})rmrnrorprEr r!rrNrrrkrrrsrts@r(rrEs*..2,0-1&*,0#'Mq<<$&Mq t+MqllT) Mq ||d* Mq  t# Mq#TkMqD[Mq * *MqMqr*rc^\rSrSrU4Sjr\SS\RS-S\RS-S\RS-S\RS-S\RS-S \S-S \S-S \ \ -4S jj5r S r U=r $)FNetForTokenClassificationic0>[TU]U5 URUl[U5Ul[ R "UR5Ul[ R"URUR5Ul UR5 gr,rr\s r(rE#FNetForTokenClassification.__init__si   ++f% zz&"<"<= ))F$6$68I8IJ r*NrerAr=rfrcrrrc UbUOURRnURUUUUUUS9n U Sn URU 5n UR U 5n Sn Ub<[ 5n U "U R SUR5UR S55n U(dU 4U SS-nU bU 4U-$U$[XU RS9$)z labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. Nrfrr?r|rz) r^rr"rUrrrgrrr)r]rerAr=rfrcrrrHrr r{r0rirs r(rk"FNetForTokenClassification.forwards &1%[TU]U5 URUl[U5Ul[ R "URUR5UlUR5 gr,) rDrErr5r"rrQrH qa_outputsr9r\s r(rE!FNetForQuestionAnswering.__init__sS   ++f% ))F$6$68I8IJ r*NrerAr=rfstart_positions end_positionsrrrc UbUOURRnURUUUUUUS9n U Sn URU 5n U R SSS9upU R S5R 5n UR S5R 5nSnUbUb[UR55S:aUR S5n[UR55S:aUR S5nU RS5nURSU5nURSU5n[US9nU"X5nU"X5nUU-S- nU(dX4U SS-nUbU4U-$U$[XXRS9$) Nrfrrr?r}) ignore_indexr|)r0 start_logits end_logitsr) r^rr"rsplitr contiguouslenrZclamprrr)r]rerAr=rfrrrrrHrr r{rrrh ignored_indexri start_lossend_lossrs r(rk FNetForQuestionAnswering.forwards&1%rs! AA&7!9   .6*   H %M>  8RYY8v# #Lbii 299  ryy   *6a"))a8   "))" 299 !bii!&bii& 9299 9 // / /  :{:  :$c #c c L Z ,Z  Z z?r)?r?rD J $7J  J Z Gg$7Gg  GgTYq/YqYqx7d!47d7dtB 2B B J r*