ZjSrSSKrSSKJr SSKJrJrJr SSKJr SSK J r J r J r SSKJrJrJrJrJrJrJrJr SSKJr SS KJr SS KJrJrJr SS KJ r J!r! SS K"J#r# S SK$J%r%J&r&J'r'J(r(J)r)J*r*J+r+J,r,J-r-J.r.J/r/J0r0J1r1J2r2J3r3J4r4J5r5 SSK6J7r7 \Rp"\95r:"SS\&5r;"SS\55r<"SS\%5r="SS\05r>"SS\45r?"SS\25r@"SS\'5rA\"SS \55rB"S!S"\35rC"S#S$\,5rD"S%S&\+5rE"S'S(\15rF"S)S*\(5rG"S+S,\*5rH"S-S.\.5rI"S/S0\)5rJ"S1S2\/5rK"S3S4\-5rL/S5QrMg)6zPyTorch ERNIE model.N)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)Cache DynamicCacheEncoderDecoderCache),BaseModelOutputWithPoolingAndCrossAttentions!CausalLMOutputWithCrossAttentionsMaskedLMOutputMultipleChoiceModelOutputNextSentencePredictorOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel)Unpack)TransformersKwargsauto_docstringlogging)can_return_tuplemerge_with_config_defaults)capture_outputs)BertCrossAttentionBertEmbeddings BertEncoderBertForMaskedLMBertForMultipleChoiceBertForNextSentencePredictionBertForPreTrainingBertForPreTrainingOutputBertForQuestionAnsweringBertForSequenceClassificationBertForTokenClassification BertLayerBertLMHeadModelBertLMPredictionHead BertModel BertPoolerBertSelfAttention) ErnieConfigc^\rSrSrSrU4SjrSS\RS-S\RS-S\RS-S\RS-S \RS-S \ S \R4S jjr S r U=r $)ErnieEmbeddings>zGConstruct the embeddings from word, position and token_type embeddings.c>[TU]U5 URUlUR(a1[R"UR UR 5Ulgg)N)super__init__ use_task_idnn Embeddingtask_type_vocab_size hidden_sizetask_type_embeddings)selfconfig __class__s x/root/GenerationalWealth/GenerationalWealth/venv/lib/python3.13/site-packages/transformers/models/ernie/modular_ernie.pyr4ErnieEmbeddings.__init__AsL  !--   (* V5P5PRXRdRd(eD % N input_idstoken_type_ids task_type_ids position_ids inputs_embedspast_key_values_lengthreturncUbUR5nOUR5SSnUupUcURSS2XiU-24nUc[US5(aQURR UR SS5n [ R"U SUS9n U R X5nO8[ R"U[ RURRS9nUcURU5nURU5n URU R5nX[-n URU5n X-n UR(aPUc8[ R"U[ RURRS9nUR!U5nX- n UR#U 5n UR%U 5n U $)NrBrr-)dimindex)dtypedevice)sizerDhasattrrBexpandshapetorchgatherzeroslongrMword_embeddingstoken_type_embeddingstoposition_embeddingsr5r: LayerNormdropout)r;rArBrCrDrErF input_shape batch_size seq_lengthbuffered_token_type_idsrW embeddingsrYr:s r>forwardErnieEmbeddings.forwardHs  #..*K',,.s3K!,  ,,Q0FVlIl0l-lmL  !t-..*.*=*=*D*D\EWEWXYEZ\^*_'*/,,7NTU]i*j'!8!?!? !W!&[ SWSdSdSkSk!l   00;M $ : :> J&(()>)E)EF ": "66|D5    $ % KuzzRVRcRcRjRj k #'#<#<]#K  .J^^J/ \\*- r@)r:r5)NNNNNr)__name__ __module__ __qualname____firstlineno____doc__r4rR LongTensor FloatTensorintTensorra__static_attributes__ __classcell__r=s@r>r0r0>sQf.226150426&'3##d*3((4/3''$. 3 &&- 3 ((4/ 3!$3 33r@r0c\rSrSrSrg)ErnieSelfAttention~Nrcrdrerfrlrrr@r>rprp~r@rpc\rSrSrSrg)ErnieCrossAttentionrrNrsrrr@r>rvrvrtr@rvc\rSrSrSrg) ErnieLayerrrNrsrrr@r>ryryrtr@ryc\rSrSrSrg) ErniePoolerrrNrsrrr@r>r|r|rtr@r|c\rSrSrSrg)ErnieLMPredictionHeadrrNrsrrr@r>rrrtr@rc\rSrSrSrg) ErnieEncoderrrNrsrrr@r>rrrtr@rcx^\rSrSr\rSrSrSrSr Sr Sr \ \ \S.r\R""5U4Sj5rSrU=r$)ErniePreTrainedModelernieT) hidden_states attentionscross_attentionsc>[TU]U5 [U[5(a![R "UR 5 g[U[5(a|[R"UR[R"URRS5RS55 [R "UR5 gg)zInitialize the weightsrI)r-rIN)r3 _init_weights isinstancerinitzeros_biasr0copy_rDrRarangerQrPrB)r;moduler=s r>r"ErniePreTrainedModel._init_weightss f% f3 4 4 KK $  0 0 JJv**ELL9L9L9R9RSU9V,W,^,^_f,g h KK-- .1r@rr)rcrdrerfr. config_classbase_model_prefixsupports_gradient_checkpointing_supports_flash_attn_supports_sdpa_supports_flex_attn_supports_attention_backendryrprv_can_record_outputsrRno_gradrrlrmrns@r>rrsVL&*#N"&#(/  ]]_//r@rc^\rSrSrS/rSU4Sjjr\\\SS\ RS-S\ RS-S\ RS-S\ RS-S \ RS-S \ RS-S \ RS-S \ RS-S \ S-S\ S-S\ \S\\ R\-4Sjj555rSrU=r$) ErnieModelryc>[TU]X5 XlSUl[ U5Ul[ U5UlU(a [U5OSUl UR5 g)NF) r3r4r<gradient_checkpointingr0r`rencoderr|pooler post_init)r;r<add_pooling_layerr=s r>r4ErnieModel.__init__sQ & &+#)&1#F+ ->k&)D  r@NrAattention_maskrBrCrDrEencoder_hidden_statesencoder_attention_maskpast_key_values use_cachekwargsrGc USLUSL- (a [S5eURR(aU bU OURRn OSn U (abU c_UcURR(a.[ [ URS9[ URS95O[ URS9n U bU R5OSn URUUUUUU S9n URUUU UU S9up(UR"U 4UUUU U US.U D6nUSnURbURU5OSn[UUURS 9$)  task_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Task type embedding is a special embedding to represent the characteristic of different tasks, such as word-aware pre-training task, structure-aware pre-training task and semantic-aware pre-training task. We assign a `task_type_id` to each task and the `task_type_id` is in the range `[0, config.task_type_vocab_size-1] Nz:You must specify exactly one of input_ids or inputs_embedsF)r<r)rArDrBrCrErF)rrembedding_outputrr)rrrrrrD)last_hidden_state pooler_outputr) ValueErrorr< is_decoderris_encoder_decoderr r get_seq_lengthr`_create_attention_masksrrr r)r;rArrBrCrDrErrrrrrFrencoder_outputssequence_output pooled_outputs r>raErnieModel.forwardsy0  -t"; <YZ Z ;; ! !%.%:  @U@UII 0)4 8V8V$L $DlZ^ZeZeFfg!5  FUE`!?!?!Afg??%)''#9+ 261M1M)#9-"7+ 2N2 .,,   )"7#9+%    *!,8< 8O O4UY ;-'+;;  r@)r<r`rrr)T) NNNNNNNNNN)rcrdrerf_no_split_modulesr4rrrrRrkrboolrrtupler rarlrmrns@r>rrs=%  *..2.2-1,0-1596:(,!%H <<$&H  t+H  t+ H ||d* H llT) H ||d*H  %||d2H !& t 3H H $;H +,H  u|| K KH  H r@rc\rSrSrSrg)ErnieForPreTrainingOutputirrNrsrrr@r>rrrtr@rcn\rSrSrSSS.r\\SS\RS-S\RS-S\RS-S \RS-S \RS-S \RS-S \RS-S \RS-S\ \ S\ \R\ -4Sjj55r Srg)ErnieForPreTrainingicls.predictions.bias'ernie.embeddings.word_embeddings.weightzcls.predictions.decoder.biaszcls.predictions.decoder.weightNrArrBrCrDrElabelsnext_sentence_labelrrGc UR"U4UUUUUSS.U D6n U SSupURX5upSnUbvUbs[5nU"U RSURR 5URS55nU"URSS5URS55nUU-n[ UU UU RU RS9$)aj task_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Task type embedding is a special embedding to represent the characteristic of different tasks, such as word-aware pre-training task, structure-aware pre-training task and semantic-aware pre-training task. We assign a `task_type_id` to each task and the `task_type_id` is in the range `[0, config.task_type_vocab_size-1] labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` next_sentence_label (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the next sequence prediction (classification) loss. Input should be a sequence pair (see `input_ids` docstring) Indices should be in `[0, 1]`: - 0 indicates sequence B is a continuation of sequence A, - 1 indicates sequence B is a random sequence. Example: ```python >>> from transformers import AutoTokenizer, ErnieForPreTraining >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("nghuyong/ernie-1.0-base-zh") >>> model = ErnieForPreTraining.from_pretrained("nghuyong/ernie-1.0-base-zh") >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> prediction_logits = outputs.prediction_logits >>> seq_relationship_logits = outputs.seq_relationship_logits ``` TrrBrCrDrE return_dictNrrI)lossprediction_logitsseq_relationship_logitsrr) rclsrviewr< vocab_sizerrr)r;rArrBrCrDrErrroutputsrrprediction_scoresseq_relationship_score total_lossloss_fctmasked_lm_lossnext_sentence_losss r>raErnieForPreTraining.forwards^**   ))'%'    *1!&48HH_4\1  "5"A')H%&7&<&rrs(>*S *..2.2-1,0-1&*37H <<$&H  t+H  t+ H ||d* H llT) H ||d*H  t#H #\\D0H +,H  u|| 8 8H H r@rc \rSrSr\\SS\RS-S\RS-S\RS-S\RS-S\RS-S\RS-S \RS-S \RS-S \RS-S \\RS-S \ S-S\ \R-S\ \ S\ \R\-4Sjj55rSrg)ErnieForCausalLMieNrArrBrCrDrErrrrrlogits_to_keeprrGc  U bSn UR"U4UUUUUUUU U SS. U D6nURn[U [5(a [ U *S5OU nUR USS2USS245nSnU b)UR "SUXRRS.U D6n[UUURURURURS9$)a@ task_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Task type embedding is a special embedding to represent the characteristic of different tasks, such as word-aware pre-training task, structure-aware pre-training task and semantic-aware pre-training task. We assign a `task_type_id` to each task and the `task_type_id` is in the range `[0, config.task_type_vocab_size-1] labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the left-to-right language modeling loss (next word prediction). Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels n `[0, ..., config.vocab_size]` NFT) rrBrCrDrErrrrr)logitsrr)rrrrrrrr)rrrrjslicer loss_functionr<rr rrrr)r;rArrBrCrDrErrrrrrrrr slice_indicesrrs r>raErnieForCausalLM.forwardfs:  I@D  A ))'%'"7#9+ A  A  11 8B>SV8W8W~ot4]k -=!(;<=  %%pVF{{OeOepiopD0#33!//))$55   r@rr) NNNNNNNNNNNr)rcrdrerfrrrRrklistrrjrrrr rarlrrr@r>rresM*..2.2-1,0-1596:&*59!%-.= <<$&=  t+=  t+ = ||d* = llT) = ||d*=  %||d2= !& t 3=  t#= ell+d2= $;= ell*= +,=  u|| @ @= = r@rc\rSrSrSSS.r\\SS\RS-S\RS-S\RS-S \RS-S \RS-S \RS-S \RS-S \RS-S\RS-S\ \ S\ \R\ -4Sjj55r Srg)ErnieForMaskedLMirrrNrArrBrCrDrErrrrrGc <UR"U4UUUUUUUSS.U D6n U Sn URU 5n SnU bF[5nU"U RSURR 5U RS55n[ UU U RU RS9$)a# task_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Task type embedding is a special embedding to represent the characteristic of different tasks, such as word-aware pre-training task, structure-aware pre-training task and semantic-aware pre-training task. We assign a `task_type_id` to each task and the `task_type_id` is in the range `[0, config.task_type_vocab_size-1] labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` T)rrBrCrDrErrrrNrIrrrr) rrrrr<rr rr)r;rArrBrCrDrErrrrrrrrrs r>raErnieForMaskedLM.forwards4**   ))'%'"7#9    "!* HH_5  ')H%&7&<&rrs(>*S *..2.2-1,0-1596:&*2 <<$&2  t+2  t+ 2 ||d* 2 llT) 2 ||d*2  %||d22 !& t 32  t#2 +,2  u|| ~ -2 2 r@rcD\rSrSr\\SS\RS-S\RS-S\RS-S\RS-S\RS-S\RS-S \RS-S \\ S \ \R\ -4S jj55r S r g)ErnieForNextSentencePredictioniNrArrBrCrDrErrrGc UR"U4UUUUUSS.UD6n U Sn URU 5n Sn Ub2[5n U "U RSS5URS55n [ U U U R U R S9$)a task_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Task type embedding is a special embedding to represent the characteristic of different tasks, such as word-aware pre-training task, structure-aware pre-training task and semantic-aware pre-training task. We assign a `task_type_id` to each task and the `task_type_id` is in the range `[0, config.task_type_vocab_size-1] labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the next sequence prediction (classification) loss. Input should be a sequence pair (see `input_ids` docstring). Indices should be in `[0, 1]`: - 0 indicates sequence B is a continuation of sequence A, - 1 indicates sequence B is a random sequence. Example: ```python >>> from transformers import AutoTokenizer, ErnieForNextSentencePrediction >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("nghuyong/ernie-1.0-base-zh") >>> model = ErnieForNextSentencePrediction.from_pretrained("nghuyong/ernie-1.0-base-zh") >>> prompt = "In Italy, pizza served in formal settings, such as at a restaurant, is presented unsliced." >>> next_sentence = "The sky is blue due to the shorter wavelength of blue light." >>> encoding = tokenizer(prompt, next_sentence, return_tensors="pt") >>> outputs = model(**encoding, labels=torch.LongTensor([1])) >>> logits = outputs.logits >>> assert logits[0, 0] < logits[0, 1] # next sentence was random ``` Trr-NrIrr)rrrrrrr)r;rArrBrCrDrErrrrseq_relationship_scoresrrs r>ra&ErnieForNextSentencePrediction.forwardsZ**   ))'%'      "&((="9!  ')H!)*A*F*Fr1*Mv{{[]!_ *#*!//))   r@rrNNNNNNN)rcrdrerfrrrRrkrrrrrarlrrr@r>rrs*..2.2-1,0-1&*D <<$&D  t+D  t+ D ||d* D llT) D ||d*D  t#D +,D  u|| : :D D r@rcD\rSrSr\\SS\RS-S\RS-S\RS-S\RS-S\RS-S\RS-S \RS-S \\ S \ \R\ -4S jj55r S r g)ErnieForSequenceClassificationi/NrArrBrCrDrErrrGc UR"U4UUUUUSS.UD6n U Sn URU 5n URU 5n Sn UGbURRcUR S:XaSURlOoUR S:aNUR [R:XdUR [R:XaSURlOSURlURRS:XaI[5n UR S:Xa&U "U R5UR55n OU "X5n OURRS:Xa=[5n U "U RSUR 5URS55n O,URRS:Xa[5n U "X5n [U U U R U R"S 9$) a task_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Task type embedding is a special embedding to represent the characteristic of different tasks, such as word-aware pre-training task, structure-aware pre-training task and semantic-aware pre-training task. We assign a `task_type_id` to each task and the `task_type_id` is in the range `[0, config.task_type_vocab_size-1] labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Trr-N regressionsingle_label_classificationmulti_label_classificationrIr)rr[ classifierr< problem_type num_labelsrLrRrUrjrsqueezerrrrrr)r;rArrBrCrDrErrrrrrrs r>ra&ErnieForSequenceClassification.forward0s0**   ))'%'       ]3 /  {{''/??a'/;DKK,__q(fllejj.HFLL\a\e\eLe/LDKK,/KDKK,{{''<7"9??a'#FNN$4fnn6FGD#F3D))-JJ+- B @&++b/R))-II,./'!//))   r@rrr)rcrdrerfrrrRrkrrrrrarlrrr@r>rr/s*..2.2-1,0-1&*B <<$&B  t+B  t+ B ||d* B llT) B ||d*B  t#B +,B  u|| 7 7B B r@rcD\rSrSr\\SS\RS-S\RS-S\RS-S\RS-S\RS-S\RS-S \RS-S \\ S \ \R\ -4S jj55r S r g)ErnieForMultipleChoiceiwNrArrBrCrDrErrrGc UbURSOURSn Ub!URSURS55OSnUb!URSURS55OSnUb!URSURS55OSnUb!URSURS55OSnUb1URSURS5URS55OSnUR"U4UUUUUSS.UD6n U Sn UR U 5n UR U 5n U RSU 5n SnUb[ 5nU"X5n[UU U RU RS9$)aQ input_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) token_type_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) task_type_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Task type embedding is a special embedding to represent the characteristic of different tasks, such as word-aware pre-training task, structure-aware pre-training task and semantic-aware pre-training task. We assign a `task_type_id` to each task and the `task_type_id` is in the range `[0, config.task_type_vocab_size-1] position_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) inputs_embeds (`torch.FloatTensor` of shape `(batch_size, num_choices, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) Nr-rITrr) rQrrNrr[rrrrr)r;rArrBrCrDrErr num_choicesrrrreshaped_logitsrrs r>raErnieForMultipleChoice.forwardxs`-6,Aiooa(}GZGZ[\G] >G>SINN2y~~b'9:Y] M[Mg,,R1D1DR1HImqM[Mg,,R1D1DR1HImqGSG_|((\->->r-BCei (   r=#5#5b#9=;M;Mb;Q R  **   ))'%'       ]3 / ++b+6  ')HO4D("!//))   r@rrr)rcrdrerfrrrRrkrrrrrarlrrr@r>rrws*..2.2-1,0-1&*U <<$&U  t+U  t+ U ||d* U llT) U ||d*U  t#U +,U  u|| 8 8U U r@rcD\rSrSr\\SS\RS-S\RS-S\RS-S\RS-S\RS-S\RS-S \RS-S \\ S \ \R\ -4S jj55r S r g)ErnieForTokenClassificationiNrArrBrCrDrErrrGc FUR"U4UUUUUSS.UD6n U Sn URU 5n URU 5n Sn Ub<[5n U "U R SUR 5UR S55n [ U U U RU RS9$)al task_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Task type embedding is a special embedding to represent the characteristic of different tasks, such as word-aware pre-training task, structure-aware pre-training task and semantic-aware pre-training task. We assign a `task_type_id` to each task and the `task_type_id` is in the range `[0, config.task_type_vocab_size-1] labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. TrrNrIr) rr[rrrrrrr)r;rArrBrCrDrErrrrrrrs r>ra#ErnieForTokenClassification.forwards,**   ))'%'    "!*,,71  ')HFKKDOOrrs*..2.2-1,0-1&*. <<$&.  t+.  t+ . ||d* . llT) . ||d*.  t#. +,.  u|| 4 4. . r@rcd\rSrSr\\SS\RS-S\RS-S\RS-S\RS-S\RS-S\RS-S \RS-S \RS-S \\ S \ \R\ -4S jj55r Sr g)ErnieForQuestionAnsweringiNrArrBrCrDrEstart_positions end_positionsrrGc UR"U4UUUUUSS.U D6n U Sn URU 5n U RSSS9upU RS5R 5n URS5R 5nSnUbUb[ UR 55S:aURS5n[ UR 55S:aURS5nU R S5nURSU5nURSU5n[US9nU"X5nU"X5nUU-S - n[UU UU RU RS 9$) rTrrr-rI)rJN) ignore_indexr)r start_logits end_logitsrr) r qa_outputssplitr contiguouslenrNclamprrrr)r;rArrBrCrDrErr rrrrr r r ignored_indexr start_lossend_losss r>ra!ErnieForQuestionAnswering.forwards{***   ))'%'    "!*1#)<<r<#: #++B/::< ''+668   &=+D?'')*Q."1"9"9""==%%'(1, - 5 5b 9 (--a0M-33A}EO)//=AM']CH!,@J :H$x/14J+%!!//))   r@rrr)rcrdrerfrrrRrkrrrrrarlrrr@r>rrs*..2.2-1,0-1/3-1< <<$&<  t+<  t+ < ||d* < llT) < ||d*< ,< ||d*< +,<  u|| ; ;< < r@r) rrrrrrrrrr)NrgrRtorch.nnr6rrrrr cache_utilsrr r modeling_outputsr r r rrrrrmodeling_utilsrprocessing_utilsrutilsrrr utils.genericrrutils.output_capturingrbert.modeling_bertrrrrr r!r"r#r$r%r&r'r(r)r*r+r,configuration_ernier. get_loggerrcloggerr0rprvryr|rrrrrrrrrrrrr__all__rrr@r>r%s AA&CC   .&@@I5&-   H %=n=@ *  ,    *  0  ; /?//2[ [ | 8 P ,P f@ @ F: : zG %BG TE %BE PX 2X v1 "<1 h? 8? D r@