Zj3:SrSSKrSSKrSSKJr SSKJrJrJr SSKJ r SSK J r SSK Jr SS KJrJrJrJrJr SS KJr SS KJr SS KJrJr S SKJr \R<"\5r "SS\RB5r""SS\RB5r#"SS\RB5r$"SS\RB5r%"SS\RB5r&"SS\RB5r'"SS\RB5r("SS\5r)"SS \RB5r*"S!S"\RB5r+\"S#S$\55r,\"S%S&\,55r-\"S'S(9"S)S*\,55r.\"S+S,\,55r/"S-S.\RB5r0\"S/S0\,55r1/S1Qr2g)2zPyTorch LiLT model.N)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)ACT2FN)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPoolingQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel)apply_chunking_to_forward)auto_docstringlogging) LiltConfigcF^\rSrSrU4SjrSSjrSrSrSrU=r $)LiltTextEmbeddings)c >[TU]5 [R"URUR UR S9Ul[R"URUR 5Ul [R"URUR 5Ul [R"UR URS9Ul [R"UR5UlUR#S[$R&"UR5R)S5SS9 UR Ul[R"URUR UR*S9Ul g)N padding_idxeps position_idsrF) persistent)super__init__r Embedding vocab_size hidden_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingstype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutregister_buffertorcharangeexpandrselfconfig __class__s w/root/GenerationalWealth/GenerationalWealth/venv/lib/python3.13/site-packages/transformers/models/lilt/modeling_lilt.pyr#LiltTextEmbeddings.__init__*s !||F,=,=v?Q?Q_e_r_rs#%<<0N0NPVPbPb#c %'\\&2H2H&J\J\%]"f&8&8f>S>STzz&"<"<=   ELL)G)GHOOPWXej  "..#%<<  * *F,>,>DL\L\$  c$UcIUb5URXR5RUR5nOUR U5nUbUR 5nOUR 5SSnUc8[ R"U[ RURRS9nUcURU5nURU5nXF-nURU5nXx- nURU5nURU5nXs4$)Nr dtypedevice)"create_position_ids_from_input_idsrtor@&create_position_ids_from_inputs_embedssizer3zeroslongrr(r,r*r-r1) r7 input_idstoken_type_idsr inputs_embeds input_shaper, embeddingsr*s r:forwardLiltTextEmbeddings.forward>s  $#FFyRbRbcff$$   $JJ=Y  #..*K',,.s3K  !"[[EJJtO`O`OgOghN   00;M $ : :> J": "66|D) ^^J/ \\*- ''r<cURU5R5n[R"USS9R U5U-nUR 5U-$)z Args: Replace non-padding symbols with their position numbers. Position numbers begin at padding_idx+1. Padding symbols are ignored. This is modified from fairseq's `utils.make_positions`. x: torch.Tensor x: Returns: torch.Tensor rdim)neintr3cumsumtype_asrF)r7rGrmaskincremental_indicess r:rA5LiltTextEmbeddings.create_position_ids_from_input_idsbsP||K(,,.$||Da8@@F$N"'')K77r<cUR5SSnUSn[R"URS-X0R-S-[RUR S9nUR S5RU5$)z Args: We are provided embeddings directly. We cannot infer which are padded so just generate sequential position ids.: inputs_embeds: torch.Tensor Returns: torch.Tensor Nr rr>r)rDr3r4rrFr@ unsqueezer5)r7rIrJsequence_lengthrs r:rC9LiltTextEmbeddings.create_position_ids_from_inputs_embedsos~$((*3B/ %a.||   q /4D4D"Dq"HPUPZPZcpcwcw %%a(// <[TU]5 [R"URUR S-5Ul[R"URUR S-5Ul[R"URUR S-5Ul[R"URUR S-5Ul URUl [R"URUR UR-URS9Ul[R"UR UR UR-S9Ul[R""UR UR-UR$S9Ul[R&"UR(5Ulg)Nr) in_features out_featuresr)r"r#rr$max_2d_position_embeddingsr&x_position_embeddingsy_position_embeddingsh_position_embeddingsw_position_embeddingsr'rr)channel_shrink_ratiobox_position_embeddingsLinearbox_linear_embeddingsr-r.r/r0r1r6s r:r#LiltLayoutEmbeddings.__init__s^ &(\\&2S2SU[UgUgklUl%m"%'\\&2S2SU[UgUgklUl%m"%'\\&2S2SU[UgUgklUl%m"%'\\&2S2SU[UgUgklUl%m"!..')||  * *   &"="= =((( $ &(YY**9K9KvOjOj9j& "f&8&8F">|"L&A&[#&*nn5P&Q#&*ll3N&O#**3 cZ[ab b csA,D## D>- D99D>) r-rrrpr1rmrrnrkrl)NN)r\r]r^r_r#rLr`rarbs@r:rdrds>*++r<rdcD^\rSrSrSU4SjjrSSjrSSjrSrU=r$) LiltSelfAttentionc>[TU]5 URUR-S:wa7[ US5(d&[ SURSURS35eURUl[ URUR- 5UlURUR-Ul[R"URUR5Ul [R"URUR5Ul [R"URUR5Ul [R"URUR-URUR-5Ul[R"URUR-URUR-5Ul[R"URUR-URUR-5Ul[R$"UR&5UlURUlX lg)Nrembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ())r"r#r&num_attention_headshasattr ValueErrorrRattention_head_size all_head_sizerrqquerykeyvaluero layout_query layout_key layout_valuer/attention_probs_dropout_probr1 layer_idx)r7r8rr9s r:r#LiltSelfAttention.__init__s     : : :a ?PVXhHiHi#F$6$6#78 445Q8  $*#=#= #&v'9'9F,,7 _kB %--aAq9DDF "/"4"4"6s";t?Q?Q>S"S%**,CD  "67  22Gr<) rrror1rrrrrrrrN)rNF) r\r]r^r_r#rrLr`rarbs@r:rrs!#>% AAr<rcz^\rSrSrU4SjrS\R S\R S\R 4SjrSrU=r $)LiltSelfOutputi!c(>[TU]5 [R"URUR5Ul[R "URURS9Ul[R"UR5Ul gNr) r"r#rrqr&denser-r.r/r0r1r6s r:r#LiltSelfOutput.__init__"s` YYv1163E3EF f&8&8f>S>STzz&"<"<= r<r input_tensorreturncpURU5nURU5nURX-5nU$rrr1r-r7rrs r:rLLiltSelfOutput.forward(5 =1  ]3 }'CD r<r-rr1 r\r]r^r_r#r3TensorrLr`rarbs@r:rr!6> U\\RWR^R^r<rc ^\rSrSrS U4SjjrS S\R S\R S\RS-S\S-S\ \R 4 S jjr S r U=r $) LiltAttentioni/Nc>[TU]5 [XS9Ul[ U5UlUR nUR UR-Ul[ U5UlX1lg)Nr) r"r#rr7routputr&ro layout_output)r7r8rori_hidden_sizer9s r:r#LiltAttention.__init__0s] %fB $V,  ,,#//63N3NN+F3,r<rrrrrcURUUUU5nURUSU5nURUSU5nXg4USS-nU$)Nrrru)r7rr) r7rrrr self_outputsattention_outputlayout_attention_outputrs r:rLLiltAttention.forward:seyy       ;;|A F"&"4"4\!_m"T#= QR@PPr<)rrr7rr) r\r]r^r_r#r3r FloatTensorbooltuplerLr`rarbs@r:rr/sj-48). ||||))D0   $;  u||  r<rcb^\rSrSrU4SjrS\R S\R 4SjrSrU=r $)LiltIntermediateiNc>[TU]5 [R"URUR 5Ul[UR[5(a[URUl gURUl gr) r"r#rrqr&intermediate_sizer isinstance hidden_actstrr intermediate_act_fnr6s r:r#LiltIntermediate.__init__Os` YYv1163K3KL f'' - -'-f.?.?'@D $'-'8'8D $r<rrcJURU5nURU5nU$rrr)r7rs r:rLLiltIntermediate.forwardWs& =1 00? r<rrrbs@r:rrNs(9U\\ellr<rcz^\rSrSrU4SjrS\R S\R S\R 4SjrSrU=r $) LiltOutputi^c(>[TU]5 [R"URUR 5Ul[R"UR URS9Ul[R"UR5Ul gr) r"r#rrqrr&rr-r.r/r0r1r6s r:r#LiltOutput.__init___s` YYv779K9KL f&8&8f>S>STzz&"<"<= r<rrrcpURU5nURU5nURX-5nU$rrrs r:rLLiltOutput.forwarderr<rrrbs@r:rr^rr<rc ^\rSrSrS U4SjjrSS\R S\R S\RS-S\S-S\ \R 4 S jjr S r S r S r U=r$) LiltLayerilNc>[TU]5 URUlSUl[ XS9Ul[ U5Ul[U5Ul URnURnURUR-Ul URUR-Ul [ U5Ul [U5UlX1l XAl g)Nrr)r"r#chunk_size_feed_forward seq_len_dimr attentionr intermediaterrr&rrolayout_intermediater)r7r8rrori_intermediate_sizer9s r:r#LiltLayer.__init__ms '-'E'E$&vC,V4 (  ,, & 8 8#//63N3NN#)#;#;v?Z?Z#Z #3F#; '/,#8 r<rrrrrcURUUUUS9nUSnUSnUSSn[URURURU5n [UR URURU5n X4U-nU$)N)rrrru)rrfeed_forward_chunkrrlayout_feed_forward_chunk) r7rrrrself_attention_outputsrrr layer_outputlayout_layer_outputs r:rLLiltLayer.forward~s"&   / "0"  2!4"8";(,0  # #T%A%A4CSCSUe 8  * *D,H,H$JZJZ\s  5?r<cJURU5nURX!5nU$r)rrr7rintermediate_outputrs r:rLiltLayer.feed_forward_chunks)"//0@A{{#6I r<cJURU5nURX!5nU$r)rrrs r:r#LiltLayer.layout_feed_forward_chunks+"667GH))*=P r<)rrrrrrrrr)r\r]r^r_r#r3rrrrrLrrr`rarbs@r:rrlst9*48). ||||))D0   $;  u||  : r<rc^\rSrSrU4SjrS S\R S\R S\RS-S\S-S\S-S \S-S \ \R \ -4S jjr S r U=r $) LiltEncoderic>[TU]5 Xl[R"[ UR 5Vs/sHn[U5PM sn5Ulgs snfr) r"r#r8r ModuleListrangenum_hidden_layersrlayer)r7r8_r9s r:r#LiltEncoder.__init__sG  ]]uVE]E]?^#_?^!If$5?^#_` #_sANrrrroutput_hidden_states return_dictrc4U(aSOSnU(aSOSn[UR5H7upU(aXq4-nU "UUUU5n U SnU SnU(dM/XS4-nM9 U(aXq4-nU(d[SUUU455$[UUUS9$)Nrrruc30# UH nUcMUv M g7frr).0vs r: &LiltEncoder.forward..s"A  s )last_hidden_stater attentions) enumerater rr ) r7rrrrrrall_hidden_statesall_self_attentionsi layer_module layer_outputss r:rLLiltEncoder.forwards#7BD$5b4(4OA#$58H$H!(! M*!,M)!,M  &91=M[TU]5 [R"URUR5Ul[R "5Ulgr)r"r#rrqr&rTanh activationr6s r:r#LiltPooler.__init__s9 YYv1163E3EF '')r<rrc\USS2S4nURU5nURU5nU$Nr)rr%)r7rfirst_token_tensor pooled_outputs r:rLLiltPooler.forwards6+1a40 #56  6 r<)r%rrrbs@r:r"r"s($ U\\ellr<r"c@^\rSrSr%\\S'SrSr/rU4Sjr Sr U=r $)LiltPreTrainedModelir8liltTc>[TU]U5 [U[5(a\[R "UR [R"UR RS5RS55 gg)Nr r) r" _init_weightsrrinitcopy_rr3r4shaper5)r7moduler9s r:r0!LiltPreTrainedModel._init_weightss^ f% f0 1 1 JJv**ELL9L9L9R9RSU9V,W,^,^_f,g h 2r<r) r\r]r^r_r__annotations__base_model_prefixsupports_gradient_checkpointing_no_split_modulesr0r`rarbs@r:r-r-s' &*#iir<r-cX^\rSrSrSU4SjjrSrSr\SS\RS-S\RS-S\RS-S \RS-S \RS-S \RS-S \ S-S \ S-S\ S-S\ \R\ -4Sjj5r SrU=r$) LiltModelic>[TU]U5 Xl[U5Ul[ U5Ul[U5UlU(a [U5OSUl UR5 g)z^ add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer N) r"r#r8rrKrdlayout_embeddingsrencoderr"pooler post_init)r7r8add_pooling_layerr9s r:r#LiltModel.__init__sX   ,V4!5f!="6* ,=j(4  r<c.URR$rrKr()r7s r:get_input_embeddingsLiltModel.get_input_embeddings s...r<c$XRlgrrD)r7rs r:set_input_embeddingsLiltModel.set_input_embeddingss */'r<NrGrxrrHrrIrrrrc @UbUOURRnUbUOURRnU bU OURRn UbUb [ S5eUb"UR X5 UR 5n O"UbUR 5SSn O [ S5eU upUb URO URnUc'[R"U S-[RUS9nUc[R"X4US9nUcr[URS5(a3URRSS2SU 24nURX5nUnO$[R"U [RUS9nUR!X;5nURUUUUS 9unnUR#X%S 9nUR%UUUUUU S 9nUS nUR&bUR'U5OSnU (d UU4US S-$[)UUUR*UR,S9$)a- bbox (`torch.LongTensor` of shape `(batch_size, sequence_length, 4)`, *optional*): Bounding boxes of each input sequence tokens. Selected in the range `[0, config.max_2d_position_embeddings-1]`. Each bounding box should be a normalized version in (x0, y0, x1, y1) format, where (x0, y0) corresponds to the position of the upper left corner in the bounding box, and (x1, y1) represents the position of the lower right corner. See [Overview](#Overview) for normalization. Examples: ```python >>> from transformers import AutoTokenizer, AutoModel >>> from datasets import load_dataset >>> tokenizer = AutoTokenizer.from_pretrained("SCUT-DLVCLab/lilt-roberta-en-base") >>> model = AutoModel.from_pretrained("SCUT-DLVCLab/lilt-roberta-en-base") >>> dataset = load_dataset("nielsr/funsd-layoutlmv3", split="train") >>> example = dataset[0] >>> words = example["tokens"] >>> boxes = example["bboxes"] >>> encoding = tokenizer(words, boxes=boxes, return_tensors="pt") >>> outputs = model(**encoding) >>> last_hidden_states = outputs.last_hidden_state ```NzDYou cannot specify both input_ids and inputs_embeds at the same timer z5You have to specify either input_ids or inputs_embeds)r>)r@rH)rGrrHrI)rxr)rrrrrr)r pooler_outputrr)r8rrrr%warn_if_padding_and_no_attention_maskrDr@r3rErFonesrrKrHr5get_extended_attention_maskr=r>r?r rr)r7rGrxrrHrrIrrrkwargsrJ batch_size seq_lengthr@buffered_token_type_ids buffered_token_type_ids_expandedextended_attention_maskembedding_outputlayout_embedding_outputencoder_outputssequence_outputr*s r:rLLiltModel.forwardsFP2C1N-TXT_T_TqTq$8$D $++JjJj &1%cd d  "  6 6y Q#..*K  &',,.s3KTU U!, %.%:!! @T@T <;;{T1FSD  !"ZZ*)A6RN  !t(899*.//*H*HKZK*X'3J3Q3QR\3i0!A!&[ SY!Z150P0PQ_0m)-%)' *9* &,#'"8"8d"8"^,,  #2/!5# ' *!,8< 8O O4UY #]3oab6II I)-')77&11   r<)r8rKr>r=r?)T) NNNNNNNNN)r\r]r^r_r#rErHrr3rrrr rLr`rarbs@r:r;r;s"/0*.$(.2.2,0-1)-,0#'h <<$&h llT!h  t+ h  t+ h llT) h ||d*h  $;h #Tkh D[h  u|| 9 9h h r<r;z LiLT Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. ) custom_introch^\rSrSrU4Sjr\SS\RS-S\RS-S\RS-S\RS-S\RS-S \RS-S \RS-S \ S-S \ S-S \ S-S\ \R\ -4Sjj5r SrU=r$)LiltForSequenceClassificationic>[TU]U5 URUlXl[ USS9Ul[ U5UlUR5 gNF)rA) r"r# num_labelsr8r;r.LiltClassificationHead classifierr@r6s r:r#&LiltForSequenceClassification.__init__sH   ++ f> 08 r<NrGrxrrHrrIlabelsrrrrc fU bU OURRn URUUUUUUUU U S9 n U Sn URU 5nSnUGbUR UR 5nURR cURS:XaSURlOoURS:aNUR[R:XdUR[R:XaSURlOSURlURR S:XaI[5nURS:Xa&U"UR5UR55nOU"X5nOURR S:Xa=[5nU"URSUR5URS55nO,URR S:Xa[!5nU"X5nU (dU4U S S-nUbU4U-$U$[#UUU R$U R&S 9$) a  bbox (`torch.LongTensor` of shape `(batch_size, sequence_length, 4)`, *optional*): Bounding boxes of each input sequence tokens. Selected in the range `[0, config.max_2d_position_embeddings-1]`. Each bounding box should be a normalized version in (x0, y0, x1, y1) format, where (x0, y0) corresponds to the position of the upper left corner in the bounding box, and (x1, y1) represents the position of the lower right corner. See [Overview](#Overview) for normalization. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Examples: ```python >>> from transformers import AutoTokenizer, AutoModelForSequenceClassification >>> from datasets import load_dataset >>> tokenizer = AutoTokenizer.from_pretrained("SCUT-DLVCLab/lilt-roberta-en-base") >>> model = AutoModelForSequenceClassification.from_pretrained("SCUT-DLVCLab/lilt-roberta-en-base") >>> dataset = load_dataset("nielsr/funsd-layoutlmv3", split="train") >>> example = dataset[0] >>> words = example["tokens"] >>> boxes = example["bboxes"] >>> encoding = tokenizer(words, boxes=boxes, return_tensors="pt") >>> outputs = model(**encoding) >>> predicted_class_idx = outputs.logits.argmax(-1).item() >>> predicted_class = model.config.id2label[predicted_class_idx] ```NrxrrHrrIrrrrr regressionsingle_label_classificationmulti_label_classificationr rulosslogitsrr)r8rr.rbrBr@ problem_typer`r?r3rFrRrsqueezerrrrrrr7rGrxrrHrrIrdrrrrPrrYrlrkloss_fctrs r:rL%LiltForSequenceClassification.forwards\&1%[TU]U5 URUl[USS9UlUR b UR O UR n[R"U5Ul [R"URUR5Ul UR5 gr_)r"r#r`r;r.classifier_dropoutr0rr/r1rqr&rbr@r7r8rwr9s r:r##LiltForTokenClassification.__init__s   ++f> )/)B)B)NF % %TZTnTn zz"45 ))F$6$68I8IJ r<NrGrxrrHrrIrdrrrrc U bU OURRn URUUUUUUUU U S9 n U Sn URU 5n UR U 5nSnUbWUR UR 5n[5nU"URSUR5URS55nU (dU4U SS-nUbU4U-$U$[UUU RU RS9$)a  bbox (`torch.LongTensor` of shape `(batch_size, sequence_length, 4)`, *optional*): Bounding boxes of each input sequence tokens. Selected in the range `[0, config.max_2d_position_embeddings-1]`. Each bounding box should be a normalized version in (x0, y0, x1, y1) format, where (x0, y0) corresponds to the position of the upper left corner in the bounding box, and (x1, y1) represents the position of the lower right corner. See [Overview](#Overview) for normalization. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. Examples: ```python >>> from transformers import AutoTokenizer, AutoModelForTokenClassification >>> from datasets import load_dataset >>> tokenizer = AutoTokenizer.from_pretrained("SCUT-DLVCLab/lilt-roberta-en-base") >>> model = AutoModelForTokenClassification.from_pretrained("SCUT-DLVCLab/lilt-roberta-en-base") >>> dataset = load_dataset("nielsr/funsd-layoutlmv3", split="train") >>> example = dataset[0] >>> words = example["tokens"] >>> boxes = example["bboxes"] >>> encoding = tokenizer(words, boxes=boxes, return_tensors="pt") >>> outputs = model(**encoding) >>> predicted_class_indices = outputs.logits.argmax(-1) ```Nrfrr rurj) r8rr.r1rbrBr@rrr`rrrros r:rL"LiltForTokenClassification.forwards V&1%[TU]5 [R"URUR5UlUR b UR O URn[R"U5Ul [R"URUR5Ul gr) r"r#rrqr&rrwr0r/r1r`out_projrxs r:r#LiltClassificationHead.__init__[s YYv1163E3EF )/)B)B)NF % %TZTnTn zz"45  &"4"4f6G6GH r<c USS2SSS24nURU5nURU5n[R"U5nURU5nUR U5nU$r()r1rr3tanhr~)r7featuresrPrs r:rLLiltClassificationHead.forwarddsY Q1W  LLO JJqM JJqM LLO MM! r<)rr1r~) r\r]r^r___doc__r#rLr`rarbs@r:raraXs7Ir<rac^\rSrSrU4Sjr\SS\RS-S\RS-S\RS-S\RS-S\RS-S \RS-S \RS-S \RS-S \ S-S \ S-S\ S-S\ \R\ -4Sjj5r SrU=r$)LiltForQuestionAnsweringinc>[TU]U5 URUl[USS9Ul[ R "URUR5UlUR5 gr_) r"r#r`r;r.rrqr& qa_outputsr@r6s r:r#!LiltForQuestionAnswering.__init__qsU   ++f> ))F$6$68I8IJ r<NrGrxrrHrrIstart_positions end_positionsrrrrc &U bU OURRn URUUUUUUU U U S9 n U SnURU5nUR SSS9unnUR S5R 5nUR S5R 5nSnUbUb[UR55S:aUR S5n[UR55S:aUR S5nURS5nURSU5nURSU5n[US9nU"UU5nU"UU5nUU-S- nU (dUU4U SS-nUbU4U-$U$[UUUU RU RS 9$) a* bbox (`torch.LongTensor` of shape `(batch_size, sequence_length, 4)`, *optional*): Bounding boxes of each input sequence tokens. Selected in the range `[0, config.max_2d_position_embeddings-1]`. Each bounding box should be a normalized version in (x0, y0, x1, y1) format, where (x0, y0) corresponds to the position of the upper left corner in the bounding box, and (x1, y1) represents the position of the lower right corner. See [Overview](#Overview) for normalization. Examples: ```python >>> from transformers import AutoTokenizer, AutoModelForQuestionAnswering >>> from datasets import load_dataset >>> tokenizer = AutoTokenizer.from_pretrained("SCUT-DLVCLab/lilt-roberta-en-base") >>> model = AutoModelForQuestionAnswering.from_pretrained("SCUT-DLVCLab/lilt-roberta-en-base") >>> dataset = load_dataset("nielsr/funsd-layoutlmv3", split="train") >>> example = dataset[0] >>> words = example["tokens"] >>> boxes = example["bboxes"] >>> encoding = tokenizer(words, boxes=boxes, return_tensors="pt") >>> outputs = model(**encoding) >>> answer_start_index = outputs.start_logits.argmax() >>> answer_end_index = outputs.end_logits.argmax() >>> predict_answer_tokens = encoding.input_ids[0, answer_start_index : answer_end_index + 1] >>> predicted_answer = tokenizer.decode(predict_answer_tokens) ```Nrfrrr rO) ignore_indexru)rk start_logits end_logitsrr)r8rr.rsplitrnrlenrDclamprr rr)r7rGrxrrHrrIrrrrrrPrrYrlrr total_loss ignored_indexrp start_lossend_lossrs r:rL LiltForQuestionAnswering.forward{s^&1%rs AA&!9.6,*   H %S=S=l5+2995+pf fT RYY BII> ryy   7*7t4 "))4 p   i/ i iA #A A H m $7m  m `^ !4^ ^ DRYY,k 2k k \ r<