ZjpuSrSSKJr SSKrSSKJr SSKJrJrJr SSK J r SSK J r J r SS KJr SS KJr SS KJr SS KJr SS KJrJrJrJr SSKJrJr SSKJr SSK J!r!J"r"J#r#J$r$ SSK%J&r& SSK'J(r( SSK)J*r* \$RV"\,5r-"SS\R\5r/S2S\R`S\RbS\RbS\RbS\RbS-S\2S\24Sjjr3"SS \R`5r4"S!S"\5r5\""S#S$\55r6"S%S&\65r7\""S'S(\655r8"S)S*\6\5r9\""S+S,9"S-S.\655r:\""S/S0\655r;/S1QrSUl[TU] XR-U5 gN)offsetsuper__init__)selfr"r# __class__s u/root/GenerationalWealth/GenerationalWealth/venv/lib/python3.13/site-packages/transformers/models/opt/modeling_opt.pyr)&OPTLearnedPositionalEmbedding.__init__2s"  ++5}ENattention_maskpast_key_values_length position_idsc>Uc5[R"USS9nX1-S- R5nUSS2US24n[TU]X0R -5$)z3`input_ids_shape` is expected to be [bsz x seqlen].Nrdim)torchcumsumlongr(forwardr')r*r/r0r1r+s r,r8%OPTLearnedPositionalEmbedding.forward8sZ   <<A>L(9A=CCEL'+A+B(BCLw|kk9::r.)r')rN) __name__ __module__ __qualname____firstlineno____doc__intr)r5 LongTensorr8__static_attributes__ __classcell__r+s@r,r r -s]FsF3F'(04 ;((;!$;&&- ;;r.r modulequerykeyvaluer/scalingdropoutc [R"XRSS55U-nUbX-n[RR US[R S9RUR5n[RRXURS9n[R"X5n U RSS5R5n X4$)N)r4dtypeptrainingrr&) r5matmul transposer functionalsoftmaxfloat32torMrIrP contiguous) rDrErFrGr/rHrIkwargs attn_weights attn_outputs r,eager_attention_forwardr[Js<<}}R'<=GL!#4 ==((2U]](SVVW\WbWbcL==((6??([L,,|3K''1-88:K  $$r.c^\rSrSrSrSS\S\S-4U4SjjjrSS\RS\ S-S \RS-S \ S \ \R\RS-\ S-44 S jjr S rU=r$) OPTAttentionaz=Multi-headed attention from 'Attention Is All You Need' paperNconfig layer_idxc >[TU]5 XlURUlUR UlURUlURUl X l Uc-[RSURRS35 URUR -UlSUlURUR -UR:wa&[#SURSUR S35eURS-Ul[&R("URURURS9Ul[&R("URURURS9Ul[&R("URURURS9Ul[&R("URURURS9Ulg) NzInstantiating z without passing a `layer_idx` is not recommended and will lead to errors during the forward call if caching is used. Please make sure to provide a `layer_idx` when creating this class.Tz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).g࿩bias)r(r)r_ hidden_size embed_dimnum_attention_heads num_headsattention_dropoutrI enable_biasr`logger warning_oncer+r:head_dim is_causal ValueErrorrHrLineark_projv_projq_projout_proj)r*r_r`rXr+s r,r)OPTAttention.__init__ds{  ++33// !--"      !8!8 9:,,  $..8  MMDNN *t~~ =MdnnM]$T^^$4B8 }}d* iiTEUEUV iiTEUEUV iiTEUEUV  $..$..tGWGWX r. hidden_statespast_key_valuesr/output_attentionsreturnc ^UR5upgnURU5UR-n U RUSURUR 5R SS5n URU5n URU5n U RUSURUR 5R SS5n U RUSURUR 5R SS5n UbURXUR5up[R"URR[5n U "UU U U U4UR (dSO UR"SS.UD6upU R%XgS5R'5n UR)U 5n X4$)z#Input shape: Batch x Time x ChannelrKrr&g?)rIrH)sizerrrHviewrgrlrRrprqupdater`r get_interfacer__attn_implementationr[rPrIreshaperWrs)r*rurvr/rwrXbsztgt_len_ query_states key_states value_statesattention_interfacerZrYs r,r8OPTAttention.forwards(,,.a{{=1DLL@ #((b$..$--PZZ[\^_` [[/ {{=1 __S"dnndmmLVVWXZ[\ #((b$..$--PZZ[\^_`  &'6'='=jX\XfXf'g $J(?(M(M KK , ,.E) %8      % $}}C$,, %  % ! "))#;FFH mmK0 ((r.) r_rIrerirlrmrpr`rgrsrrrHrqN)NNF)r:r;r<r=r>rr?r)r5Tensorr booltupler8rArBrCs@r,r]r]asG !%!Y!Y:!Y!YL)-.2"' .)||.).) t+ .)  .) u||U\\D0%$,> ?.).)r.r]c^\rSrSrSS\S\S-4U4SjjjrSS\RS\RS-S\ S-S \ S-S \RS-S \ \ S \R4S jjrSrU=r$)OPTDecoderLayerNr_r`cp>[TU]5 URUl[ XS9UlUR UlURUl[URUl [R"URURS9Ul[R"URUR UR"S9Ul[R"UR URUR"S9Ul[R"URURS9Ulg)N)r_r`elementwise_affinerb)r(r)rdrer] self_attndo_layer_norm_beforerIr activation_function activation_fnr LayerNormlayer_norm_elementwise_affineself_attn_layer_normroffn_dimrifc1fc2final_layer_norm)r*r_r`r+s r,r)OPTDecoderLayer.__init__s ++%VI$*$?$?!~~ #F$>$>?$&LL NNv/S/S% !99T^^V^^&BTBTU99V^^T^^&BTBTU " T^^PVPtPt ur.rur/rv use_cacher1rXrxc UnUR(aURU5nUR"SUUUUS.UD6up[RR XR UR S9nXq-nUR(dURU5nURn URSURS55nUnUR(aURU5nURU5nURU5nURU5n[RR XR UR S9nXq-RU 5nUR(dURU5nU$)N)rurvr1r/rNrK)rrrrrSrIrPshaperr{rrrrr|) r*rur/rvrr1rXresidualrhidden_states_shapes r,r8OPTDecoderLayer.forwardsk!  $ $ 55mDM >> '+%)      --m||VZVcVc-d  0 (( 55mDM,11%--b-2D2DR2HI    $ $ 11-@M/ **=9 /  --m||VZVcVc-d !1778KL (( 11-@Mr.) rrrIrerrrrrr)NNFN)r:r;r<r=rr?r)r5rr rr@rrr8rArBrCs@r,rrsvyvS4Zvv(/3(,!&04 3||3 t+3 3 $; 3 &&- 3-.3 33r.rcL\rSrSr%\\S'SrSrS/rSr Sr Sr Sr Sr \\S.rSrg) OPTPreTrainedModelr_modelTr)ru attentionsrN)r:r;r<r=r__annotations__base_model_prefixsupports_gradient_checkpointing_no_split_modules_supports_attention_backend_supports_flash_attn_supports_sdpa_supports_flex_attn_can_compile_fullgraphrr]_can_record_outputsrArr.r,rrsH &*#*+"&N!("r.rc^\rSrSrSrS\4U4Sjjr\\\ SS\ RS-S\ RS-S\ S-S \ RS-S \S-S \ RS-S \\S \4Sjj555rSrU=r$) OPTDecoderiz Transformer decoder consisting of *config.num_hidden_layers* layers. Each layer is a [`OPTDecoderLayer`] Args: config: OPTConfig r_c F>[TU]U5 URUlURUlURUlUR UlURUl[R"URURUR 5Ul [UR UR5UlURUR:wa0[R "URURSS9UlOSUlURUR:wa0[R "URURSS9UlOSUlUR&(a@UR((d/[R*"URUR,S9UlOSUl[R0"[3UR45Vs/sH n[7XS9PM sn5UlSUlUR=5 gs snf)NFrbr)r`)r(r)rI layerdrop pad_token_id padding_idxmax_position_embeddingsmax_target_positions vocab_sizer Embeddingword_embed_proj_dim embed_tokensr rdembed_positionsro project_out project_inr_remove_final_layer_normrrr ModuleListrangenum_hidden_layersrlayersgradient_checkpointing post_init)r*r_ir+s r,r)OPTDecoder.__init__s  ~~ ))!..$*$B$B! ++LL):):Fs  -t"; <YZ Z  !r9??2+>?I   --i8M 0*$++>O?N?Z?99;`a  !),?,?,BBJ"ZZ (;(;A(> SgSghN   <<A>L(9A=CCEL'+;+<(<=L(;;')+  )).Ye)f ?? & OOM:M% m6J6J(KK #,DKK"8 C}}&+jjn#&7)*) /#   M#9  , 11-@M    ' ,,];M&++  r.) rIrrrrrrrrrrrNNNNNN)r:r;r<r=r>rr)rrrr5r@rr FloatTensorrrrrr8rArBrCs@r,rrs#y#J .2.2(,26!%04K ##d*K  t+K  K ((4/ K $; K &&-K +,K  !K  K r.rc^\rSrSrS\4U4SjjrSrSr\\ SS\ RS-S\ RS-S \ S-S \ RS-S \S-S \ RS-S \\S\4Sjj55rSrU=r$)OPTModelir_cd>[TU]U5 [U5UlUR 5 gr)r(r)rdecoderrr*r_r+s r,r)OPTModel.__init__s&  !&)  r.c.URR$rrrr*s r,get_input_embeddingsOPTModel.get_input_embeddingss||(((r.c$XRlgrrr*rGs r,set_input_embeddingsOPTModel.set_input_embeddingss $) !r.Nrr/rvrrr1rXrxc UR"SUUUUUUS.UD6n[URURURUR S9$)Nrr/r1rvrr)rrvrurr)rrrrvrur) r*rr/rvrrr1rXdecoder_outputss r,r8OPTModel.forwardsf48<<4 )%+' 4 4 '-??+;;)77&11   r.)rr)r:r;r<r=rr)rrrrr5r@rr rrrrrr8rArBrCs@r,rrsy )*.2.2(,26!%04 ##d*  t+   ((4/  $;  &&- +,  !  r.rcJ^\rSrSrSS0rU4SjrSrSr\\ SS\ RS-S \ RS-S \ S-S \ RS-S \ RS-S \S-S\ RS-S\\ R-S\\S\\-4Sjj55rSrU=r$)OPTForCausalLMizlm_head.weightz!model.decoder.embed_tokens.weightc>[TU]U5 [U5Ul[R "UR URSS9UlUR5 gNFrb) r(r)rrrrorrlm_headrrs r,r)OPTForCausalLM.__init__sK  f% yy!;!;V=N=NUZ[  r.cBURRR$rrrrrs r,r#OPTForCausalLM.get_input_embeddingszz!!...r.c8XRRlgrrrs r,r#OPTForCausalLM.set_input_embeddings*/ 'r.Nrr/rvrlabelsrr1logits_to_keeprXrxc URR"SUUUUUUS.U D6n U Rn [U[5(a [ U*S5OUn UR U SS2U SS245R5n SnUb)UR"SXURRS.U D6n[UU U RU RU RS9$)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, OPTForCausalLM >>> model = OPTForCausalLM.from_pretrained("facebook/opt-350m") >>> tokenizer = AutoTokenizer.from_pretrained("facebook/opt-350m") >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious. I'm just a little bit of a weirdo." ```rN)logitsrrlossrrvrurr)rrr isinstancer?slicerrW loss_functionr_rrrvrur)r*rr/rvrrrr1rrXoutputsru slice_indicesrrs r,r8OPTForCausalLM.forwardsJ,0::+=+=, )%+' , ,  11 8B>SV8W8W~ot4]k mA}a,?@ALLN  %%pVt{{OeOepiopD%#33!//))   r.)rr)NNNNNNNr)r:r;r<r=_tied_weights_keysr)rrrrr5r@rr rrr?rrrrr8rArBrCs@r,rrs*,OP/0.2.2(,26*.!%04-.< ##d*<  t+<  < ((4/ <   4' < $;< &&-< ell*< +,<  ' '< < r.ra The OPT Model transformer with a sequence classification head on top (linear layer). [`OPTForSequenceClassification`] uses the last token in order to do the classification, as other causal models (e.g. GPT-2) do. Since it does classification on the last token, it requires to know the position of the last token. If a `pad_token_id` is defined in the configuration, it finds the last token that is not a padding token in each row. If no `pad_token_id` is defined, it simply takes the last value in each row of the batch. Since it cannot guess the padding tokens when `inputs_embeds` are passed instead of `input_ids`, it does the same (take the last value in each row of the batch). ) custom_introc*^\rSrSrS\4U4Sjjr\\SS\RS-S\RS-S\ S-S\RS-S \RS-S \ S-S \RS-S \ \S \\-4Sjj55rSrSrSrU=r$)OPTForSequenceClassificationir_c>[TU]U5 URUl[U5Ul[ R "URURSS9UlUR5 gr) r(r) num_labelsrrrrorscorerrs r,r)%OPTForSequenceClassification.__init__sT   ++f% YYv994??QVW  r.Nrr/rvrrrr1rXrxc UR"U4UUUUUS.UD6n U Rn URU 5n UbURSSupOURSSupURR cU S:wa [ S5eURR cSnOUbXRR :gRU R[R5n[R"URSU R[RS9nUU-RS5nO.Sn[RURR S35 U [R"XRS 9U4nSnUGbURR"cUR$S:XaS URlOoUR$S:aNUR&[R(:XdUR&[R*:XaS URlOS URlURR"S :XaJ[-5nUR$S:Xa&U"UR/5UR/55nOU"UU5nOURR"S :Xa=[15nU"UR3SUR$5UR3S55nO-URR"S :Xa[55nU"UU5n[7UUU R8U R:U R<S 9$)ae labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). rvr/r1rrNr&rz=Cannot handle batch sizes > 1 if no padding token is defined.rK)rrMz will not detect padding tokens in `inputs_embeds`. Results may be unexpected if using padding tokens in conjunction with `inputs_embeds.`r regressionsingle_label_classificationmulti_label_classificationr)rrrrr_rrnrVrr5int32arangeargmaxrjrkr+r: problem_typerrMr7r?rsqueezerr|rrrvrur)r*rr/rvrrrr1rXtransformer_outputsrur batch_sizesequence_lengthlast_non_pad_token non_pad_mask token_indices pooled_logitsrloss_fcts r,r8$OPTForSequenceClassification.forward's&8!F!Fr!J !#    >>**+,ZZ  u||J}}MOaab   {{''/??a'/;DKK,__q(fllejj.HFLL\a\e\eLe/LDKK,/KDKK,{{''<7"9??a'#M$9$9$;V^^=MND#M6:D))-JJ+- 2 22t GUWY))-II,. v6/ /??-;;*55   r.cBURRR$rrrs r,r1OPTForSequenceClassification.get_input_embeddings|rr.c8XRRlgrrrs r,r1OPTForSequenceClassification.set_input_embeddingsrr.)rrr)NNNNNNN)r:r;r<r=rr)rrr5r@rr rrrrrr8rrrArBrCs@r,rrsy.237(,26*.!%04Q ##d*Q ))D0Q  Q ((4/ Q   4' Q $;Q &&-Q +,Q  1 1Q Q f/00r.rcJ^\rSrSrS\4U4Sjjr\\SS\RS-S\RS-S\ S-S\RS-S \RS-S \RS-S \ S-S \RS-S \ \S\\-4Sjj55rSrSrSrU=r$)OPTForQuestionAnsweringir_c>[TU]U5 [U5Ul[R "UR S5UlUR5 gr%) r(r)rrrror qa_outputsrrs r,r) OPTForQuestionAnswering.__init__s@  f% ))F$>$>B r.Nrr/rvrstart_positions end_positionsrr1rXrxc (UR"U4UUUUUS.U D6n U Rn URU 5n U RSSS9upU R S5R 5n UR S5R 5nSnUbUb[ UR55S:aUR S5n[ UR55S:aUR S5nU RS5nURSU5RU R5nURSU5RU R5n[US9nU"X5nU"X5nUU-S- n[UU UU RU RS 9$) a Example: ```python >>> from transformers import AutoTokenizer, OPTForQuestionAnswering >>> import torch >>> torch.manual_seed(4) # doctest: +IGNORE_RESULT >>> tokenizer = AutoTokenizer.from_pretrained("facebook/opt-350m") >>> # note: we are loading a OPTForQuestionAnswering from the hub here, >>> # so the head will be randomly initialized, hence the predictions will be random >>> model = OPTForQuestionAnswering.from_pretrained("facebook/opt-350m") >>> question, text = "Who was Jim Henson?", "Jim Henson was a nice puppet" >>> inputs = tokenizer(question, text, return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**inputs) >>> answer_start_index = outputs.start_logits.argmax() >>> answer_end_index = outputs.end_logits.argmax() >>> answer_offset = len(tokenizer(question)[0]) >>> predict_answer_tokens = inputs.input_ids[ ... 0, answer_offset + answer_start_index : answer_offset + answer_end_index + 1 ... ] >>> predicted = tokenizer.decode(predict_answer_tokens) >>> predicted ' a nice puppet' ```rrrKr3Nr) ignore_indexr&)r start_logits end_logitsrur)rrr/splitrrWlenr{clamprVrrrrur)r*rr/rvrr1r2rr1rXrrurr5r6 total_loss ignored_indexr& start_lossend_losss r,r8OPTForQuestionAnswering.forwards\8collections.abcrr5rtorch.nnrrr activationsr cache_utilsr r generationr masking_utilsr modeling_flash_attention_utilsrmodeling_layersrmodeling_outputsrrrrmodeling_utilsrrprocessing_utilsrutilsrrrr utils.genericrutils.output_capturingrconfiguration_optr get_loggerr:rjrr Modulerfloatr[r]rrrrrrr-__all__rr.r,rVs$ AA!.)/B9 G&RR75(   H %;BLL;H% II% <<% % << % LL4' %  %%.T)299T)nE0EP    { #{ |( !( ( VQ 'Q h  c0#5c0 c0Lb00b0b0J r.