Zj|2SrSSKrSSKrSSKJr SSKJrJrJrJr SSKJ r SSK J r J r SSKJr SS KJr SS KJr SS KJrJrJrJrJr SS KJr SS KJrJr SSKJ r \RB"\"5r#S'Sjr$"SS\RJ5r&"SS\RJ5r'"SS\5r(\"SS\55r)\"SS\)55r*\"SS9"SS\)\55r+\"SS9"S S!\)55r,\"S"S#\)55r-\"S$S%\)55r./S&Qr/g)(zPyTorch MPT model.N)nn)BCEWithLogitsLossCrossEntropyLoss LayerNormMSELoss) functional)Cache DynamicCache)GenerationMixin)create_causal_mask)GradientCheckpointingLayer))BaseModelOutputWithPastAndCrossAttentions!CausalLMOutputWithCrossAttentionsQuestionAnsweringModelOutput SequenceClassifierOutputWithPastTokenClassifierOutput)PreTrainedModel)auto_docstringlogging) MptConfigcH[R"SU- S[RUS9RSSSU5nS[R "[R "U55-n[R"SUS-[RUS9R5nXbU- -nS[R"SU5- nURSUSS5nXP:wa7[R"USS2SSS2S4USS2SSS2S4/SS9SS2SU2S4nXG-nURS5$) a Link to paper: https://huggingface.co/papers/2108.12409 - Alibi tensor is not causal as the original paper mentions, it relies on a translation invariance of softmax for quick implementation. This implementation has been copied from the alibi implementation of MPT source code that led to slightly different results than the Bloom alibi: https://huggingface.co/mosaicml/mpt-7b/blob/main/attention.py#L292 r)dtypedeviceg?N.dimr) torcharangeint32viewmathceillog2int64floatpowconcatsqueeze) num_headssequence_lengthalibi_bias_maxralibinum_heads_power_of_2baseslopess u/root/GenerationalWealth/GenerationalWealth/venv/lib/python3.13/site-packages/transformers/models/mpt/modeling_mpt.pybuild_mpt_alibi_tensorr3*s LL_,au{{6 R W WXY[\^_ap qE $))I*> ?? <</!35;;v V \ \ ^D $88 9D 599Q% %F [[0!Q 7F(vaAsl3VAssCK5HIqQRSU_V_U_adRde NE == c ^\rSrSrSrS S\S\S-4U4SjjjrSS\RS\RS \ S-S \RS-4S jjr S r U=r $) MptAttentionAzrMulti-head self attention. Using torch or triton attention implementation enables user to also use additive bias. Nconfig layer_idxc>[TU]5 URUlURUlURUlURUR-UlURRUlURc5S[R"URUR- 5- UlURRUl URRUl [R"URSUR-SS9Ul[R"URURSS9UlX lg)Nrr Fbias)super__init__ hidden_sizen_heads max_seq_lenmax_seq_lengthhead_dim attn_config softmax_scaler#sqrt attn_pdropattn_dropout_pclip_qkvrLinearWqkvout_projr9)selfr8r9 __class__s r2r>MptAttention.__init__Fs !--~~ $00((DLL8 #//==    %!"TYYt/?/?$,,/N%O!OD $00;;**33 IId..D4D4D0D5Q  $"2"2D4D4D5Q "r4 hidden_states position_biaspast_key_valuesattention_maskc RURSSupgURU5nUR(a%URUR*URS9nUR SSS9upn U R XgUR UR5RSS5n U R XgUR UR5RSS5n U R XgUR UR5RSS5n UbURXUR5up[R"XRSS55UR-n UcUOXsR5-n Ub[UR5S:wa![!S[UR535eU RSn[#S UR%S5U - 5n[#S UR%S5U- 5nUSS2US2US24nX-n Ub:U R'U[R("U R*5R,5n [.R0R3U R55SS9R7U R*5n[.R0R9UUR:UR<S 9n[R"UU 5nUR?S SSS5RA5RCXgS5nUREU5nUU4$) Nr)minmaxr rrz6Expecting position_bias shape to be 3 dimensions, got rptraining)#shaperKrIclampchunkreshaper@rC transposeupdater9rmatmulrEget_seq_lengthlen ValueErrorrVsize masked_fillfinforrUrrsoftmaxr'todropoutrHr[permute contiguousr"rL)rMrPrQrRrSkwargs batch_size seq_length mixed_qkv query_states key_states value_statesattention_scores query_length key_lengthposition_bias_query_indexposition_bias_key_index attn_weightscontext_states attn_outputs r2forwardMptAttention.forwardVs"/!4!4Ra!8 IIm, ==!T]]N NI1:1J. ,#++JDLLRVR_R_`jjklnop ''  dmm\ffghjkl #++JDLLRVR_R_`jjklnop  &'6'='=jX\XfXf'g $J << 6J6J2r6RSVZVhVhh%4%rTensorr r}__static_attributes__ __classcell__rNs@r2r6r6Asq#y#S4Z##()-.2 0)||0)||0) 0)  t+ 0)0)r4r6c^\rSrSrS\4U4SjjrS\RS\RS\R4SjrSr U=r $) MptMLPr8c>[TU]5 URn[R"USU-SS9Ul[R "SS9Ul[R"SU-USS9UlURRUl g)NFr;none) approximate) r=r>r?rrJup_projGELUact down_projrDrGhidden_dropout)rMr8r?rNs r2r>MptMLP.__init__sm (( yya+oEJ 77v.1{?KeL$00;;r4rPresidualreturncURURU55nURU5n[R"X0R UR S9nXB-nU$)NrY)rrrFrkrr[)rMrPrintermediate_outputoutputs r2r}MptMLP.forwardsSm!<= "nn];.2E2EPTP]P]^" r4)rrrr) rrrrrr>rrr}rrrs@r2rrs:[TU]5 URn[X1RS9UlSUR lURUl[X5Ul [X1RS9Ul SURl[U5Ul URRUl["R$"UR 5Ulg)Neps)r=r>r?rlayer_norm_epsilonnorm_1r<r@r+r6attnnorm_2rffnrDrG dropout_raterDropoutresid_attn_dropout)rMr8r9r?rNs r2r>MptBlock.__init__s ((  1J1JK   3  1J1JK  &>"..99"$**T->->"?r4rPrQrS layer_past use_cacheoutput_attentionsc URU5nUn URUUUUS9upURU 5U -nURU5nUn UR X5n X4$)N)rQrSrR)rrrrr) rMrPrQrSrrrrnlayernorm_outputr attn_outputsrzrs r2r}MptBlock.forwards ;;}5 &*YY ')& &/& " // =H ;;}5!*5##r4)rrrrrr+rr)NFF)rrrrrrr>rrr boolr}rrrs@r2rrs@y@S4Z@@2$("'!$||!$||!$ !$ DL !$  !$ !$!$r4rc.\rSrSr%\\S'SrSrS/rSr g)MptPreTrainedModelr8 transformerTrN) rrrrr__annotations__base_model_prefixsupports_gradient_checkpointing_no_split_modulesrrr4r2rrs %&*## r4rc>^\rSrSrS\4U4SjjrSrSSjrS\R4Sjr \ SS \RS-S \ S-S \RS-S \RS-S \S-S\S-S\S-S\S-S\\RS4\-4Sjj5rSrU=r$)MptModelr8c >[TU]U5 URUlURUl[ R "URUR5Ul[ R"[UR5Vs/sH n[XS9PM sn5Ul [URURS9UlSUR lSUlUR'5 gs snf)N)r9rF)r=r>r?r@r+r Embedding vocab_sizewte ModuleListrangen_layersrblocksrrnorm_fr<gradient_checkpointing post_init)rMr8irNs r2r>MptModel.__init__s  !--<< 1 143C3CDmmERXRaRaLb$cLbqXf%BLb$cd   0 0f6O6OP  &+# %ds C5cUR$rr)rMs r2get_input_embeddingsMptModel.get_input_embeddingss xxr4Nc[XX45$r)r3)rMr+r,r-rs r2r3MptModel.build_mpt_alibi_tensors%i.YYr4new_embeddingscXlgrrrMrs r2set_input_embeddingsMptModel.set_input_embeddingss!r4 input_idsrRrS inputs_embedsrroutput_hidden_states return_dictr.c PUbUOURRnUbUOURRnUbUOURRnUbUOURRnUbUb [ S5eUbUR upOUbUR upn O [ S5eUR(a/UR(aU(a[RS5 SnUcURU5nU(aUc[URS9nUn U(aSOSnU(aSOSnURURURRU R S9n[#URUUUS 9R%[&R(5nUR*H1nU(aX4-nU"U UUUUUS 9nUS n U(dM(UUS 4-nM3 UR-U 5n U(aX4-nU(d[/S XX455$[1U UUUS9$)j input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values.get_seq_length()` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only `input_ids` that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) NzDYou cannot specify both input_ids and inputs_embeds at the same timez5You have to specify either input_ids or inputs_embedszZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...F)r8rr)r8rrSrR)rrSrrrQrrc3.# UH ocMUv M g7frr).0vs r2 #MptModel.forward..^scacs )last_hidden_staterRrP attentions)r8rrrrrer\rr[logger warning_oncerr r3r+rArr rjrrrrtupler)rMrrRrSrrrrrrnrorp_rPall_self_attentionsall_hidden_statesr. causal_maskblockoutputss r2r}MptModel.forwards642C1N-TXT_T_TqTq$8$D $++JjJj "+!6IDKKcd d  "%.__ "J  &(5(;(; %JATU U  & &4==##p"   HHY/M 0*$++>O% $5b4"6BD++DNNDKKrr3rrrr LongTensorr rrrr}rrrs@r2rrsy,Z"5<<".2(,.215!%)-,0#'f ##d*f f  t+ f ''$. f $; f  $;f #Tkf D[f  u||S !$M Mf f r4rz The MPT Model transformer with a language modeling head on top (linear layer with weights tied to the input embeddings). ) custom_introcr^\rSrSrSS0rS\4U4SjjrS\R4Sjr \ SS \RS-S \ S-S \RS-S \RS-S \RS-S\ S-S\ S-S\ S-S\ S-S\\R-S\\R\-4Sjj5rSrU=r$)MptForCausalLMijzlm_head.weightztransformer.wte.weightr8c>[TU]U5 [U5Ul[R "UR URSS9UlUR5 gNFr;) r=r>rrrrJr?rlm_headrrMr8rNs r2r>MptForCausalLM.__init__ssI  #F+yy!3!3V5F5FUS  r4rcXlgr)rrs r2set_output_embeddings$MptForCausalLM.set_output_embeddings{s% r4NrrRrSrlabelsrrrrlogits_to_keeprc U bU OURRn URUUUUUUUU S9n U Sn [U [5(a [ U *S5OU nUR U SS2USS245nSnUb)UR"SXURRS.U D6nU (dU4U SS-nUbU4U-$U$[UUU RU RU RS9$)a input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values.get_seq_length()` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only `input_ids` that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for language modeling. Note that the labels **are shifted** inside the model, i.e. you can set `labels = input_ids` Indices are selected in `[-100, 0, ..., config.vocab_size]` All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` NrRrSrrrrrr)logitsrrrlossrrRrPrr) r8rr isinstancerslicer loss_functionrrrRrPr)rMrrRrSrrrrrrrrntransformer_outputsrP slice_indicesrrrs r2r}MptForCausalLM.forward~s@&1%SV8W8W~ot4]k mA}a,?@A  %%pVt{{OeOepiopDY!4QR!88F)-)9TGf$ Ev E0/??-;;*55   r4)rr) NNNNNNNNNr)rrrr_tied_weights_keysrr>rrrrrr rrrrr}rrrs@r2rrjs+,DEy&ELL&.2(,.2-1&*!%)-,0#'-.? ##d*? ?  t+ ? ||d* ?  t# ? $;?  $;? #Tk? D[? ell*?  u|| @ @? ? r4ra The MPT Model transformer with a sequence classification head on top (linear layer). [`MptForSequenceClassification`] uses the last token in order to do the classification, as other causal models (e.g. GPT-1) do. Since it does classification on the last token, it requires to know the position of the last token. If a `pad_token_id` is defined in the configuration, it finds the last token that is not a padding token in each row. If no `pad_token_id` is defined, it simply takes the last value in each row of the batch. Since it cannot guess the padding tokens when `inputs_embeds` are passed instead of `input_ids`, it does the same (take the last value in each row of the batch). cJ^\rSrSrS\4U4SjjrS\R4Sjr\ SS\RS-S\ S-S \RS-S \RS-S \RS-S \ S-S \ S-S\ S-S\ S-S\ \R\-4Sjj5rSrU=r$)MptForSequenceClassificationir8c>[TU]U5 URUl[U5Ul[ R "URURSS9UlUR5 gr) r=r> num_labelsrrrrJr?scorerrs r2r>%MptForSequenceClassification.__init__sV   ++#F+YYv1163D3D5Q  r4rcXlgr)rrs r2r2MptForSequenceClassification.set_output_embeddingss# r4NrrRrSrrrrrrrc U bU OURRn URUUUUUUUU S9n U Sn URU 5n UbURSnOURSnURR cUS:wa [ S5eURR cSnOUbXRR :gRU R[R5n[R"URSU R[RS9nUU-RS5nO.Sn[RURR S35 U [R"XRS 9U4nSnUGbgURR"cUR$S:XaS URlOoUR$S:aNUR&[R(:XdUR&[R*:XaS URlOS URlURR"S :XaJ[-5nUR$S:Xa&U"UR/5UR/55nOeU"UU5nO[URR"S :Xa[15nU"UU5nO-URR"S :Xa[35nU"UU5nU (dU4U SS-nUbU4U-$U$[5UUU R6U R8U R:S 9$) input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values.get_seq_length()` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only `input_ids` that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Nrrrz=Cannot handle batch sizes > 1 if no padding token is defined.rW)rrz will not detect padding tokens in `inputs_embeds`. Results may be unexpected if using padding tokens in conjunction with `inputs_embeds.`r regressionsingle_label_classificationmulti_label_classificationr)r8rrrr\ pad_token_idrerjrrr!r argmaxrrrNr problem_typerrlongrrr*rrrrRrPr)rMrrRrSrrrrrrrnr rPrrolast_non_pad_token non_pad_mask token_indices pooled_logitsrloss_fctrs r2r}$MptForSequenceClassification.forwards>&1%!F!Fr!J !#    >>**+,ZZ  u||J}}MOaab   {{''/??a'/;DKK,__q(fllejj.HFLL\a\e\eLe/LDKK,/KDKK,{{''<7"9??a'#M$9$9$;V^^=MND#M6:D))-JJ+- v6))-II,. v6#%(;AB(??F)-)9TGf$ Ev E/ /??-;;*55   r4)rrr NNNNNNNNN)rrrrrr>rrrrrr rrrr}rrrs@r2rrsy$ELL$.2(,.2-1&*!%)-,0#'e ##d*e e  t+ e ||d* e  t# e $;e  $;e #Tke D[e  u|| ? ?e e r4rc(^\rSrSrS\4U4Sjjr\SS\RS-S\ S-S\RS-S\RS-S \RS-S \ S-S \ S-S \ S-S \ S-S\ \R\ -4Sjj5rSrU=r$)MptForTokenClassificationiEr8c>[TU]U5 URUl[U5Ul[ US5(aUR b UR nO-[ US5(aURb URnOSn[R"U5Ul [R"URUR5Ul UR5 g)Nclassifier_dropoutrg?)r=r>rrrhasattrr)rrrrkrJr? classifierr)rMr8r)rNs r2r>"MptForTokenClassification.__init__Gs   ++#F+ 6/ 0 0V5N5N5Z!'!:!:  V- . .63H3H3T!'!6!6 !$ zz"45 ))F$6$68I8IJ r4NrrRrSrrrrrrrc U bU OURRn URUUUUUUUU S9n U Sn URU 5n UR U 5n SnUblUR U R 5nURunn[5nU"U RUU-UR5URUU-55nU (dU 4U SS-nUbU4U-$U$[UU U RU RS9$)rNrrr)rrrPr)r8rrrkr+rjrr\rr"rrrPr)rMrrRrSrrrrrrdeprecated_argumentsr rPrrrorpr#rs r2r}!MptForTokenClassification.forwardXs+>&1%rrrr rrrrr}rrrs@r2r'r'Esy".2(,.2-1&*!%)-,0#'B ##d*B B  t+ B ||d* B  t# B $;B  $;B #TkB D[B  u|| 4 4B B r4r'c^\rSrSrU4Sjr\SS\RS-S\RS-S\RS-S\RS-S\RS-S \ S-S \ S-S \ S-S \ \ -4S jj5r Sr U=r$)MptForQuestionAnsweringic>[TU]U5 [U5Ul[R "UR S5UlUR5 g)Nr) r=r>rrrrJr? qa_outputsrrs r2r> MptForQuestionAnswering.__init__sA  #F+))F$6$6: r4NrrSrstart_positions end_positionsrrrrc UbUOURRnURUUUUUUS9n U Sn URU 5n U R SSS9upU R S5R 5n UR S5R 5nSnUbUb[UR55S:aUR S5n[UR55S:aUR S5nU RS5nURSU5nURSU5n[US9nU"X5nU"X5nUU-S- nU(dX4U SS-nUbU4U-$U$[UU UU RU RS 9$) rN)rSrrrrrrrWr) ignore_indexr)r start_logits end_logitsrPr)r8rrr3splitr*rmrdrfr]rrrPr)rMrrSrr5r6rrrrnrsequence_outputrr9r: total_loss ignored_indexr# start_lossend_lossrs r2r}MptForQuestionAnswering.forwards4&1%rrr FloatTensorrrrr}rrrs@r2r1r1s.237263715)-,0#'F ##d*F ))D0F ((4/ F ))D0 F ''$. F  $;F #TkF D[F  - -F F r4r1)rrrrr'r1r)0rr#rrtorch.nnrrrrrr cache_utilsr r generationr masking_utilsr modeling_layersrmodeling_outputsrrrrrmodeling_utilsrutilsrrconfiguration_mptr get_loggerrrr3Moduler6rrrrrrr'r1__all__rr4r2rOs] LL$.)/9.,(   H %.E)299E)PRYY*6$)6$r%%%G !G G T N 'N  N b  s #5s  s lU 2U U pP 0P P f r4