Zj(SSKJr SSKrSSKJr SSKJr SSKJ r J r SSK J r SSK Jr SSKJrJr SS KJr SS KJr SS KJrJrJrJr SS KJr SS KJr SSKJ r SSK!J"r" SSK#J$r$J%r%J&r&J'r'J(r(J)r)J*r*J+r+ SSK,J-r-J.r. SSK/J0r0 \Rb"\25r3\"SS9\"SS\555r4"SS\)5r5"SS\*5r6"SS\'5r7"SS \$5r8"S!S"\%5r9\"S#S9\"S$S%\555r:\"S&S'\(\:55r;"S(S)\Rx5r=\"S*S9"S+S,\&\ 55r>"S-S.\Rx5r?\"S/S0\(55r@\"S1S9"S2S3\:\055rA/S4QrBg)5) dataclassN)initialization)Cache DynamicCache)GenerationMixin)create_causal_mask)BaseModelOutputWithPastCausalLMOutputWithPast)PreTrainedModel)Unpack) ModelOutputauto_docstringcan_return_tuplelogging)merge_with_config_defaults)is_torchdynamo_compiling)capture_outputs) AutoModel)LlamaAttentionLlamaDecoderLayerLlamaForCausalLMLlamaMLP LlamaModel LlamaRMSNormLlamaRotaryEmbeddingTransformersKwargs) CsmConfigCsmDepthDecoderConfig)CsmGenerationMixinz: Base class for the model autoregressive outputs. ) custom_introc\rSrSr%SrSr\RS-\S'Sr \RS-\S'Sr \ S-\S'Sr \ \RS4S-\S'Sr\ \RS4S-\S 'Sr\RS-\S 'Sr\RS-\S 'Sr\ S-\S 'Sr\ \RS4S-\S 'Sr\ \RS4S-\S'Sr\RS-\S'Srg)CsmOutputWithPast1a= loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. depth_decoder_loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction) of the depth decoder model. depth_decoder_logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the depth decoder (scores for each vocabulary token before SoftMax). depth_decoder_past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). depth_decoder_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. depth_decoder_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. backbone_loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction) of the backbone model. Nlosslogitspast_key_values. hidden_states attentionsdepth_decoder_lossdepth_decoder_logitsdepth_decoder_past_key_valuesdepth_decoder_hidden_statesdepth_decoder_attentions backbone_loss)__name__ __module__ __qualname____firstlineno____doc__r'torch FloatTensor__annotations__r(r)rr*tupler+r,r-r.r/r0r1__static_attributes__r2t/root/GenerationalWealth/GenerationalWealth/venv/lib/python3.13/site-packages/transformers/models/csm/modular_csm.pyr%r%1s8&*D%  d ")'+FE   $+$(OUT\(:>M5**C/047>7;Je'',-4;37))D0759%++d2926!54<6HLu'8'8#'=!>!ELEIeE$5$5s$:;dBI.2M5$$t+2r=r%c\rSrSrSrg) CsmRMSNormbr2Nr3r4r5r6r<r2r=r>r@r@br=r@c\rSrSrSrg)CsmRotaryEmbeddingfr2NrBr2r=r>rErEfrCr=rEc\rSrSrSrg)CsmMLPjr2NrBr2r=r>rHrHjrCr=rHc\rSrSrSrg) CsmAttentionnr2NrBr2r=r>rKrKnrCr=rKc\rSrSrSrg)CsmDecoderLayerrr2NrBr2r=r>rNrNrrCr=rNz[ The bare Csm Model outputting raw hidden-states without any specific head on top. c^\rSrSr%\\S'SrSrSrS/r S/r Sr Sr Sr Sr\\S.r\R&"5U4S j5rS rU=r$) CsmPreTrainedModelvconfigmodel)audiotextTrNr))r*r+c>[TU]U5 [U[5(aVURn[ US- 5H7n[ R"URSURRS9 M9 g[U[5(aa[ R"UR[R"URR5URR -5 gg)Nrg)meanstd)super _init_weights isinstanceCsmCodebooksHead num_codebooksrangeinitnormal_weightrSinitializer_rangeCsmBackboneModelEmbeddingscopy_audio_tokens_offsetsr8arange vocab_size)selfmoduler^i __class__s r>r[ CsmPreTrainedModel._init_weightss f% f. / /"00M=1,- V]]$++:W:WX.  : ; ; JJv22ELLAZAZ4[^b^i^i^t^t4t urQrQvsr (&*#*+#4"5N""&("  ]]_vvr=rQc,^\rSrSr%\\S'U4Sjr\\\ SS\ RS-S\ RS-S\ RS-S\ RS-S \S-S \ RS-S \S-S \\S \\-4Sjj555rSrU=r$)CsmDepthDecoderModelrSc>[TU]U5 [R"URUR -UR 5Ul[R"UR URSS9Ul gNF)bias) rZ__init__nn Embeddingr^rhbackbone_hidden_size embed_tokensLinear hidden_sizeinputs_embeds_projectorrirSrls r>rCsmDepthDecoderModel.__init__s]  LL&*>*>ARAR*RU[UpUpq')yy1L1LfN`N`gl'm$r=N input_idsbackbone_last_hidden_stateattention_mask position_idsr) inputs_embeds use_cachekwargsreturnc Ub&[5(d[RS5 SnUSLUSL- (a [S5eU(aUc[ UR S9nUbUR 5OSn UbURSOURSn Ub URO URn [R"XU -U S9nUcx[R"US- SS9n XR-n URX-5nUSS:HnUb X&SS2S4'O+[5(dU(a[RS 5 URU5n[!UR UUUUS 9nUnUR#S5nUR%UUS 9nUR&SUR R(HnU"U4UUUUUS .UD6nM UR+U5n[-UU(aUS 9$SS 9$)a* backbone_last_hidden_state (`torch.FloatTensor` of shape `(batch_size, backbone_hidden_size)`, *optional*): The last hidden state of the backbone model. Such input is required when the first codebook token (the one generated by the backbone model) is provided in the `input_ids` argument. NzCustom `position_ids` were provided but will be ignored. CSM depth decoder automatically determines position_ids and as it requires them to be identical across the batch, the provided position_ids will be ignored.z;You must specify exactly one of input_ids or inputs_embeds.)rSrrdevice)minzvWhen the first codebook token is provided, `backbone_last_hidden_state` should also be provided for correct inference.)rSrrr)r)r)rrr)rposition_embeddings)last_hidden_stater))rlogger warning_once ValueErrorrrSget_seq_lengthshaperr8rgclamprhrwarningrr unsqueeze rotary_emblayersnum_hidden_layersnormr )rirrrrr)rrrpast_seen_tokensinputs_seq_lengthr codebook_idxsoffsetinput_ids_are_first_codebook causal_maskr*r decoder_layers r>forwardCsmDepthDecoderModel.forwards &  #,D,F,F   w  L  -t"; <Z[ [ 0*$++>O?N?Z?99;`a6C6OM//2U^UdUdefUg)6)B%% HXHX||$4IZ6Zcij  !KK q(8a@M"__4F --i.@AM+7?a+? ()5&@ad#/116RNNQ44]C (;;')+%  & $--a0 "oom,oW![[)H4;;+H+HIM)*) /#$7 MJ -0 &+/8O  >B  r=)rr)NNNNNNN)r3r4r5r6r!r:rrrrr8 LongTensorr9Tensorrboolr rr;r rr<ryrzs@r>r|r|s !!n  .2?C.204(,26!%N ##d*N %*$5$5$<N  t+ N &&- N  N ((4/N $;N +,N  ( (N  N r=r|c2^\rSrSrU4SjrSSjrSrU=r$)r]c>[TU]5 X l[R"[ R "URS- X55Ulg)Nr)rZrr^r Parameterr8emptyrb)rirr^rhrls r>rCsmCodebooksHead.__init__s: *ll5;;t/A/AA/E{#_` r=c US- nURUn[URS5Vs/sH9n[RR USS2USS24X4R 5PM; nn[R"USS9nU$s snf)Nrrdim) rbr_rr functionallinearTr8stack)rir*codebook_indicescodebook_weight codebook_idxs r>rCsmCodebooksHead.forwards+a/++&67!&o&;&;A&> ? ?  MM q,/A!BODaDcDc d ?   Mq9   sAB)r^rbNr3r4r5r6rrr<ryrzs@r>r]r]sa  r=r]a$ The CsmDepthDecoder Model transformer, with a [`CsmCodebooksHead`] on top, which can be seen a position-specific language modeling head, allowing to use a different linear layer for each codebook (e.g. position 0 is the first codebook and uses the first codebook head, etc.) c^\rSrSrSrSrSrU4SjrSS\RS\ S-S\ S-S\RS-S\RS-S \ S-4 U4S jjjr\\SS\RS-S \RS-S\R"S-S \RS-S\ S-S\RS-S \RS-S\ S-S\ \R"-S\\S\\-4Sjj55rSrU=r$)CsmDepthDecoderForCausalLMi Nc>[TU]U5 U?[URUR UR 5Ul[U5Ul gr) rZrlm_headr]rr^rhcodebooks_headr|rTrs r>r#CsmDepthDecoderForCausalLM.__init__sE  L.v/A/A6CWCWY_YjYjk)&1 r=rnext_sequence_lengthr)rris_first_iterationc >[T U]"XX4U40UD6nU(dURS5 URS5 U$)Nrr)rZprepare_inputs_for_generationpop) rirrr)rrrr model_inputsrls r>r8CsmDepthDecoderForCausalLM.prepare_inputs_for_generationsMw< _m _e "   9 : (r=rrlabelsrlogits_to_keeprrc UbUR5OSn UbURSOURSn Ub URO URn [R"XS9U -nUR "S UUUUUUUS.U D6nUSn[ U [5(a!U S:Xa [SS5nO[U *S5nOU nURUSS2USS24UU5nUR5nSnUbBUSSS24R5nUR"S USURRUS.U D6n[UUURUR UR"S9$) a backbone_last_hidden_state (`torch.FloatTensor` of shape `(batch_size, backbone_hidden_size)`, *optional*): The last hidden state of the backbone model. Such input is required when the first codebook token (the one generated by the backbone model) is provided in the `input_ids` argument. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Nrrr)rrrrr)rr.)r(rrh shift_labels)r'r(r)r*r+r2)rrrr8rgrTr\intslicer contiguous loss_functionrSrhr r)r*r+)rirrrrr)rrrrrrseq_lenrroutputsr* slice_indicesr(r'rs r>r"CsmDepthDecoderForCausalLM.forward3s0@O?Z?99;`a,9,E-%%a(9??[\K])6)B%% HXHX <<?BRR**  'A)%+'      nc * *" %a %~ot < *M$$]1mQ3F%GIYZgIhi""$  !#qr'?557L%%dt{{7M7M\hlrD&#33!//))   r=)rrT)NNNNF) NNNNNNNNr)r3r4r5r6_tied_weights_keys_tp_plan_pp_planrr8rrrr9rrrrrr rr;r rr<ryrzs@r>rr sHH2,0(,2626*/##"Dj  ((4/  ((4/ !4K,.2?C.204(,26*.!%-.A ##d*A %*$5$5$<A  t+ A &&- A  A ((4/A   4'A $;A ell*A +,A  ' 'A A r=rc.^\rSrSrU4SjrSrSrU=r$)rdiyc>[TU]5 [R"URUR -UR 5UlURS[R"UR5UR -SS9 g)NrfF) persistent) rZrrrr^ codebook_sizerembed_audio_tokensregister_bufferr8rgrs r>r#CsmBackboneModelEmbeddings.__init__zsn "$,,0D0DvG[G[0[^d^p^p"q  "ELL1E1E$FI]I]$]jo  r=c^URXR-5nURSS9nU$)Nrr)rrfsum)rirrs r>r"CsmBackboneModelEmbeddings.forwards4// rdrdys r=rdcR^\rSrSrU4Sjr\\\U4Sj555rSr U=r $)CsmBackboneModelicD>[TU]U5 [U5Ulgr)rZrrdrrs r>rCsmBackboneModel.__init__s  6v>r=c $>[TU]"S0UD6$)a input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length, num_codebooks) or (batch_size, sequence_length)`): 1. (batch_size, sequence_length): corresponds to the input sequence prepared with the processor from the text prompt. Such input requires `input_values` to be provided so that audio can be encoded in codebook tokens and then merged with the text tokens. 2. (batch_size, sequence_length, num_codebooks): codebook tokens generated during the autoregressive decoding. Such input is not meant to be used by end users. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) r2)rZr)ri super_kwargsrls r>rCsmBackboneModel.forwards w...r=)r) r3r4r5r6rrrrrr<ryrzs@r>rrs,?  /  /r=rz The Csm model consists of two llama-like auto-regressive transformer models: a backbone model that predicts the first codebook token and a depth decoder that predicts the other codebook tokens. c^\rSrSrSS0rU4SjrSrSr\U4Sj5r U4Sjr SS \ RS -S \ RS -S \ RS -S \ RS -S\ RS -4 Sjjr SS \ RS\S -S\S -S\ RS -S\ R"S -4 U4Sjjjr\\SS \ RS -S \ RS -S\ RS -S \ RS -S\ RS -S\S -S\ R"S -S \ RS -S\S -S\\ R-S\\S\\-4Sjj55rSrU=r$)CsmForConditionalGenerationiz5backbone_model.embed_tokens.embed_audio_tokens.weightz'depth_decoder.model.embed_tokens.weightc>[TU]U5 URUl[R"UR URSS9Ul[R"URUR 5Ul [RU5Ul [RUR5Ul[ R""UR$5UlUR)5 gr)rZrrhrrrrrtext_vocab_sizeembed_text_tokensr _from_configbackbone_modelrdepth_decoder_config depth_decoderr from_config codec_config codec_model post_initrs r>r$CsmForConditionalGeneration.__init__s   ++yy!3!3V5F5FUS !#f.D.DfFXFX!Y.;;FC7DDVE`E`a$001D1DE r=c.URR$rrr)ris r>get_input_embeddings0CsmForConditionalGeneration.get_input_embeddingss""///r=c$XRlgrr)rivalues r>set_input_embeddings0CsmForConditionalGeneration.set_input_embeddingss+0(r=c>URSS5(a[T U]"U0UD6up4O[T U]"U0UD6nSn[U5n[ UR 5R 5VVs0sH"upxURU5(dMXvSU_M$ n nn[ URR 5RSS0U E5 U Hn[UR XW-5 M SU;aUW4$U$s snnf)Noutput_loading_infoFdepth_decoder__from_model_config) getrZfrom_pretrainedlenvarsgeneration_configitems startswithrupdatedelattr) clsargsrrT loading_infoprefix prefix_lenattrrdepth_decoder_attrsrls r>r +CsmForConditionalGeneration.from_pretraineds  ::+U 3 3"''"94"J6"J E<G+TSnURRR5nURSS5 UR 5HupV[ URX5-U5 M [ TU]"U0UD6 g)Nr transformers_version)rr to_diff_dictrrsetattrrZsave_pretrained)rirrrrrrrls r>r +CsmForConditionalGeneration.save_pretrainedsq!"00BBOOQ 6=.446KD D**FM5 A7 00r=Nr input_valuesinput_values_cutoffsrrcURU5nUGbN[RRUS5nX3S:R 5nXfS:n[ R "UR5URS9R[U5S5nXvRS5:n[ R"5 /n[X#5HupXS:n [U RSS- 5Hpn Xn XS-n U SX24nUR R#URS55nUR$R'SS5nUR)US5 Mr M [SU55n[ R*"UVs/sH7n[RRUSSSUURS- 45PM9 sn5nUR R-U5nSSS5 UR.R0nUU:HnUR2R5W5nUWUU'[ R6"SSUR.R84UR[ R:S 9UR.R<-nUR2R5U5R?S5nXR.R@:HnURCURE5S5UU'UbgURS5RCSSUR.R85nUUUU'UUU'US :HRGS S 9nS UUSUSSS24'UnXTS.$s snf!,(df  GN=f)a8 Merges the input_ids and input_values to produce a single inputs_embeds tensor: 1 - Infers the codec model on the input_values to retrieve codebook token. 2 - Embeds codebook tokens and places them at the correct positions in the inputs_embeds tensor. 3 - If labels are provided, expands them to match codebook dimensions and position the target codebook tokens in the inputs_embeds tensor. Args: input_ids (`torch.Tensor` of shape `(batch_size, sequence_length)`): The input ids to embed. input_values (`torch.Tensor` of shape `(batch_size, channels, audio_sequence_length)`): The audio input values to embed. input_values_cutoffs (`torch.Tensor` of shape `(batch_size, max_num_audio)`): The cutoffs of the audio input values relative to its batch index, padded with -1 when no audio. Nrrrrr.c3># UHoRSv M g7f)rN)r).0els r> QCsmForConditionalGeneration._merge_input_ids_with_input_values..s&O=Nrxx{=Ns)rdtypeiTas_tuple)rr)$rrrpaddiffr8rgmaxrexpandr rrxzipr_rrencode audio_codes transposeappendrget_audio_codes_maskrSaudio_token_idrronesr^longcodebook_eos_token_idsqueezeaudio_eos_token_idrepeatrnonzero)rirr"r#rr audio_lengthsinput_values_maskaudio_tokens_listbatch_input_valuesbatch_input_values_cutoffsrk start_idxend_idx audio_batch codec_outputs codebook_idsmax_audio_framesr)batched_audio_token_idsaudio_codes_maskr:audio_token_mask audio_embedsaudio_eos_frame_idsaudio_eos_embedsaudio_eos_token_masklabels_expanded depth_decoder_ignore_frames_idxss r>"_merge_input_ids_with_input_values>CsmForConditionalGeneration._merge_input_ids_with_input_valuessu*..y9  ##%==#4#45I6#R 01JKPPRM)!*;$A "<U"C&8i>O9O&P (,(8(8(?(? @U@UVW@X(Y '4'@'@'J'J1b'Q )00aA LGn$'&O=N&O#O */++`qr`qZ\R]]&&rAq!5EQR 5S+TU`qr+'$(#3#3#H#HIZ#[ !!$"[[77N(N: ..;;M( "M-(M-- M<rr)rrc 2>[T U]"S UUUUUS.UD6nUb|URS:XalURS5cZUR UURS5URS5URS5S9nUR USUSSS.5 U$) N)rrr)rrrrr"r#r)rr"r#r)rrrr2)rZrndimr rVr) rirrr)rrrr merged_inputsrls r>r9CsmForConditionalGeneration.prepare_inputs_for_generation1sw< !5+)'      Y^^q%8\=M=Mo=^=f CC##ZZ7%+ZZ0F%Gzz(+ DM   "/"@MZbLcrvw r=rrrrc  Ub.URS:XaURXXH5n U SnU SnSnUR"SUUUUUU S.U D6n U Sn[U [5(a [ U *S5OU nUR USS2USS245nSnSnSnSnUbUSS2SS2S4nUR"SUUURRS.U D6nUSS2SS2SS24S :HRS S 9)nUUS SURRS- 24n[RRUS SS9nURSS9nUUSUSS- SS24nUUnUR "SUUU SUS.U D6nUR"nUU-n[%UUUUU R&U R(U R*Ub UR,OSUb UR&OSUb UR(OSUbUR*S9 $SS9 $)a` input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length, num_codebooks) or (batch_size, sequence_length)`): 1. (batch_size, sequence_length): corresponds to the input sequence prepared with the processor from the text prompt. Such input requires `input_values` to be provided so that audio can be encoded in codebook tokens and then merged with the text tokens. 2. (batch_size, sequence_length, num_codebooks): codebook tokens generated during the autoregressive decoding. Such input is not meant to be used by end users. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) input_values_cutoffs (`torch.Tensor` of shape `(batch_size, max_num_audio)`, *optional*): Specify the end positions of audio segments within each batch entry, relative to the concatenated audio input. If a batch entry has fewer segments than the maximum, it is padded with -1. For example, in a batch of 2 sequences where the first contains 2 audio segments of length l1, and the second contains 1 audio segment of length l2, the input_values_cutoffs would be: [[l1, 2 * l1], [l2, -1]]. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[config.audio_token_id, -100, -101]`. Requires targeted `input_values` to be provided as audio tokens will be inferred from it using the `codec_model`. - `config.audio_token_id` indicates an audio frames (considering sequence length elements as frames) - `-100` will be ignored in the loss computation - `-101` indicates the audio frame will be used only for the backbone model (using the first codebook token as labels) Such labels can be prepared using `output_labels=True` when calling [`CsmProcessor`]. logits_to_keep (`int` or `torch.Tensor`, *optional*): Kept for compatibility. Does not support another value than: 1. `0`, which is equivalent to keeping all logits, used in the training regime 2. `1`, which is equivalent to keeping only the last logit, used in the generation regime Example: ```python >>> import torch >>> from transformers import CsmForConditionalGeneration, AutoProcessor >>> from datasets import load_dataset, Audio >>> model_id = "sesame/csm-1b" >>> torch_device = "cuda" if torch.cuda.is_available() else "cpu" >>> processor = AutoProcessor.from_pretrained(model_id) >>> ds = load_dataset("hf-internal-testing/dailytalk-dummy", split="train") >>> # ensure the audio is 24kHz >>> ds = ds.cast_column("audio", Audio(sampling_rate=24000)) >>> conversation = [] >>> # prepare a conversation with text and corresponding audio >>> for text, audio, speaker_id in zip(ds[:4]["text"], ds[:4]["audio"], ds[:4]["speaker_id"]): ... conversation.append( ... { ... "role": f"{speaker_id}", ... "content": [{"type": "text", "text": text}, {"type": "audio", "path": audio["array"]}], ... } ... ) >>> inputs = processor.apply_chat_template( ... conversation, ... tokenize=True, ... return_dict=True, ... output_labels=True, ... ).to(torch_device) >>> model = CsmForConditionalGeneration.from_pretrained(model_id, device_map=torch_device) >>> output = model(**inputs) >>> output.loss.backward() ```Nrrr)rrrr)rrr)r(rrhrr/r&r.r%)rTr-)rrr return_dictr) r'r1r,r(r)r*r+r-r.r/r0r2)rYrVrr\rrrrrSrhallr^rrr0rArr'r%r)r*r+r()rirr"rr#rr)rrrrrrZbackbone_outputsbackbone_hidden_statesrbackbone_logitsr'r1r,depth_decoder_outputsbackbone_labels train_maskdepth_decoder_input_ids train_idxsbackbone_last_hidden_statesdepth_decoder_labelss r>r#CsmForConditionalGeneration.forwardPsd  Y^^q%8 CC)=M*/:M"8,FI.. )%+'   "2!!48B>SV8W8W~ot4]k ,,'=aPQ>Q'RS ! $  $Q1WoO ..&4;;KaKaekM "!Q(+t388R8@@J&,Z&8>] @Y@Y\]@]>]9]&^ #&(mm&7&78OQW_`&7&a ##++T+:J*@APZ[\P]`aPacdAd*e '#)*#5 $($6$6%1+F# + %  % !"7!;!;  #55D '1",<<*88'22AVAb!6!=!=hl$0+@*O*O$0)>(K(KI^Ij%:%E%E  qu  r=)rrrrrrh)NNNN) NNNNNNNNNr)r3r4r5r6rrrr classmethodr r r8rrVrrrr9rrrrr rr;r%rr<ryrzs@r>rrsI @Aj0141*.,048&* PB<<$&PBllT)PB$llT1 PB  t# PB   PBj,0(,2626 ##"Dj  ((4/  ((4/ >.2,0.24804(,26*.!%-.Y ##d*Y llT)Y  t+ Y $llT1 Y &&- Y Y ((4/Y   4'Y $;Y ell*Y +,Y  " "Y Y r=r)rQrr|rr)C dataclassesrr8torch.nnrrr` cache_utilsrr generationr masking_utilsr modeling_outputsr r modeling_utilsr processing_utilsr utilsrrrr utils.genericrutils.import_utilsrutils.output_capturingrautorllama.modeling_llamarrrrrrrrconfiguration_csmr r!generation_csmr" get_loggerr3rr%r@rErHrKrNrQr|Moduler]rrdrr__all__r2r=r>rs" &.)/O-&KK7:5   @.   H %  '3 '3  '3V   -  X  >  '  vv  v<Y :'9Y Y xryy(d !1?d d N  /z//0 F "46HF  F R  r=