ZjmSSKJr SSKJr SSKJr SSKrSSKJr SSK J r SSK J r SSKJrJr SS KJr SS KJrJrJr SS KJr SS KJr SS KJrJr SSKJrJ r SSK!J"r"J#r# SSK$J%r% SSK&J'r'J(r(J)r)J*r*J+r+ SSK,J-r-J.r. SSK/J0r0 SSK1J2r2 SSK3J4r4 SSK5J6r6J7r7 SSK8J9r9 \+Rt"\;5r<\)"SS9\"SS\'555r=\"S5"SS \R|55r?"S!S"\R|5r@"S#S$\R|5rAS%rB\"S&5SKS'j5rCS(\RS)\ES*\R4S+jrFSLS,\R|S-\RS.\RS/\RS0\RS-S1\GS2\GS3\%\(4S4jjrH\"\C5"S5S6\R|55rI"S7S8\5rJ\)"S9S9\)"S:S;\#555rK\)"S<S=\K55rL"S>S?\R|5rM\)"S@S9"SASB\K\55rN"SCSD\R|5rO\)"SESF\K55rP\)"SGS9"SHSI\K\955rQ/SJQrRg)M)Callable) dataclass)OptionalN)initialization)ACT2FN)Cache DynamicCache)GenerationMixin)use_kernel_forward_from_hubuse_kernel_func_from_hubuse_kernelized_func)create_causal_mask)GradientCheckpointingLayer)BaseModelOutputWithPastCausalLMOutputWithPast)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack) ModelOutputTransformersKwargsauto_docstringcan_return_tuplelogging)maybe_autocastmerge_with_config_defaults)is_torchdynamo_compiling)capture_outputs) AutoModel) CsmConfigCsmDepthDecoderConfig)CsmGenerationMixinz: Base class for the model autoregressive outputs. ) custom_introc\rSrSr%SrSr\RS-\S'Sr \RS-\S'Sr \ S-\S'Sr \ \RS4S-\S'Sr\ \RS4S-\S 'Sr\RS-\S 'Sr\RS-\S 'Sr\ S-\S 'Sr\ \RS4S-\S 'Sr\ \RS4S-\S'Sr\RS-\S'Srg)CsmOutputWithPast3a= loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. depth_decoder_loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction) of the depth decoder model. depth_decoder_logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the depth decoder (scores for each vocabulary token before SoftMax). depth_decoder_past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). depth_decoder_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. depth_decoder_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. backbone_loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction) of the backbone model. Nlosslogitspast_key_values. hidden_states attentionsdepth_decoder_lossdepth_decoder_logitsdepth_decoder_past_key_valuesdepth_decoder_hidden_statesdepth_decoder_attentions backbone_loss)__name__ __module__ __qualname____firstlineno____doc__r+torch FloatTensor__annotations__r,r-r r.tupler/r0r1r2r3r4r5__static_attributes__r6u/root/GenerationalWealth/GenerationalWealth/venv/lib/python3.13/site-packages/transformers/models/csm/modeling_csm.pyr)r)3s8&*D%  d ")'+FE   $+$(OUT\(:>M5**C/047>7;Je'',-4;37))D0759%++d2926!54<6HLu'8'8#'=!>!ELEIeE$5$5s$:;dBI.2M5$$t+2rAr)RMSNormcx^\rSrSrS S\SS4U4SjjjrS\RS\R4SjrSr S r U=r $) CsmRMSNormcepsreturnNc>[TU]5 [R"[R "U55UlX lg)z) CsmRMSNorm is equivalent to T5LayerNorm N)super__init__nn Parameterr<onesweightvariance_epsilon)self hidden_sizerG __class__s rBrKCsmRMSNorm.__init__es/ ll5::k#:; #rAr.cURnUR[R5nUR S5R SSS9nU[R "X0R-5-nURURU5-$)Nr!T)keepdim) dtypetor<float32powmeanrsqrtrPrO)rQr. input_dtypevariances rBforwardCsmRMSNorm.forwardmsw#)) %((7  $$Q',,R,>% H?T?T4T(UU {{]--k:::rAc^[URR5SUR3$)Nz, eps=)r?rOshaperPrQs rB extra_reprCsmRMSNorm.extra_reprts* ))*+6$2G2G1HIIrA)rPrO)gư>) r7r8r9r:floatrKr<Tensorr`rer@ __classcell__rSs@rBrErEcsB$$$$$;U\\;ell;JJrArEc ^\rSrSr%\R \S'SS\4U4Sjjjr\ SS\S-S\ SS\ S-S \ S \ 44S jj5r\R"5\S 55rS rU=r$)CsmRotaryEmbeddingxinv_freqNconfigc>[TU]5 URUlURUlXlUR R SUlURnURS:wa[URnU"UR U5uo@l URSUSS9 URSUR5SS9 g)N rope_typedefaultrnF persistentoriginal_inv_freq) rJrKmax_position_embeddingsmax_seq_len_cachedoriginal_max_seq_lenrorope_parametersrqcompute_default_rope_parametersrattention_scalingregister_bufferclone)rQrodevice rope_init_fnrnrSs rBrKCsmRotaryEmbedding.__init__{s "("@"@$*$B$B! 44[A!%!E!E >>Y &.t~~>L+7 V+L(( ZeD 0(..2BuUrAr~z torch.deviceseq_lenrHz torch.Tensorc URSn[USS5=(d URUR-nSnSU[R "SUS[R S9RU[RS9U- -- nXe4$) aH Computes the inverse frequencies according to the original RoPE implementation Args: config ([`~transformers.PreTrainedConfig`]): The model configuration. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin (unused in this type of RoPE). rope_thetahead_dimNg?rr!rXr~rX) rygetattrrRnum_attention_headsr<arangeint64rYrg)ror~rbasedimattention_factorrns rBrz2CsmRotaryEmbedding.compute_default_rope_parameterss&%%l3fj$/c63E3EIcIc3c U\\!S!5;;?BB&X]XcXcBdgjj k ))rAcLURSSS2S4R5RURSSS5R UR 5nUSS2SSS24R5n[ UR R[5(a0UR RS:waUR ROSn[USS9 UR5UR5-RSS5n[R"Xf4SS 9nUR5UR-nUR5UR-n SSS5 WR UR S 9W R UR S 94$!,(df  N@=f) NrrVr#mpscpuF) device_typeenabledr!rr)rnrgexpandrcrYr~ isinstancetypestrr transposer<catcosr{sinrX) rQx position_idsinv_freq_expandedposition_ids_expandedrfreqsembrrs rBr`CsmRotaryEmbedding.forwardsN!MM$4-8>>@GG HZHZ[\H]_acdehhijiqiqr ,QaZ 8 > > @'1!((--'E'E!((--[`J`ahhmmfk  U C&,,.1F1L1L1NNYYZ[]^_E))UN3C'')d444C'')d444C D vvAGGv$cff177f&;;; D Cs BF F#)r{rorwrxrqNNNN)r7r8r9r:r<rhr>r$rK staticmethodrintr?rgrzno_gradrr`r@rirjs@rBrlrlxsllVyVV #'+/"*D *(*t* ~u$ % **: ]]_ < [TU]5 XlURUlURUl[ R "URURURS9Ul[ R "URURURS9Ul [ R "URURURS9Ul [URUl g)Nbias)rJrKrorRintermediate_sizerLLinearmlp_bias gate_projup_proj down_projr hidden_actact_fnrQrorSs rBrKCsmMLP.__init__s  !--!'!9!94#3#3T5K5KRXRaRabyy!1!143I3IPVP_P_` 4#9#94;K;KRXRaRabV../ rAcURURURU55URU5-5nU$r)rrrr)rQrrs rBr`CsmMLP.forwards6NN4;;t~~a/@#ADLLQRO#ST rA)rrorrrRrrr7r8r9r:rKr`r@rirjs@rBrrs0rArcUSSURSS-24nUSURSS-S24n[R"U*U4SS9$)z*Rotates half the hidden dims of the input..NrVr!r)rcr<r)rx1x2s rB rotate_halfrsZ 3"!''"+"" " #B 3 q " " #B 99rc2YB ''rArotary_pos_embcURU5nURU5nX-[U5U--nX-[U5U--nXV4$)aIApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. ) unsqueezer)qkrr unsqueeze_dimq_embedk_embeds rBapply_rotary_pos_embrsS& -- &C -- &Cw;q>C/0Gw;q>C/0G  rAr.n_reprHcURup#pEUS:XaU$USS2SS2SSS2SS24RX#XU5nURX#U-XE5$)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) r#N)rcrreshape)r.rbatchnum_key_value_headsslenrs rB repeat_kvrs_ 2?1D1D.E z!!Qa"23::5W\dlmM  e(CT TTrAmodulequerykeyvalueattention_maskscalingdropoutkwargsc [X R5n[X0R5n [R"XR SS55U-n UbX-n [ R RU S[RS9RUR5n [ R RXURS9n [R"X5n U R SS5R5n X4$)Nr!rrV)rrX)ptrainingr#)rnum_key_value_groupsr<matmulrrL functionalsoftmaxrZrYrXrr contiguous) rrrrrrrr key_states value_states attn_weights attn_outputs rBeager_attention_forwardrs3 ; ;[TU]5 XlX l[ USUR UR -5UlUR UR-Ul URS-Ul URUl SUl [R"UR UR UR-URS9Ul[R"UR URUR-URS9Ul[R"UR URUR-URS9Ul[R"UR UR-UR URS9Ulg)NrgTr)rJrKrorrrRrrrrrattention_dropout is_causalrLrattention_biasq_projk_projv_projo_projrQrorrSs rBrKCsmAttention.__init__sI  " F4F4F&JdJd4de $*$>$>&B\B\$\!}}d* !'!9!9ii    : :T]] JQWQfQf  ii    : :T]] JQWQfQf  ii    : :T]] JQWQfQf  ii  & & 68J8JQWQfQf  rANr.position_embeddingsrr-rrHc URSSn/UQSPURP7nURU5RU5R SS5nUR U5RU5R SS5n UR U5RU5R SS5n Uup[XX5upUbURXUR5up[R"URR[5n U "UUU U U4UR(dSO UR UR"S.UD6upUR$"/UQSP76R'5nUR)U5nX4$)NrVr#r!)rr)rcrrviewrrrrupdaterr get_interfacero_attn_implementationrrrrrrr)rQr.rrr-r input_shape hidden_shape query_statesrrrrattention_interfacerrs rBr`CsmAttention.forward*s~$))#2. 88b8$--8 {{=166|DNNqRST [[/44\BLLQPQR {{=166|DNNqRST &#7 RU#[  &'6'='=jX\XfXf'g $J(?(M(M KK , ,.E) %8      % $}}C$2H2HLL %  % ! "));;;;FFH kk+. ((rA) rrorrrrrrrrrr)r7r8r9r:r;r$rrKr<rhr?r rrr`r@rirjs@rBrrsG y S 4IM.2(, &)||&)#5<<#=>E&) t+ &)  &) +, &) u||U\\) *&)&)rArc^\rSrSrS\S\4U4SjjrSS\RS\RS-S\RS-S \ S-S \ S-S \ \R\R4S-S \ \S \R4SjjrSrU=r$)CsmDecoderLayeriSrorc >[TU]5 URUl[XS9Ul[ U5Ul[URURS9Ul [URURS9Ul g)N)rorrG) rJrKrRr self_attnrmlprE rms_norm_epsinput_layernormpost_attention_layernormrs rBrKCsmDecoderLayer.__init__Tsi !--%VI&>)&*<*<&BUBUV(263E3E6K^K^(_%rANr.rrr- use_cacherrrHc UnURU5nUR"SUUUUUUS.UD6upX-nUnURU5nURU5nX-nU$)N)r.rrr-rrr6)rrrr) rQr.rrr-rrrresidual_s rBr`CsmDecoderLayer.forward^s!,,]; >> ')%+ 3    !0 !55mD /  0 rA)rRrrrr)NNNFN)r7r8r9r:r$rrKr<rh LongTensorr boolr?rrr`r@rirjs@rBrrSs`y`S`/304(,!&HL|| t+&&-    $; #5<<#=>E+, rArz[ The bare Csm Model outputting raw hidden-states without any specific head on top. c^\rSrSr%\\S'SrSrSrS/r S/r Sr Sr Sr Sr\\S.r\R&"5U4S j5rS rU=r$) CsmPreTrainedModeli~romodel)audiotextTrr-)r.r/c>[TU]U5 [U[5(aVURn[ US- 5H7n[ R"URSURRS9 M9 g[U[5(aa[ R"UR[R"URR5URR -5 gg)Nr#r)r\std)rJ _init_weightsrCsmCodebooksHead num_codebooksrangeinitnormal_rOroinitializer_rangeCsmBackboneModelEmbeddingscopy_audio_tokens_offsetsr<r vocab_size)rQrrirSs rBr CsmPreTrainedModel._init_weightss f% f. / /"00M=1,- V]]$++:W:WX.  : ; ; JJv22ELLAZAZ4[^b^i^i^t^t4t ubase_model_prefixinput_modalitiessupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_supports_sdpa_can_compile_fullgraph_supports_attention_backendrr_can_record_outputsr<rrr@rirjs@rBrr~sr (&*#*+#4"5N""&("  ]]_vvrArc,^\rSrSr%\\S'U4Sjr\\\ SS\ RS-S\ RS-S\ RS-S\ RS-S \S-S \ RS-S \S-S \\S \\-4Sjj555rSrU=r$)CsmDepthDecoderModeliroc j>[TU]U5 URUlURUl[ R "URUR-UR5Ul [ R"[UR5Vs/sHn[X5PM sn5Ul[UR UR"S9Ul['US9UlSUl[ R,"URUR SS9UlUR15 gs snf)NrroFr)rJrK pad_token_id padding_idxr rL Embeddingrbackbone_hidden_size embed_tokens ModuleListrnum_hidden_layersrlayersrErRrnormrl rotary_embgradient_checkpointingrinputs_embeds_projector post_initrs rBrKCsmDepthDecoderModel.__init__s  !.. ++LL&*>*>ARAR*RU[UpUpqmmAFvG_G_A` aA`I_V /A` a  v11v7J7JK ,F;&+#')yy1L1LfN`N`gl'm$  bsD0N input_idsbackbone_last_hidden_staterrr- inputs_embedsrrrHc Ub&[5(d[RS5 SnUSLUSL- (a [S5eU(aUc[ UR S9nUbUR 5OSn UbURSOURSn Ub URO URn [R"XU -U S9nUcx[R"US- SS9n XR-n URX-5nUSS:HnUb X&SS2S4'O+[5(dU(a[RS 5 URU5n[!UR UUUUS 9nUnUR#S5nUR%UUS 9nUR&SUR R(HnU"U4UUUUUS .UD6nM UR+U5n[-UU(aUS 9$SS 9$)a* backbone_last_hidden_state (`torch.FloatTensor` of shape `(batch_size, backbone_hidden_size)`, *optional*): The last hidden state of the backbone model. Such input is required when the first codebook token (the one generated by the backbone model) is provided in the `input_ids` argument. NzCustom `position_ids` were provided but will be ignored. CSM depth decoder automatically determines position_ids and as it requires them to be identical across the batch, the provided position_ids will be ignored.z;You must specify exactly one of input_ids or inputs_embeds.r0rr#r~)minzvWhen the first codebook token is provided, `backbone_last_hidden_state` should also be provided for correct inference.rorArr-rr)rrr-rrlast_hidden_stater-)rlogger warning_once ValueErrorr roget_seq_lengthrcr~r<rclampr r5warningr<rrr:r8r7r9r)rQr?r@rrr-rArrpast_seen_tokensinputs_seq_lengthr~ codebook_idxsoffsetinput_ids_are_first_codebook causal_maskr.r decoder_layers rBr`CsmDepthDecoderModel.forwards &  #,D,F,F   w  L  -t"; <Z[ [ 0*$++>O?N?Z?99;`a6C6OM//2U^UdUdefUg)6)B%% HXHX||$4IZ6Zcij  !KK q(8a@M"__4F --i.@AM+7?a+? ()5&@ad#/116RNNQ44]C (;;')+%  & $--a0 "oom,oW![[)H4;;+H+HIM)*) /#$7 MJ -0 &+/8O  >B  rA)r5r;r<r8r9r2r:r )NNNNNNN)r7r8r9r:r%r>rKrr rr<r r=rhr rrrr?rr`r@rirjs@rBr.r.s !!  .2?C.204(,26!%N ##d*N %*$5$5$<N  t+ N &&- N  N ((4/N $;N +,N  ( (N  N rAr.c2^\rSrSrU4SjrSSjrSrU=r$)ri c>[TU]5 X l[R"[ R "URS- X55Ulg)Nr#)rJrKrrLrMr<emptyrO)rQrRrr rSs rBrKCsmCodebooksHead.__init__ s: *ll5;;t/A/AA/E{#_` rAc US- nURUn[URS5Vs/sH9n[RR USS2USS24X4R 5PM; nn[R"USS9nU$s snf)Nr#rr) rOrrcrLrlinearTr<stack)rQr.codebook_indicescodebook_weight codebook_idxs rBr`CsmCodebooksHead.forwards+a/++&67!&o&;&;A&> ? ?  MM q,/A!BODaDcDc d ?   Mq9   sAB)rrOrrrjs@rBrr sa  rAra$ The CsmDepthDecoder Model transformer, with a [`CsmCodebooksHead`] on top, which can be seen a position-specific language modeling head, allowing to use a different linear layer for each codebook (e.g. position 0 is the first codebook and uses the first codebook head, etc.) c^\rSrSrSrSrSrU4Sjr\\ SS\ RS-S\ RS-S\ RS-S\ RS-S\S-S \ RS-S \ RS-S \S-S \\ R-S \\S\\-4Sjj55rSS\ RS\S-S\S-S\ RS-S \ RS-S\S-4 U4SjjjrSrU=r$)CsmDepthDecoderForCausalLMiNc>[TU]U5 [U5UlURUl[ UR URUR5UlUR5 gr) rJrKr.rr rrRrcodebooks_headr=rs rBrK#CsmDepthDecoderForCausalLM.__init__*sY  )&1  ++.v/A/A6CWCWY_YjYjk rAr?r@rrr-rAlabelsrlogits_to_keeprrHc UbUR5OSn UbURSOURSn Ub URO URn [R"XS9U -nUR "S UUUUUUUS.U D6nUSn[ U [5(a!U S:Xa [SS5nO[U *S5nOU nURUSS2USS24UU5nUR5nSnUbBUSSS24R5nUR"S USURRUS.U D6n[UUURUR UR"S9$) a backbone_last_hidden_state (`torch.FloatTensor` of shape `(batch_size, backbone_hidden_size)`, *optional*): The last hidden state of the backbone model. Such input is required when the first codebook token (the one generated by the backbone model) is provided in the `input_ids` argument. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Nrr#rC)r?r@rrr-rAr.)r,rhr  shift_labels)r+r,r-r.r/r6)rLrcr~r<rrrrslicerfr loss_functionror rr-r.r/)rQr?r@rrr-rArhrrirrOrr~r_outputsr. slice_indicesr,r+rks rBr`"CsmDepthDecoderForCausalLM.forward3s0@O?Z?99;`a,9,E-%%a(9??[\K])6)B%% HXHX <<?BRR**  'A)%+'      nc * *" %a %~ot < *M$$]1mQ3F%GIYZgIhi""$  !#qr'?557L%%dt{{7M7M\hlrD&#33!//))   rAnext_sequence_lengthis_first_iterationc >[T U]"XX4U40UD6nU(dURS5 URS5 U$)Nr@r)rJprepare_inputs_for_generationpop) rQr?rqr-rrArrr model_inputsrSs rBrt8CsmDepthDecoderForCausalLM.prepare_inputs_for_generationxsMw< _m _e "   9 : (rA)rfrr ) NNNNNNNNr)NNNNF)r7r8r9r:_tied_weights_keys_tp_plan_pp_planrKrrr<r r=rhr rrrrr?rr`rtr@rirjs@rBrdrdsHH.2?C.204(,26*.!%-.A ##d*A %*$5$5$<A  t+ A &&- A  A ((4/A   4'A $;A ell*A +,A  ' 'A A L,0(,2626*/##"Dj  ((4/  ((4/ !4KrArdc.^\rSrSrU4SjrSrSrU=r$)ric>[TU]5 [R"URUR -UR 5UlURS[R"UR5UR -SS9 g)NrFrs) rJrKrLr3r codebook_sizerRembed_audio_tokensr|r<rrs rBrK#CsmBackboneModelEmbeddings.__init__sn "$,,0D0DvG[G[0[^d^p^p"q  "ELL1E1E$FI]I]$]jo  rAc^URXR-5nURSS9nU$)Nr!r)r~rsum)rQr?rAs rBr`"CsmBackboneModelEmbeddings.forwards4// [TU]U5 URUlURUl[ U5Ul[R"[UR5Vs/sHn[X5PM sn5Ul [URURS9Ul[#US9UlSUlUR)5 gs snf)Nrr0F)rJrKr1r2r rr5rLr6rr7rr8rErRrr9rlr:r;r=rs rBrKCsmBackboneModel.__init__s  !.. ++6v>mmAFvG_G_A` aA`I_V /A` a  v11v7J7JK ,F;&+#  bs*CNr?rrr-rArrrHc >USLUSL- (a [S5eUcURU5nU(aUc[URS9nUcUUbUR 5OSn[ R "URSURS9U-nURS5n[URUUUUS9n Un URXS9n URSURRHn U "U 4U U UUUS .UD6n M URU 5n [U US 9$) a input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length, num_codebooks) or (batch_size, sequence_length)`): 1. (batch_size, sequence_length): corresponds to the input sequence prepared with the processor from the text prompt. Such input requires `input_values` to be provided so that audio can be encoded in codebook tokens and then merged with the text tokens. 2. (batch_size, sequence_length, num_codebooks): codebook tokens generated during the autoregressive decoding. Such input is not meant to be used by end users. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) Nz:You must specify exactly one of input_ids or inputs_embedsr0rr#rCrErF)rrrr-rrG)rKr5r rorLr<rrcr~rrr:r8r7r9r) rQr?rrr-rArrrOrTr.rrUs rBr`CsmBackboneModel.forwardsF2  -t"; <YZ Z  *.*;*;I*FM 0*$++>O  CRC^==?de  << (;(;A(>}G[G[\_ooL'11!4L(;;')+%  & "oomoW![[)H4;;+H+HIM)*$7) /# MJ -0 &++  rA)r5r;r8r9r2r:r )NNNNNN)r7r8r9r:rKrr rr<r rhr r=rrrrr`r@rirjs@rBrrs  .2.204(,26!%> ##d*>  t+> &&- >  > ((4/ > $;> +,>  !>  > rArz The Csm model consists of two llama-like auto-regressive transformer models: a backbone model that predicts the first codebook token and a depth decoder that predicts the other codebook tokens. c^\rSrSrSS0rU4SjrSrSr\U4Sj5r U4Sjr SS \ RS -S \ RS -S \ RS -S \ RS -S\ RS -4 Sjjr SS \ RS\S -S\S -S\ RS -S\ R"S -4 U4Sjjjr\\SS \ RS -S \ RS -S\ RS -S \ RS -S\ RS -S\S -S\ R"S -S \ RS -S\S -S\\ R-S\\S\\-4Sjj55rSrU=r$)CsmForConditionalGenerationiz5backbone_model.embed_tokens.embed_audio_tokens.weightz'depth_decoder.model.embed_tokens.weightc>[TU]U5 URUl[R"UR URSS9Ul[R"URUR 5Ul [RU5Ul [RUR5Ul[ R""UR$5UlUR)5 g)NFr)rJrKr rLrrRlm_headr3text_vocab_sizeembed_text_tokensr _from_configbackbone_modelrddepth_decoder_config depth_decoderr" from_config codec_config codec_modelr=rs rBrK$CsmForConditionalGeneration.__init__s   ++yy!3!3V5F5FUS !#f.D.DfFXFX!Y.;;FC7DDVE`E`a$001D1DE rAc.URR$rrr5rds rBget_input_embeddings0CsmForConditionalGeneration.get_input_embeddingss""///rAc$XRlgrr)rQrs rBset_input_embeddings0CsmForConditionalGeneration.set_input_embeddings s+0(rAc>URSS5(a[T U]"U0UD6up4O[T U]"U0UD6nSn[U5n[ UR 5R 5VVs0sH"upxURU5(dMXvSU_M$ n nn[ URR 5RSS0U E5 U Hn[UR XW-5 M SU;aUW4$U$s snnf)Noutput_loading_infoFdepth_decoder__from_model_config) getrJfrom_pretrainedlenvarsgeneration_configitems startswithrrdelattr) clsargsrr loading_infoprefix prefix_lenattrrdepth_decoder_attrsrSs rBr+CsmForConditionalGeneration.from_pretrained s  ::+U 3 3"''"94"J6"J E<G+TSnURRR5nURSS5 UR 5HupV[ URX5-U5 M [ TU]"U0UD6 g)Nrtransformers_version)rr to_diff_dictrursetattrrJsave_pretrained)rQrrrrrrrSs rBr+CsmForConditionalGeneration.save_pretrained'sq!"00BBOOQ 6=.446KD D**FM5 A7 00rANr? input_valuesinput_values_cutoffsrhrHcURU5nUGbN[RRUS5nX3S:R 5nXfS:n[ R "UR5URS9R[U5S5nXvRS5:n[ R"5 /n[X#5HupXS:n [U RSS- 5Hpn Xn XS-n U SX24nUR R#URS55nUR$R'SS5nUR)US5 Mr M [SU55n[ R*"UVs/sH7n[RRUSSSUURS- 45PM9 sn5nUR R-U5nSSS5 UR.R0nUU:HnUR2R5W5nUWUU'[ R6"SSUR.R84UR[ R:S 9UR.R<-nUR2R5U5R?S5nXR.R@:HnURCURE5S5UU'UbgURS5RCSSUR.R85nUUUU'UUU'US :HRGS S 9nS UUSUSSS24'UnXTS.$s snf!,(df  GN=f)a8 Merges the input_ids and input_values to produce a single inputs_embeds tensor: 1 - Infers the codec model on the input_values to retrieve codebook token. 2 - Embeds codebook tokens and places them at the correct positions in the inputs_embeds tensor. 3 - If labels are provided, expands them to match codebook dimensions and position the target codebook tokens in the inputs_embeds tensor. Args: input_ids (`torch.Tensor` of shape `(batch_size, sequence_length)`): The input ids to embed. input_values (`torch.Tensor` of shape `(batch_size, channels, audio_sequence_length)`): The audio input values to embed. input_values_cutoffs (`torch.Tensor` of shape `(batch_size, max_num_audio)`): The cutoffs of the audio input values relative to its batch index, padded with -1 when no audio. Nr#rrrCrVr#.c3># UHoRSv M g7f)rN)rc).0els rB QCsmForConditionalGeneration._merge_input_ids_with_input_values..as&O=Nrxx{=NsriTas_tuple)rArh)$rrLrpaddiffr<rmaxr~rrrrziprrcrencode audio_codesrappendr^get_audio_codes_maskroaudio_token_idrr5rNrlongcodebook_eos_token_idsqueezeaudio_eos_token_idrepeatrnonzero)rQr?rrrhrA audio_lengthsinput_values_maskaudio_tokens_listbatch_input_valuesbatch_input_values_cutoffsr! start_idxend_idx audio_batch codec_outputs codebook_idsmax_audio_framesrbatched_audio_token_idsaudio_codes_maskraudio_token_mask audio_embedsaudio_eos_frame_idsaudio_eos_embedsaudio_eos_token_masklabels_expanded depth_decoder_ignore_frames_idxss rB"_merge_input_ids_with_input_values>CsmForConditionalGeneration._merge_input_ids_with_input_values1su*..y9  ##%==#4#45I6#R 01JKPPRM)!*;$A "<U"C&8i>O9O&P (,(8(8(?(? @U@UVW@X(Y '4'@'@'J'J1b'Q )00aA LGn$'&O=N&O#O */++`qr`qZ\R]]&&rAq!5EQR 5S+TU`qr+'$(#3#3#H#HIZ#[ !!$"[[77N(N: ..;;M( "M-(M-- M<rqr-rrAc 2>[T U]"S UUUUUS.UD6nUb|URS:XalURS5cZUR UURS5URS5URS5S9nUR USUSSS.5 U$) N)r?rqr-rrAr!rArrrh)r?rrrh)rArhr?r6)rJrtndimrrr) rQr?rqr-rrArrv merged_inputsrSs rBrt9CsmForConditionalGeneration.prepare_inputs_for_generationsw< !5+)'      Y^^q%8\=M=Mo=^=f CC##ZZ7%+ZZ0F%Gzz(+ DM   "/"@MZbLcrvw rArrrirc  Ub.URS:XaURXXH5n U SnU SnSnUR"SUUUUUU S.U D6n U Sn[U [5(a [ U *S5OU nUR USS2USS245nSnSnSnSnUbUSS2SS2S4nUR"SUUURRS.U D6nUSS2SS2SS24S :HRS S 9)nUUS SURRS- 24n[RRUS SS9nURSS9nUUSUSS- SS24nUUnUR "SUUU SUS.U D6nUR"nUU-n[%UUUUU R&U R(U R*Ub UR,OSUb UR&OSUb UR(OSUbUR*S9 $SS9 $)a` input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length, num_codebooks) or (batch_size, sequence_length)`): 1. (batch_size, sequence_length): corresponds to the input sequence prepared with the processor from the text prompt. Such input requires `input_values` to be provided so that audio can be encoded in codebook tokens and then merged with the text tokens. 2. (batch_size, sequence_length, num_codebooks): codebook tokens generated during the autoregressive decoding. Such input is not meant to be used by end users. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) input_values_cutoffs (`torch.Tensor` of shape `(batch_size, max_num_audio)`, *optional*): Specify the end positions of audio segments within each batch entry, relative to the concatenated audio input. If a batch entry has fewer segments than the maximum, it is padded with -1. For example, in a batch of 2 sequences where the first contains 2 audio segments of length l1, and the second contains 1 audio segment of length l2, the input_values_cutoffs would be: [[l1, 2 * l1], [l2, -1]]. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[config.audio_token_id, -100, -101]`. Requires targeted `input_values` to be provided as audio tokens will be inferred from it using the `codec_model`. - `config.audio_token_id` indicates an audio frames (considering sequence length elements as frames) - `-100` will be ignored in the loss computation - `-101` indicates the audio frame will be used only for the backbone model (using the first codebook token as labels) Such labels can be prepared using `output_labels=True` when calling [`CsmProcessor`]. logits_to_keep (`int` or `torch.Tensor`, *optional*): Kept for compatibility. Does not support another value than: 1. `0`, which is equivalent to keeping all logits, used in the training regime 2. `1`, which is equivalent to keeping only the last logit, used in the generation regime Example: ```python >>> import torch >>> from transformers import CsmForConditionalGeneration, AutoProcessor >>> from datasets import load_dataset, Audio >>> model_id = "sesame/csm-1b" >>> torch_device = "cuda" if torch.cuda.is_available() else "cpu" >>> processor = AutoProcessor.from_pretrained(model_id) >>> ds = load_dataset("hf-internal-testing/dailytalk-dummy", split="train") >>> # ensure the audio is 24kHz >>> ds = ds.cast_column("audio", Audio(sampling_rate=24000)) >>> conversation = [] >>> # prepare a conversation with text and corresponding audio >>> for text, audio, speaker_id in zip(ds[:4]["text"], ds[:4]["audio"], ds[:4]["speaker_id"]): ... conversation.append( ... { ... "role": f"{speaker_id}", ... "content": [{"type": "text", "text": text}, {"type": "audio", "path": audio["array"]}], ... } ... ) >>> inputs = processor.apply_chat_template( ... conversation, ... tokenize=True, ... return_dict=True, ... output_labels=True, ... ).to(torch_device) >>> model = CsmForConditionalGeneration.from_pretrained(model_id, device_map=torch_device) >>> output = model(**inputs) >>> output.loss.backward() ```Nr!rArh)r?rrr-rArr)r,rhr r#rrVr.r)rTr)r?r@r return_dictrh) r+r5r0r,r-r.r/r1r2r3r4r6)rrrrrrlrrmror allrrLrrrrr+r)r-r.r/r,)rQr?rrrrr-rArhrrirrbackbone_outputsbackbone_hidden_statesrobackbone_logitsr+r5r0depth_decoder_outputsbackbone_labels train_maskdepth_decoder_input_ids train_idxsbackbone_last_hidden_statesdepth_decoder_labelss rBr`#CsmForConditionalGeneration.forwardsd  Y^^q%8 CC)=M*/:M"8,FI.. )%+'   "2!!48B>SV8W8W~ot4]k ,,'=aPQ>Q'RS ! $  $Q1WoO ..&4;;KaKaekM "!Q(+t388R8@@J&,Z&8>] @Y@Y\]@]>]9]&^ #&(mm&7&78OQW_`&7&a ##++T+:J*@APZ[\P]`aPacdAd*e '#)*#5 $($6$6%1+F# + %  % !"7!;!;  #55D '1",<<*88'22AVAb!6!=!=hl$0+@*O*O$0)>(K(KI^Ij%:%E%E  qu  rA)rrrrrr )NNNN) NNNNNNNNNr)r7r8r9r:rxrKrr classmethodrrr<rhrr rr r=rtrrrrrr?r)r`r@rirjs@rBrrsI @Aj0141*.,048&* PB<<$&PBllT)PB$llT1 PB  t# PB   PBj,0(,2626 ##"Dj  ((4/  ((4/ >.2,0.24804(,26*.!%-.Y ##d*Y llT)Y  t+ Y $llT1 Y &&- Y Y ((4/Y   4'Y $;Y ell*Y +,Y  " "Y Y rAr)rrr.rdr)r#)r)Scollections.abcr dataclassesrtypingrr<torch.nnrLrr activationsr cache_utilsr r generationr integrationsr r r masking_utilsrmodeling_layersrmodeling_outputsrrmodeling_rope_utilsrrmodeling_utilsrrprocessing_utilsrutilsrrrrr utils.genericrrutils.import_utilsrutils.output_capturingr autor"configuration_csmr$r%generation_csmr& get_loggerr7rIr)ModulerErlrrrrhrrrgrrrrr.rrdrrr__all__r6rArBrs*%! &!.)ff/9OKF&__G:5?.   H %  '3 '3  '3TY'JJ(J(><>