ZjMdSSKrSSKJrJr SSKJr SSKrSSKJr SSK Js J r SSK J r SSKJr SSKJr SSKJr SS KJrJr SS KJr SS KJrJrJrJr SS KJ r SS K!J"r" SSK#J$r$ \RJ"\&5r'\\"SS\555r(\\"SS\555r)"SS\RT5r+"SS\RT5r,\"S5"SS\RT55r-"SS\RT5r.S1S\RTS\R^S\R^S \R^S!\R^S-S"\0S#\0\1-S$\\4S%jjr2"S&S'\RT5r3"S(S)\RT5r4\"S*S+\55r5\"S,S-\555r6"S.S/\55r7/S0Qr8g)2N)CallableSequence) dataclass)initialization)use_kernel_forward_from_hub)FlashAttentionKwargs)BaseModelOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tuplelogging)merge_with_config_defaults)capture_outputs) TimesFmConfigcj\rSrSr%SrSr\RS-\S'Sr \RS-\S'Sr g) TimesFmOutput,z loc (`torch.Tensor` of shape `(batch_size, )`): The mean of the time series inputs. scale (`torch.Tensor` of shape `(batch_size,)`): The scale of the time series inputs. Nlocscale) __name__ __module__ __qualname____firstlineno____doc__rtorchTensor__annotations__r__static_attributes__r}/root/GenerationalWealth/GenerationalWealth/venv/lib/python3.13/site-packages/transformers/models/timesfm/modeling_timesfm.pyrr,s/ $C #!%E5<<$ %r%rc\rSrSr%SrSr\RS-\S'Sr \RS-\S'Sr \R\ -S-\S'Sr g)TimesFmOutputForPrediction:a mean_predictions (`torch.Tensor` of shape `(batch_size, sequence_length)`): The mean predictions of the time series. full_predictions (`torch.Tensor` of shape `(batch_size, sequence_length)`): The full predictions of the time series including the mean and the quantiles. loss (`torch.Tensor` of shape `(1,)`, *optional*, returned when `future_values` is provided): The loss of the TimesFM model. Nmean_predictionsfull_predictionslossr) rrrrr r*r!r"r#r+r,floatr$rr%r&r(r(:sI-1ellT)0,0ellT)0(,D%,,  %,r%r(c>^\rSrSrSrS\4U4SjjrSSjrSrU=r $) TimesFmMLPKzPax MLP in pytorch.configc>[TU]5 URnURn[R "X#5Ul[R "X25Ul[R"USS9Ul g)Nư>)normalized_shapeeps) super__init__ hidden_sizeintermediate_sizennLinear gate_proj down_proj LayerNorm layer_norm)selfr1r8r9 __class__s r&r7TimesFmMLP.__init__NsX (( "44;B#4B,, Nr%cURU5nURU5n[R"U5nUR U5nUbUSUSS2SS2S4- -nXQ-$)N?)r?r<Frelur=)r@xpaddingsgate_inpgateoutputss r&forwardTimesFmMLP.forwardWsa??1%~~h'vvd|..&  x1d ';!;[TU]5 XlX lX0l[ R "X5Ul[ R"5Ul [ R "X#5Ul [ R "X5Ul grN) r6r7 input_dims hidden_dims output_dimsr:r; input_layerSiLU activation output_layerresidual_layer)r@rVrWrXrAs r&r7TimesFmResidualBlock.__init__ds_ $&&99Z='')IIk? ii @r%cURU5nURU5nURU5nURU5nX4-$rN)rYr[r\r])r@rGhiddenoutputresiduals r&rLTimesFmResidualBlock.forwardosI!!!$(""6*&&q)  r%)r[rWrVrYrXr\r]) rrrrr r7rLr$rPrQs@r&rSrSas! A!!r%rSRMSNormcx^\rSrSrS S\SS4U4SjjjrS\RS\R4SjrSr S r U=r $) TimesFmRMSNormwr5returnNc>[TU]5 [R"[R "U55UlX lg)z- TimesFmRMSNorm is equivalent to T5LayerNorm N)r6r7r: Parameterr!onesweightvariance_epsilon)r@r8r5rAs r&r7TimesFmRMSNorm.__init__ys/ ll5::k#:; #r% hidden_statescURnUR[R5nUR S5R SSS9nU[R "X0R-5-nURURU5-$)NT)keepdim) dtypetor!float32powmeanrsqrtrmrl)r@ro input_dtypevariances r&rLTimesFmRMSNorm.forwardsw#)) %((7  $$Q',,R,>% H?T?T4T(UU {{]--k:::r%c^[URR5SUR3$)Nz, eps=)tuplerlshaperm)r@s r& extra_reprTimesFmRMSNorm.extra_reprs* ))*+6$2G2G1HIIr%)rmrl)r3) rrrrr-r7r!r"rLrr$rPrQs@r&rfrfwsB$$$$$;U\\;ell;JJr%rfc>^\rSrSrSrS\4U4SjjrSSjrSrU=r $)TimesFmPositionalEmbeddingz6Generates position embedding for a given 1-d sequence.r1c >[TU]5 URnURnX#sUlUlURUlUR S-n[ R"[U5[U5- 5[US- S5- nURSU[R"[R"U[RS9U*-5-5 g)Nrqrinv_timescalesrt)r6r7 min_timescale max_timescaler8embedding_dimsmathlogr-maxregister_bufferr!exparangerv)r@r1rrnum_timescaleslog_timescale_incrementrAs r&r7#TimesFmPositionalEmbedding.__init__s ,, ,, 1>.D.$00,,1"&((5+?% BV+V"WZ]^lop^prsZt"t   EIIell>&W[rZr&rs s r%cTUcUc [S5eUcH[R"U[RURR S9R S5nO(URS:wa[SUR35eUR"/URQSP76URRSSS5-n[R"[R"U5[R"U5/SS9n[R"USSSURS-45nU$) aGenerates a Tensor of sinusoids with different frequencies. Args: seq_length: an optional Python int defining the output sequence length. if the `position` argument is specified. position: [B, seq_length], optional position for each token in the sequence, only required when the sequence is packed. Returns: [B, seqlen, D] if `position` is specified, else [1, seqlen, D] z.Either position or seq_length must be providedrtdevicerrqz*position must be 2-dimensional, got shape rrrdim) ValueErrorr!rrvrr unsqueezendimrviewcatsincosrEpadr)r@ seq_lengthposition scaled_timesignals r&rL"TimesFmPositionalEmbedding.forwards   2MN N  ||JemmDL_L_LfLfgqqrstH ]]a I(..IYZ[ [mm7X^^7Q7$:M:M:R:RSTVWY[:\\ EIIk2EIIk4JKQRSv1a)<)[TU]5 XlSUlURUlX lUR UlURUlURUl URUR-Ul URUR-Ul [R"[R"UR455Ul[R""URURUR-5Ul[R""URURUR-5Ul[R""URURUR-5Ul[R""URUR-UR5Ulg)NT)r6r7r1 is_causalattention_dropoutrnum_attention_heads num_headsr8head_dimq_sizekv_sizer:rjr!emptyrr;q_projk_projv_projo_projr@r1rrAs r&r7TimesFmAttention.__init__s/  !'!9!9"33!-- nnt}}4 ~~ 5 ||EKK0@$AB ii 0 0$..4==2PQ ii 0 0$..4==2PQ ii 0 0$..4==2PQ ii >@P@PQ r%queryrhc[R"UR5RS[R "UR 5- 5nXSSSSS24-$)Ng^$3eG?)rEsoftplusrmulrsqrtr)r@rrs r& _scale_queryTimesFmAttention._scale_querysH 4<<(,,[499T]];S-STT4q0111r%Nrorrc URSSn/UQSPURP7nURU5RU5R SS5nUR U5nUR U5RU5R SS5nURU5RU5R SS5n[R"URR[5n U "UUUUU4UR(dSO URSS.UD6upU R"/UQSP76R!5n UR#U 5n X4$)NrrrrqrD)rr)rrrrrrrrr get_interfacer1_attn_implementationrrrreshaperr) r@rorr input_shape hidden_shaperrrattention_interfacerrs r&rLTimesFmAttention.forwardsQ $))#2. 88b8$--8 {{=166|DNNqRST ((6 [[/44\BLLQPQR {{=166|DNNqRST (?(M(M KK , ,.L) %8      % $}}C$2H2H %  % ! "));;;;FFH kk+. ((r%)rr1rr8rrrrrrrrrrrN)rrrrr rintr7r!r"rr r r~rLr$rPrQs@r&rrsvR}RR(2%,,25<<2/3)||) t+)-. ) u||U\\D00 1 ))r%rc^\rSrSrSrS\S\4U4SjjrS\RS\RS\RS \R4S jr S r U=r $) TimesFmDecoderLayeri zTransformer layer.r1rc>[TU]5 [XS9Ul[ U5Ul[ URURS9Ul g)N)r)r5) r6r7r self_attnr/mlprfr8 rms_norm_epsinput_layernormrs r&r7TimesFmDecoderLayer.__init__sA )&Ff%-f.@.@fFYFYZr%rorrHrhc vUnURU5nURUUS9upXQ-nURXS9nU$)N)ror)rH)rrr)r@rorrHrrb_s r&rLTimesFmDecoderLayer.forwardsW!,,]; >>')*  !0 B r%)rrr) rrrrr rrr7r!r"rLr$rPrQs@r&rr s\[}[[|| ,,   r%rc|^\rSrSr%\\S'SrS/rSrSr Sr \ \ S.r \R"5U4S j5rS rU=r$) TimesFmPreTrainedModeli,r1timesfmr past_values)timeT)ro attentionsc 8>[TU]U5 [U[5(a![R "UR 5 g[U[5(aURS-nURURpC[R"[U5[U5- 5[US- S5- n[R"UR U["R$"["R&"U["R(S9U*-5-5 gg)Nrqrr)r6 _init_weights isinstancerinitones_rrrrrrrr-rcopy_rr!rrrv)r@rrrrrrAs r&r$TimesFmPreTrainedModel._init_weights9s f% f. / / JJv~~ &  : ; ;#22a7N+1+?+?AUAU=&*hhu]/CeMFZ/Z&[^a"A_' # JJ%%))ELLu}}MQhPhhij  [TU]U5 Xl[SUR-UR UR S9Ul[R"URUR S9Ul [R"[UR5Vs/sHn[X5PM sn5UlURR"(a[%US9UlUR)5 gs snf)NrqrVrXrW)num_embeddings embedding_dim)r1)r6r7r1rS patch_lengthr8r9input_ff_layerr: Embedding freq_sizefreq_emb ModuleListrangenum_hidden_layersrlayersuse_positional_embeddingr position_emb post_initrs r&r7TimesFmModel.__init__Ns   26...**00   F4D4DTZTfTfg mmEJ6KcKcEd eEd  3Ed e   ;; / / :& ID   fsC8inputs patched_padsrhc URX5up4[R"X@RRS9nXSS2SS4- USS2SS4- n[R "[R "XRR- 5URR:[R"URRURURS9U5nXSU44$)zInput is of shape [B, N, P].minNr) _timesfm_masked_mean_stdr!clampr1 tolerancewhereabspad_valtensorrtr)r@rrmusigmarKs r&_forward_transformTimesFmModel._forward_transformas11&G  E{{'<'<=q$}--q$}1EE++ IIf{{222 3dkk6K6K K LL,,GMM'.. Y   U ##r%rpast_values_paddingfreqrc ZURSnURUSURR5nURUSURR5n[R "[R "US- 5URR:[R"SURURS9U5n[R "[R "X`RR- 5URR:[R"SURURS9U5nURXg5uphUSU- -n[R"Xg/SS9n URU 5n [R"USS9Sn URR (aYUR#U RS5n [R$"U /U RS-SS9n UR'X5n X- n UR)U5n X- n U nUR+U URSURURSS 9nUR,S URR.HnU"U4UU S .UD6nM [1UUSUSS 9$) ax past_values (`torch.FloatTensor` of shape `(batch_size, sequence_length)`): Past values of the time series that serves as input to the model. past_values_padding (`torch.LongTensor` of shape `(batch_size, sequence_length)`): The padding indicator of the time series. freq (`torch.LongTensor` of shape `(batch_size,)`): Frequency indices for the time series data. rrrrDrrrrT)rsequence_lengthrtrrN)rrH)last_hidden_staterr)rrr1rr!rrrrrtrrrrrrr r concat_timesfm_shift_padded_seqr_prepare_4d_attention_maskr r r)r@rr r!rbsizepatched_inputsrstats concat_inputs model_inputpatched_paddingpos_embf_embrorlayers r&rLTimesFmModel.forwardqss&!!!$$))%T[[5M5MN*//r4;;;S;ST  IIlS( )DKK,A,A A LLN$8$8AVAV W   {{ IIn{{':':: ;dkk>S>S S LLL$6$6|?R?R S  !% 7 7 U(3+=> >"@bI ))-8  ))Lb9!< ;; / /'' (9(9!(<=GllG9{/@/@/C#CKG44_NG  "K d# $ 88*)//2%% '' 9 [[!@4;;#@#@AE!-( MB+a(  r%rNr#rtrrcUR(a [R"U5RO[R"U5RnUb%UR UR SSSS5nX-nU(a\[R"[R"X4X#S9U-SS9nUR SSX5nUb[R"X5nU$UnU$)a Creates 4D attention mask and combines causal and padding masks if needed. Args: attention_mask: Optional tensor of shape (batch_size, seq_length) containing padding mask sequence_length: Length of the sequence dtype: Data type of the mask device: Device of the mask is_causal: Whether to apply causal masking Returns: 4D attention mask of shape (batch_size, 1, seq_length, seq_length) rrrrr)diagonal) is_floating_pointr!finforiinforrtriurkminimum)rr#rtrr min_value causal_masks r&r''TimesFmModel._prepare_4d_attention_masks,/4.E.EEKK&**5;;W\K]KaKa   %+001E1Ea1H!QPRSN+7N ** O=UZ]ffK&**1aRK)!&~!K"-r%paddingc(S[R4Sjn[R"SU- SS9nU"U5n[R"URS5nXUSS24nXUSS24nSU- n[R"USS9n [R "U SS 9n [R"Xh-SS9n X- n XkR S 5- U-n [R"U S-SS9U - n [R "U S S 9n [R"U 5nX4$) aCalculates mean and standard deviation of `inputs` across axis 1. It excludes values where `padding` is 1. Args: inputs: A PyTorch tensor of shape [b, n, p]. padding: A PyTorch tensor of shape [b, n, p] with values 0 or 1. Returns: A tuple containing the mean and standard deviation. We return the statistics of the first patch with more than three non-padded values. arrc [R"US:R[R5SS9nUS:R[R5R SS9n[R "US:HUR SS- U5$)Nrrrr)r!argmaxruint32sumrr)r>indicesrow_sums r&_get_patch_index?TimesFmModel._timesfm_masked_mean_std.._get_patch_indexskllC1H==#=1EGaxmmEKK0444;G;;w!|SYYq\A-=wG Gr%rrqrrNrDrrrr)r!r"rBrrrrr)rr<rEpad_sum patch_indicesbidxsr>rmasknum_valid_elements masked_sum masked_meanmasked_centered_arr masked_var masked_stds r&r%TimesFmModel._timesfm_masked_mean_stds  H%,, H ))AKQ/(1  V\\!_-M1,-]A-.3w#YYt3"[[);EYYszq1  5  #%:%:2%>>$FYY2A51=@RR [[5 ZZ + &&r%rJseqclURup#nUS:HnUR[R5R SS9nSXeR SS9)'[R "X1RS9RSSS5RUSU5nXvSS2SS4- U-nURSU5n U $)zShifts rows of seq based on the first 0 in each row of the mask. Args: mask: mask tensor of shape [B, N] seq: seq tensor of shape [B, N, P] Returns: The shifted sequence. rrrrr)rN) rrur!rAr@anyrrrexpandgather) rJrR batch_sizenum_seq feature_dimnew_maskrC idx_range shifted_idx shifted_seqs r&r&&TimesFmModel._timesfm_shift_padded_seqs,/99( [%)QY++ekk*11a18)+!$$%LL<AA!RKRRS]_acno !1dD=#99WD jjK0 r%)r1rrr r )T)rrrrrr7r!r"r~rrrr LongTensorr rrrL staticmethodrrtrboolr'rr&r$rPrQs@r&rrLs}&$ll$27,,$ u||U5<<#=>> ?$  F \\F #--F ll F +, F  F  F P  + t+++{{+ +  +   ++Z,',' ,'QVW\WcWcejeqeqWqQr,','\ 5<<ELLr%rcl^\rSrSrSrS\4U4SjjrSS\\RS\\ S-S\ S-S \ \RS 44S jjr S \RS \ \R\R4S \R4Sjr S\RS\RS \R4Sjr\\SS\\RS\\R\ -S-S\ S-S\RS-S\ S-S\S\S\\S \4Sjj55r\S\RS\ S \\R4Sj5rSrU=r$)TimesFmModelForPredictioni:z/TimesFM model for quantile and mean prediction.r1cB>[TU]U5 XlURUlUR Ul[U5Ul[URUR S[UR5--URS9UlUR5 g)Nrr)r6r7r1context_length context_lenhorizon_length horizon_lenrdecoderrSr8len quantilesr9horizon_ff_layerr)r@r1rAs r&r7"TimesFmModelForPrediction.__init__=s   !00!00#F+ !5))--S9I9I5J1JK00!  r%Nrr!rfrh.c 2Uc URn//pTUGH nURSn[R"XpR-UR UR S9nXs:aX7- n [R"[R"XR UR S9U/SS9n[R"[R"XR UR S9U/SS9nOXs:aXc*SnXUR-*SnURU5 URU5 GM [R"USS9[R"USS94n UbCU [R"US[U5[RS9RSS54-n U $)aPad/truncate input time series to `context_len` and build a padding mask. Args: inputs: A list of 1d Tensors. Each Tensor is the context time series of a single forecast task. freq: Optional list of frequencies (returned as a tensor when provided). context_len: Optional context length override (defaults to `self.context_len`). Returns: Tuple of (padded_inputs, padding_mask) and optionally a freq tensor. Nrrrrrrr)rfrr!zerosrhrtrrrkappendstackrrjrAr) r@rr!rfinput_ts input_paddingts input_lenr< num_front_padresults r& _preprocess%TimesFmModelForPrediction._preprocessPsi  **K"$b-B Ikk).>.>">bhhWYW`W`aG& + 7 YY MRTR[R[ \^`aghi))UZZ XXV]VdVd%egn$ouvw( &!$2B2B$B"C"EF OOB    )++hA. Mq0QR  u||D3v;,?u{{S[[\^`abddF r% model_outputr*c URU5nURupEnURXEURR[ URR 5S-5nUupxX8SS2SSS4-USS2SSS4-$)z*Postprocess output of stacked transformer.rN)rlrrr1rgrjrk) r@rzr* output_tsbnrrrs r&_postprocess_output-TimesFmModelForPrediction._postprocess_outputus )),7 //aNN1)C)CSI^I^E_bcEcd  D$!4551dD$;N8OOOr% predictionstargetsc,/n[URR5HLupEX!SU4- n[R"US- U-XV-5nUR UR 55 MN [R"U5R 5$)N.r) enumerater1rkr!rrprxrq)r@rrlossesiqerrorsr,s r&_quantile_loss(TimesFmModelForPrediction._quantile_lossszdkk334DA3622F99a!ev-qz:D MM$))+ &5{{6"''))r%r window_size future_valuesforecast_context_lenreturn_forecast_on_contexttruncate_negativerc : Uc URn OUn USRn UV s/sHoU *SPM n n [R"[R"U V s/sHn [R"U 5PM sn 55n Ub]/n/n[ U 5HCunn UR URX55 UcM+UR UU/S-5 ME Un UbUnUc$[RS5 S/[U 5-nURX5unnnURU 5nURU 5nURU 5nUnURSn/nURSURSUR-:wa8[SURSSURSSUR35eUR R"nURU-S- U-n[%U5GH4nUSS2SURS24nUSS2U *S24nUSS2U *S24nUR&"SUUUS .UD6nUR)UR*UR,UR.45nU(amUS:XagUSS2SS 2SUR R02SS24nUR3UR5S5S UR5S 55nUR7U5 USS2S SU2S4nUSS2S SU2SS24nUR7U5 [R8"UU/S S 9nGM7 U(aF[R8"USS 9SS2SUUR R0- UR-2SS24nO+[R8"USS 9SS2SUR2SS24nUSS2SS2S4nUb*USSS2S 4USSS2S 4-nUSSS2S 4USSS2S 4-nU S:a5U(a.[R:"US5n[R:"US5nSn Ub:[<R>"UU5n!URAUSS2SS2SS24U5n"U!U"-n [CWR*URDURFUUU S9$s sn fs sn f)a past_values (`torch.FloatTensor` of shape `(batch_size, sequence_length)`): Past values of the time series that serves as input to the model. freq (`torch.LongTensor` of shape `(batch_size,)`): Frequency indices for the time series data. window_size (`int`, *optional*): Window size of trend + residual decomposition. If None then we do not do decomposition. future_values (`torch.Tensor`, *optional*): Optional future time series values to be used for loss computation. forecast_context_len (`int`, *optional*): Optional max context length. return_forecast_on_context (`bool`, *optional*): True to return the forecast on the context when available, i.e. after the first input patch. truncate_negative (`bool`, *optional*): Truncate to only non-negative values if any of the contexts have non-negative values, otherwise do nothing. Example: ```python >>> from transformers import TimesFmModelForPrediction >>> model = TimesFmModelForPrediction.from_pretrained("google/timesfm-2.0-500m-pytorch") >>> forecast_input = [torch.linspace(0, 20, 100).sin(), torch.linspace(0, 20, 200).sin(), torch.linspace(0, 20, 400).sin()] >>> frequency_input = torch.tensor([0, 1, 2], dtype=torch.long) >>> # Generate >>> with torch.no_grad(): >>> outputs = model(past_values=forecast_input, freq=frequency_input, return_dict=True) >>> point_forecast_conv = outputs.mean_predictions >>> quantile_forecast_conv = outputs.full_predictions ``` Nrrqz6No frequency provided via `freq`. Default to high (0).rz=Length of paddings must match length of input + horizon_len: z != z + )rr r!rrr)axis.r)r$rror*r+r,r)$rfrr!rrqrextend_timesfm_moving_averageloggerinforjrxrurrhrr1rgr rirr$rrrrsizerp concatenatemaximumrEmse_lossrr(rro)#r@rr!rrrrrr fcontext_lenrrtrinp_min new_inputs new_freqsrrrrsinp_freq final_outrf full_outputsoutput_patch_lennum_decode_patches step_indexcurrent_paddingdecoder_output fprop_outputs new_full_tsnew_ts mean_outputsr,r quantile_losss# r&rL!TimesFmModelForPrediction.forwards^ '++L/LQ&&/:;{l]^${;))EKK(H22(HIJ  "JI"6*2!!$">">r"OP#$$d1gY]3+ F  < KKP Q3V$D,0,<,M,0LL-$$1- -N !4400##^%9%9:M *jAo+Ass4Ndkk6N6N4NPQ,QR )11+2B2B12Er;K[K[\]K^_ ##K0"1b*;+;*;Q#>?F'2/@0@/@!(CDK    ,))9f*=BGI/42 & ,,\BPkDKK$<$<H // Q12X0F VMm+D),>>%00(66))   c<(Hs R R r>c.[R"XS- S4SS5n[R"XRUR S9U- n[R "URSSS5URSSS55R5nX@U- /$)zCCalculates the moving average using PyTorch's convolution function.rrconstantrrr) rErr!rkrtrconv1drsqueeze)r>r arr_paddedkernel smoothed_arrs r&r1TimesFmModelForPrediction._timesfm_moving_averages}UU3q! 4j!D KyyL{Zxx 1b 96;;q!R;PQYY[ L011r%)r1rfrirlrhr)NNNNFF)rrrrr rr7rr!r"rr~rxrrrrrar rr(rLr`listrr$rPrQs@r&rcrc:s9}(lp#u||,#4rs* .! &7B/F&RR750   H %  &O &  &  - -  -,!299!,Y'JRYYJ(J(++j% II%,,% %,, % LL4' %  %S[%'(%,9)ryy9)x"))@_>j)jjZk2 6k2\ Rr%