ZjPwSrSSKrSSKJrJr SSKJr SSKrSSKJ r SSK J s J r SSK Jr SSKJr SSKJr SS KJrJr SS KJr SS KJrJrJrJr SS KJr SS K J!r! SSK"J#r# SSK$J%r% SSK&J'r' \RP"\)5r*\\"SS\555r+\\"SS\555r,"SS\ RZ5r."SS\ RZ5r/"SS\#5r0"SS\ RZ5r1"SS \ RZ5r2"S!S"\ RZ5r3\"S#S$\55r4\"S%S&\455r5"S'S(\45r6/S)Qr7g)*zPyTorch TimesFM model.N)CallableSequence) dataclass)initialization)FlashAttentionKwargs)BaseModelOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tuplelogging)merge_with_config_defaults)capture_outputs) LlamaRMSNorm)simple_eager_attention_forward) TimesFmConfigcj\rSrSr%SrSr\RS-\S'Sr \RS-\S'Sr g) TimesFmOutput(z loc (`torch.Tensor` of shape `(batch_size, )`): The mean of the time series inputs. scale (`torch.Tensor` of shape `(batch_size,)`): The scale of the time series inputs. Nlocscale) __name__ __module__ __qualname____firstlineno____doc__rtorchTensor__annotations__r__static_attributes__r|/root/GenerationalWealth/GenerationalWealth/venv/lib/python3.13/site-packages/transformers/models/timesfm/modular_timesfm.pyrr(s/ $C #!%E5<<$ %r'rc\rSrSr%SrSr\RS-\S'Sr \RS-\S'Sr \R\ -S-\S'Sr g)TimesFmOutputForPrediction6a mean_predictions (`torch.Tensor` of shape `(batch_size, sequence_length)`): The mean predictions of the time series. full_predictions (`torch.Tensor` of shape `(batch_size, sequence_length)`): The full predictions of the time series including the mean and the quantiles. loss (`torch.Tensor` of shape `(1,)`, *optional*, returned when `future_values` is provided): The loss of the TimesFM model. Nmean_predictionsfull_predictionslossr) rrr r!r"r,r#r$r%r-r.floatr&rr'r(r*r*6sI-1ellT)0,0ellT)0(,D%,,  %,r'r*c>^\rSrSrSrS\4U4SjjrSSjrSrU=r $) TimesFmMLPGzPax MLP in pytorch.configc>[TU]5 URnURn[R "X#5Ul[R "X25Ul[R"USS9Ul g)Ngư>)normalized_shapeeps) super__init__ hidden_sizeintermediate_sizennLinear gate_proj down_proj LayerNorm layer_norm)selfr3r9r: __class__s r(r8TimesFmMLP.__init__JsX (( "44;B#4B,, Nr'cURU5nURU5n[R"U5nUR U5nUbUSUSS2SS2S4- -nXQ-$)N?)r@r=Frelur>)rAxpaddingsgate_inpgateoutputss r(forwardTimesFmMLP.forwardSsa??1%~~h'vvd|..&  x1d ';!;r=r@N rrr r!r"rr8rMr& __classcell__rBs@r(r1r1GsO}Or'r1c2^\rSrSrSrU4SjrSrSrU=r$)TimesFmResidualBlock]zTimesFM residual block.c>[TU]5 XlX lX0l[ R "X5Ul[ R"5Ul [ R "X#5Ul [ R "X5Ul grO) r7r8 input_dims hidden_dims output_dimsr;r< input_layerSiLU activation output_layerresidual_layer)rArWrXrYrBs r(r8TimesFmResidualBlock.__init__`s_ $&&99Z='')IIk? ii @r'cURU5nURU5nURU5nURU5nX4-$rO)rZr\r]r^)rArHhiddenoutputresiduals r(rMTimesFmResidualBlock.forwardksI!!!$(""6*&&q)  r')r\rXrWrZrYr]r^) rrr r!r"r8rMr&rQrRs@r(rTrT]s! A!!r'rTc\rSrSrSrg)TimesFmRMSNormsrN)rrr r!r&rr'r(rfrfssr'rfc>^\rSrSrSrS\4U4SjjrSSjrSrU=r $)TimesFmPositionalEmbeddingwz6Generates position embedding for a given 1-d sequence.r3c >[TU]5 URnURnX#sUlUlURUlUR S-n[ R"[U5[U5- 5[US- S5- nURSU[R"[R"U[RS9U*-5-5 g)Nrrinv_timescalesdtype)r7r8 min_timescale max_timescaler9embedding_dimsmathlogr/maxregister_bufferr#exparangefloat32)rAr3rorpnum_timescaleslog_timescale_incrementrBs r(r8#TimesFmPositionalEmbedding.__init__zs ,, ,, 1>.D.$00,,1"&((5+?% BV+V"WZ]^lop^prsZt"t   EIIell>&W[rZr&rs s r'cTUcUc [S5eUcH[R"U[RURR S9R S5nO(URS:wa[SUR35eUR"/URQSP76URRSSS5-n[R"[R"U5[R"U5/SS9n[R"USSSURS-45nU$) aGenerates a Tensor of sinusoids with different frequencies. Args: seq_length: an optional Python int defining the output sequence length. if the `position` argument is specified. position: [B, seq_length], optional position for each token in the sequence, only required when the sequence is packed. Returns: [B, seqlen, D] if `position` is specified, else [1, seqlen, D] z.Either position or seq_length must be providedrndevicerrz*position must be 2-dimensional, got shape rdim) ValueErrorr#rwrxrlr~ unsqueezendimshapeviewcatsincosrFpadrq)rA seq_lengthposition scaled_timesignals r(rM"TimesFmPositionalEmbedding.forwards   2MN N  ||JemmDL_L_LfLfgqqrstH ]]a I(..IYZ[ [mm7X^^7Q7$:M:M:R:RSTVWY[:\\ EIIk2EIIk4JKQRSv1a)<)[TU]5 XlSUlURUlX lUR UlURUlURUl URUR-Ul URUR-Ul [R"[R"UR455Ul[R""URURUR-5Ul[R""URURUR-5Ul[R""URURUR-5Ul[R""URUR-UR5Ulg)NT)r7r8r3 is_causalattention_dropoutrnum_attention_heads num_headsr9head_dimq_sizekv_sizer; Parameterr#emptyscalingr<q_projk_projv_projo_projrAr3rrBs r(r8TimesFmAttention.__init__s/  !'!9!9"33!-- nnt}}4 ~~ 5 ||EKK0@$AB ii 0 0$..4==2PQ ii 0 0$..4==2PQ ii 0 0$..4==2PQ ii >@P@PQ r'queryreturnc[R"UR5RS[R "UR 5- 5nXSSSSS24-$)Ng^$3eG?)rFsoftplusrmulrrsqrtr)rArrs r( _scale_queryTimesFmAttention._scale_querysH 4<<(,,[499T]];S-STT4q0111r'N hidden_statesattention_maskkwargsc URSSn/UQSPURP7nURU5RU5R SS5nUR U5nUR U5RU5R SS5nURU5RU5R SS5n[R"URR[5n U "UUUUU4UR(dSO URSS.UD6upU R"/UQSP76R!5n UR#U 5n X4$)NrrrrE)dropoutr)rrrr transposerrrr get_interfacer3_attn_implementationrtrainingrreshape contiguousr) rArrr input_shape hidden_shape query_states key_states value_statesattention_interface attn_output attn_weightss r(rMTimesFmAttention.forwardsQ $))#2. 88b8$--8 {{=166|DNNqRST ((6 [[/44\BLLQPQR {{=166|DNNqRST (?(M(M KK , ,.L) %8      % $}}C$2H2H %  % ! "));;;;FFH kk+. ((r')rr3rr9rrrrrrrrrrrO)rrr r!r"rintr8r#r$rr rtuplerMr&rQrRs@r(rrsvR}RR(2%,,25<<2/3)||) t+)-. ) u||U\\D00 1 ))r'rc^\rSrSrSrS\S\4U4SjjrS\RS\RS\RS \R4S jr S r U=r $) TimesFmDecoderLayerzTransformer layer.r3rc>[TU]5 [XS9Ul[ U5Ul[ URURS9Ul g)N)r)r6) r7r8r self_attnr1mlprfr9 rms_norm_epsinput_layernormrs r(r8TimesFmDecoderLayer.__init__sA )&Ff%-f.@.@fFYFYZr'rrrIrc vUnURU5nURUUS9upXQ-nURXS9nU$)N)rr)rI)rrr)rArrrIrrc_s r(rMTimesFmDecoderLayer.forwardsW!,,]; >>')*  !0 B r')rrr) rrr r!r"rrr8r#r$rMr&rQrRs@r(rrs\[}[[|| ,,   r'rc|^\rSrSr%\\S'SrS/rSrSr Sr \ \ S.r \R"5U4S j5rS rU=r$) TimesFmPreTrainedModelir3timesfmr past_values)timeT)r attentionsc 8>[TU]U5 [U[5(a![R "UR 5 g[U[5(aURS-nURURpC[R"[U5[U5- 5[US- S5- n[R"UR U["R$"["R&"U["R(S9U*-5-5 gg)Nrrrm)r7 _init_weights isinstancerinitones_rrirqrprorrrsr/rtcopy_rlr#rvrwrx)rAmoduleryrprorzrBs r(r$TimesFmPreTrainedModel._init_weightss f% f. / / JJv~~ &  : ; ;#22a7N+1+?+?AUAU=&*hhu]/CeMFZ/Z&[^a"A_' # JJ%%))ELLu}}MQhPhhij  [TU]U5 Xl[SUR-UR UR S9Ul[R"URUR S9Ul [R"[UR5Vs/sHn[X5PM sn5UlURR"(a[%US9UlUR)5 gs snf)NrrWrYrX)num_embeddings embedding_dim)r3)r7r8r3rT patch_lengthr9r:input_ff_layerr; Embedding freq_sizefreq_emb ModuleListrangenum_hidden_layersrlayersuse_positional_embeddingri position_emb post_initrs r(r8TimesFmModel.__init__#s   26...**00   F4D4DTZTfTfg mmEJ6KcKcEd eEd  3Ed e   ;; / / :& ID   fsC8inputs patched_padsrc URX5up4[R"X@RRS9nXSS2SS4- USS2SS4- n[R "[R "XRR- 5URR:[R"URRURURS9U5nXSU44$)zInput is of shape [B, N, P].minNr}) _timesfm_masked_mean_stdr#clampr3 tolerancewhereabspad_valtensorrnr~)rArrmusigmarLs r(_forward_transformTimesFmModel._forward_transform6s11&G  E{{'<'<=q$}--q$}1EE++ IIf{{222 3dkk6K6K K LL,,GMM'.. Y   U ##r'rpast_values_paddingfreqrc ZURSnURUSURR5nURUSURR5n[R "[R "US- 5URR:[R"SURURS9U5n[R "[R "X`RR- 5URR:[R"SURURS9U5nURXg5uphUSU- -n[R"Xg/SS9n URU 5n [R"USS9Sn URR (aYUR#U RS5n [R$"U /U RS-SS9n UR'X5n X- n UR)U5n X- n U nUR+U URSURURSS 9nUR,S URR.HnU"U4UU S .UD6nM [1UUSUSS 9$) ax past_values (`torch.FloatTensor` of shape `(batch_size, sequence_length)`): Past values of the time series that serves as input to the model. past_values_padding (`torch.LongTensor` of shape `(batch_size, sequence_length)`): The padding indicator of the time series. freq (`torch.LongTensor` of shape `(batch_size,)`): Frequency indices for the time series data. rrrErr}rrT)rsequence_lengthrnr~rN)rrI)last_hidden_staterr)rrr3rr#rrrrrnr~rrrrrrrconcat_timesfm_shift_padded_seqr_prepare_4d_attention_maskrrr)rArr r rbsizepatched_inputsrstats concat_inputs model_inputpatched_paddingpos_embf_embrrlayers r(rMTimesFmModel.forwardFss&!!!$$))%T[[5M5MN*//r4;;;S;ST  IIlS( )DKK,A,A A LLN$8$8AVAV W   {{ IIn{{':':: ;dkk>S>S S LLL$6$6|?R?R S  !% 7 7 U(3+=> >"@bI ))-8  ))Lb9!< ;; / /'' (9(9!(<=GllG9{/@/@/C#CKG44_NG  "K d# $ 88*)//2%% '' 9 [[!@4;;#@#@AE!-( MB+a(  r'rNr rnr~rcUR(a [R"U5RO[R"U5RnUb%UR UR SSSS5nX-nU(a\[R"[R"X4X#S9U-SS9nUR SSX5nUb[R"X5nU$UnU$)a Creates 4D attention mask and combines causal and padding masks if needed. Args: attention_mask: Optional tensor of shape (batch_size, seq_length) containing padding mask sequence_length: Length of the sequence dtype: Data type of the mask device: Device of the mask is_causal: Whether to apply causal masking Returns: 4D attention mask of shape (batch_size, 1, seq_length, seq_length) rrrr})diagonal) is_floating_pointr#finforiinforrtriuonesminimum)rr rnr~r min_value causal_masks r(r'TimesFmModel._prepare_4d_attention_masks,/4.E.EEKK&**5;;W\K]KaKa   %+001E1Ea1H!QPRSN+7N ** O=UZ]ffK&**1aRK)!&~!K"-r'paddingc(S[R4Sjn[R"SU- SS9nU"U5n[R"URS5nXUSS24nXUSS24nSU- n[R"USS9n [R "U SS 9n [R"Xh-SS9n X- n XkR S 5- U-n [R"U S-SS9U - n [R "U S S 9n [R"U 5nX4$) aCalculates mean and standard deviation of `inputs` across axis 1. It excludes values where `padding` is 1. Args: inputs: A PyTorch tensor of shape [b, n, p]. padding: A PyTorch tensor of shape [b, n, p] with values 0 or 1. Returns: A tuple containing the mean and standard deviation. We return the statistics of the first patch with more than three non-padded values. arrc [R"US:R[R5SS9nUS:R[R5R SS9n[R "US:HUR SS- U5$)Nrrrr)r#argmaxtoint32sumrr)r(indicesrow_sums r(_get_patch_index?TimesFmModel._timesfm_masked_mean_std.._get_patch_indexskllC1H==#=1EGaxmmEKK0444;G;;w!|SYYq\A-=wG Gr'rrrrNrErrr)r#r$r-rwrrrr)rr&r0pad_sum patch_indicesbidxsr(rmasknum_valid_elements masked_sum masked_meanmasked_centered_arr masked_var masked_stds r(r%TimesFmModel._timesfm_masked_mean_stds  H%,, H ))AKQ/(1  V\\!_-M1,-]A-.3w#YYt3"[[);EYYszq1  5  #%:%:2%>>$FYY2A51=@RR [[5 ZZ + &&r'r5seqclURup#nUS:HnUR[R5R SS9nSXeR SS9)'[R "X1RS9RSSS5RUSU5nXvSS2SS4- U-nURSU5n U $)zShifts rows of seq based on the first 0 in each row of the mask. Args: mask: mask tensor of shape [B, N] seq: seq tensor of shape [B, N, P] Returns: The shifted sequence. rrrr)r~N) rr+r#r,r*anyrwr~rexpandgather) r5r= batch_sizenum_seq feature_dimnew_maskr. idx_range shifted_idx shifted_seqs r(r&TimesFmModel._timesfm_shift_padded_seqs,/99( [%)QY++ekk*11a18)+!$$%LL<AA!RKRRS]_acno !1dD=#99WD jjK0 r')r3rrrr)T)rrr r!rr8r#r$rrrrr LongTensorr r rrM staticmethodrrnr~boolrrrr&rQrRs@r(rr!s}&$ll$27,,$ u||U5<<#=>> ?$  F \\F #--F ll F +, F  F  F P  + t+++{{+ +  +   ++Z,',' ,'QVW\WcWcejeqeqWqQr,','\ 5<<ELLr'rcl^\rSrSrSrS\4U4SjjrSS\\RS\\ S-S\ S-S \ \RS 44S jjr S \RS \ \R\R4S \R4Sjr S\RS\RS \R4Sjr\\SS\\RS\\R\ -S-S\ S-S\RS-S\ S-S\S\S\\S \4Sjj55r\S\RS\ S \\R4Sj5rSrU=r$)TimesFmModelForPredictioniz/TimesFM model for quantile and mean prediction.r3cB>[TU]U5 XlURUlUR Ul[U5Ul[URUR S[UR5--URS9UlUR5 g)Nrr)r7r8r3context_length context_lenhorizon_length horizon_lenrdecoderrTr9len quantilesr:horizon_ff_layerr)rAr3rBs r(r8"TimesFmModelForPrediction.__init__s   !00!00#F+ !5))--S9I9I5J1JK00!  r'Nrr rQr.c 2Uc URn//pTUGH nURSn[R"XpR-UR UR S9nXs:aX7- n [R"[R"XR UR S9U/SS9n[R"[R"XR UR S9U/SS9nOXs:aXc*SnXUR-*SnURU5 URU5 GM [R"USS9[R"USS94n UbCU [R"US[U5[RS9RSS54-n U $)aPad/truncate input time series to `context_len` and build a padding mask. Args: inputs: A list of 1d Tensors. Each Tensor is the context time series of a single forecast task. freq: Optional list of frequencies (returned as a tensor when provided). context_len: Optional context length override (defaults to `self.context_len`). Returns: Tuple of (padded_inputs, padding_mask) and optionally a freq tensor. Nrr}rrmrr)rQrr#zerosrSrnr~rr!appendstackrrUr,r) rArr rQinput_ts input_paddingts input_lenr& num_front_padresults r( _preprocess%TimesFmModelForPrediction._preprocess%si  **K"$b-B Ikk).>.>">bhhWYW`W`aG& + 7 YY MRTR[R[ \^`aghi))UZZ XXV]VdVd%egn$ouvw( &!$2B2B$B"C"EF OOB    )++hA. Mq0QR  u||D3v;,?u{{S[[\^`abddF r' model_outputrc URU5nURupEnURXEURR[ URR 5S-5nUupxX8SS2SSS4-USS2SSS4-$)z*Postprocess output of stacked transformer.rN)rWrrr3rRrUrV) rArer output_tsbnrrrs r(_postprocess_output-TimesFmModelForPrediction._postprocess_outputJs )),7 //aNN1)C)CSI^I^E_bcEcd  D$!4551dD$;N8OOOr' predictionstargetsc,/n[URR5HLupEX!SU4- n[R"US- U-XV-5nUR UR 55 MN [R"U5R 5$)N.r) enumerater3rVr#rtr[meanr\)rArlrmlossesiqerrorsr.s r(_quantile_loss(TimesFmModelForPrediction._quantile_lossYszdkk334DA3622F99a!ev-qz:D MM$))+ &5{{6"''))r'r window_size future_valuesforecast_context_lenreturn_forecast_on_contexttruncate_negativerc : Uc URn OUn USRn UV s/sHoU *SPM n n [R"[R"U V s/sHn [R"U 5PM sn 55n Ub]/n/n[ U 5HCunn UR URX55 UcM+UR UU/S-5 ME Un UbUnUc$[RS5 S/[U 5-nURX5unnnURU 5nURU 5nURU 5nUnURSn/nURSURSUR-:wa8[SURSSURSSUR35eUR R"nURU-S- U-n[%U5GH4nUSS2SURS24nUSS2U *S24nUSS2U *S24nUR&"SUUUS .UD6nUR)UR*UR,UR.45nU(amUS:XagUSS2SS 2SUR R02SS24nUR3UR5S5S UR5S 55nUR7U5 USS2S SU2S4nUSS2S SU2SS24nUR7U5 [R8"UU/S S 9nGM7 U(aF[R8"USS 9SS2SUUR R0- UR-2SS24nO+[R8"USS 9SS2SUR2SS24nUSS2SS2S4nUb*USSS2S 4USSS2S 4-nUSSS2S 4USSS2S 4-nU S:a5U(a.[R:"US5n[R:"US5nSn Ub:[<R>"UU5n!URAUSS2SS2SS24U5n"U!U"-n [CWR*URDURFUUU S9$s sn fs sn f)a past_values (`torch.FloatTensor` of shape `(batch_size, sequence_length)`): Past values of the time series that serves as input to the model. freq (`torch.LongTensor` of shape `(batch_size,)`): Frequency indices for the time series data. window_size (`int`, *optional*): Window size of trend + residual decomposition. If None then we do not do decomposition. future_values (`torch.Tensor`, *optional*): Optional future time series values to be used for loss computation. forecast_context_len (`int`, *optional*): Optional max context length. return_forecast_on_context (`bool`, *optional*): True to return the forecast on the context when available, i.e. after the first input patch. truncate_negative (`bool`, *optional*): Truncate to only non-negative values if any of the contexts have non-negative values, otherwise do nothing. Example: ```python >>> from transformers import TimesFmModelForPrediction >>> model = TimesFmModelForPrediction.from_pretrained("google/timesfm-2.0-500m-pytorch") >>> forecast_input = [torch.linspace(0, 20, 100).sin(), torch.linspace(0, 20, 200).sin(), torch.linspace(0, 20, 400).sin()] >>> frequency_input = torch.tensor([0, 1, 2], dtype=torch.long) >>> # Generate >>> with torch.no_grad(): >>> outputs = model(past_values=forecast_input, freq=frequency_input, return_dict=True) >>> point_forecast_conv = outputs.mean_predictions >>> quantile_forecast_conv = outputs.full_predictions ``` Nrrz6No frequency provided via `freq`. Default to high (0).rz=Length of paddings must match length of input + horizon_len: z != z + )rr r rr)axis.r)r rrr,r-r.r)$rQr~r#rr\roextend_timesfm_moving_averageloggerinforUrcr+rrSrr3rRrrTrjr rrrrsizer[ concatenatemaximumrFmse_lossrur*rr)#rArr rwrxryrzr{r fcontext_lenr~r_rinp_min new_inputs new_freqsrrr]r^inp_freq final_outrQ full_outputsoutput_patch_lennum_decode_patches step_indexcurrent_paddingdecoder_output fprop_outputs new_full_tsnew_ts mean_outputsr.r quantile_losss# r(rM!TimesFmModelForPrediction.forwardas^ '++L/LQ&&/:;{l]^${;))EKK(H22(HIJ  "JI"6*2!!$">">r"OP#$$d1gY]3+ F  < KKP Q3V$D,0,<,M,0LL-$$1- -N !4400##^%9%9:M *jAo+Ass4Ndkk6N6N4NPQ,QR )11+2B2B12Er;K[K[\]K^_ ##K0"1b*;+;*;Q#>?F'2/@0@/@!(CDK    ,))9f*=BGI/42 & ,,\BPkDKK$<$<H // Q12X0F VMm+D),>>%00(66))   c<(Hs R R r(c.[R"XS- S4SS5n[R"XRUR S9U- n[R "URSSS5URSSS55R5nX@U- /$)zCCalculates the moving average using PyTorch's convolution function.rrconstantr}r) rFrr#r!rnr~conv1drsqueeze)r(rw arr_paddedkernel smoothed_arrs r(r1TimesFmModelForPrediction._timesfm_moving_averages}UU3q! 4j!D KyyL{Zxx 1b 96;;q!R;PQYY[ L011r')r3rQrTrWrSr)NNNNFF)rrr r!r"rr8rr#r$rrrcrjrurrrLr r r*rMrKlistrr&rQrRs@r(rNrNs9}(lp#u||,#4rsM .! &B/F&RR75/U0   H %  &O &  &  - -  -,!299!, \ ++\9)ryy9)x"))@_>j)jjZk2 6k2\ Rr'