Rj\1#SrSSKrSSKJr SSKJrJr SSKr\R"\ 5r SS/r S\ \ S-S\ \ 4S jr\"S S 9S \ S\4S j5r"SS\5r\R$R'S0S9S0S\R(S\R(S\R(S\R(S\R(S\ S\ S\S\S-S\ \ S-S\\R(\R(\R(44Sjj5r\R0S0S\R(S\R(S\R(S\R(S\R(S\ S\ S\S\S-S\ \ S-S\\R(\R(\R(44Sjj5rSSSS.S\R(S\R(S\R(S\R(S\R(S\ S\ S\S-S\S-S\\ \ 4S\R(\\R(\R(4-4SjjrS \S!\\S"4S#\SS4S$jr\R$R'S%0S9S1S&\R(S\R(S\R(S\R(S'\R(S(\R(S\R(S\R(S\ S\ S\S)\R(S\S-S\ \ S-S\\R(\R(\R(44S*jj5r\R0S1S&\R(S\R(S\R(S\R(S'\R(S(\R(S\R(S\R(S\ S\ S\S)\R(S\S-S\ \ S-S\\R(\R(\R(44S+jj5rS \S&\R(S,\R(S-\R(S\\R(S-S"44 S.jr\R?\\S/9 g)2z Variable-length attention implementation using Flash Attention. This module provides a high-level Python interface for variable-length attention that calls into the optimized Flash Attention kernels. N) lru_cache)Any NamedTuple varlen_attn AuxRequest window_sizereturnc`UcSS/n[U5S:wa[S[U535eU$)Nz$window_size must have length 2, got )len ValueError)rs j/root/GenerationalWealth/GenerationalWealth/venv/lib/python3.13/site-packages/torch/nn/attention/varlen.py_normalize_window_sizers=2h  ;1?K@P?QRSS )maxsize device_indexcg)z;Cache device capability check to avoid repeated CUDA calls.F)rs r_should_use_cudnnrs rc(\rSrSr%SrSr\\S'Srg)r#z Request which auxiliary outputs to compute from varlen_attn. Each field is a boolean indicating whether that auxiliary output should be computed. FlserN) __name__ __module__ __qualname____firstlineno____doc__rbool__annotations____static_attributes__rrrrr#s Crztorch_attn::_varlen_attn) mutates_argsquerykeyvaluecu_seq_qcu_seq_kmax_qmax_k is_causalscalec |[U 5n UR=(a [URR5n U (at[ R S5 U SS:wd U SS:wa [S5e[RRRUUUSUUUUSSUS US 9 n U SU SU S pn OS[ R S 5 [RRRUUUUUUUSUS UU SU SS 9 upn n[R"S[RURS9nXU4$)z Private custom op for variable-length attention. This is the internal implementation. Users should use the public varlen_attn function instead. #Using cuDNN backend for varlen_attnrr TcuDNN backend does not support window attention. Please use Flash Attention backend.NTFr,-Using Flash Attention backend for varlen_attn)return_debug_maskr,window_size_leftwindow_size_rightr dtypedevice)ris_cudarr;indexloginfo RuntimeErrortorchopsaten_cudnn_attention_forward_flash_attention_forwardzerosuint64)r$r%r&r'r(r)r*r+r,r use_cudnnresultoutput softmax_lse rng_state_ rng_state_s r _varlen_attnrO-sS$)5K G"3ELL4F4F"GI 67 q>R ;q>R#7f 88            9  *0F1IvayYY @A/4yy~~/V/V         #(^)!n0W0 ,Y1  ELLJ  **rc [U 5n [R"U5n URS5n URS5n [RR (aEURS5S- n [R "XU4[RURS9nO/[R "X4[RURS9n[R "S[RURS9nXU4$)z Fake implementation for meta tensor computation and tracing. Based on the 3D varlen path from meta__flash_attention_forward: - query shape: (total, num_heads, head_dim) - logsumexp shape: (num_heads, total_q) rr/r9r8) rrA empty_likesizeversionhipemptyfloatr;rG)r$r%r&r'r(r)r*r+r,rrJtotal_q num_heads batch_size logsumexprLs r_varlen_attn_faker[ss()5K  e $FjjmG 1 I }}]]1%) KK E *%++ell KK   ELL  D U\\JI i ''r)r r ) return_auxr,rr\cU S:Hn [RRRUUUUUUUU U[ U 55 upn UbUR (aX4$U $)a Compute variable-length attention using Flash Attention. This function is similar to scaled_dot_product_attention but optimized for variable-length sequences using cumulative sequence position tensors. Args: query (Tensor): Query tensor; shape :math:`(T_q, H, D)` key (Tensor): Key tensor; shape :math:`(T_k, H, D)` value (Tensor): Value tensor; shape :math:`(T_k, H, D)` cu_seq_q (Tensor): Cumulative sequence positions for queries; shape :math:`(N+1,)` cu_seq_k (Tensor): Cumulative sequence positions for keys/values; shape :math:`(N+1,)` max_q (int): Maximum query sequence length in the batch. max_k (int): Maximum key/value sequence length in the batch. return_aux (Optional[AuxRequest]): If not None and ``return_aux.lse`` is True, also returns the logsumexp tensor. scale (float, optional): Scaling factor for attention scores window_size (tuple[int, int], optional): Window size for sliding window attention as (left, right). Use (-1, -1) for full attention (default), (-1, 0) for causal attention, or (W, 0) for causal attention with sliding window of size W. Returns: output (Tensor): Output tensor from attention computation; shape :math:`(T_q, H, D)`. If ``return_aux`` is not None and ``return_aux.lse`` is True: lse (Tensor): Log-sum-exp of attention scores; shape :math:`(T_q, H)`. Shape legend: - :math:`N`: Batch size - :math:`T_q`: Total number of query tokens in the batch (sum of all query sequence lengths) - :math:`T_k`: Total number of key/value tokens in the batch (sum of all key/value sequence lengths) - :math:`H`: Number of attention heads - :math:`D`: Head dimension Example:: >>> # xdoctest: +REQUIRES(env:TORCH_DOCTEST_CUDA) >>> batch_size, max_seq_len, embed_dim, num_heads = 2, 512, 1024, 16 >>> head_dim = embed_dim // num_heads >>> seq_lengths = [] >>> for _ in range(batch_size): ... length = torch.randint(1, max_seq_len // 64 + 1, (1,)).item() * 64 ... seq_lengths.append(min(length, max_seq_len)) >>> seq_lengths = torch.tensor(seq_lengths, device="cuda") >>> total_tokens = seq_lengths.sum().item() >>> >>> # Create packed query, key, value tensors >>> query = torch.randn( ... total_tokens, num_heads, head_dim, dtype=torch.float16, device="cuda" ... ) >>> key = torch.randn( ... total_tokens, num_heads, head_dim, dtype=torch.float16, device="cuda" ... ) >>> value = torch.randn( ... total_tokens, num_heads, head_dim, dtype=torch.float16, device="cuda" ... ) >>> >>> # Build cumulative sequence tensor >>> cu_seq = torch.zeros(batch_size + 1, device="cuda", dtype=torch.int32) >>> cu_seq[1:] = seq_lengths.cumsum(0) >>> max_len = seq_lengths.max().item() >>> >>> # Call varlen_attn >>> output = varlen_attn( ... query, key, value, cu_seq, cu_seq, max_len, max_len ... ) )r r)rArB torch_attnrOlistr)r$r%r&r'r(r)r*r\r,rr+outrrMs rrrsj\w&I))&&33       [ KCa*..x Jrctxinputs.rJc Uu nnnnnnn n n n UupnURX4XVX}X5 XlXlXlXlXlgN)save_for_backwardr)r*r+r,r)rarbrJr$r%r&r'r(r)r*r+r,rr`rrLs r_setup_contextrfsa         Ci%excUIIMI!Orz!torch_attn::_varlen_attn_backwardgrad_outr`rrLcV[U 5n [R"SURS9nUR=(a [ URR 5nU(al[RS5 U SS:wd U SS:wa [S5e[RRRUUUUUUUUUU SU U UU S9unnnOV[RS 5 [RRRUUUUUUUUUU SU U UU U SU SS 9unnnUUU4$) Nr)r;r.r r/r0r1r2r4)r,r6r7)rrArUr;r<rr=r>r?r@rBrC_cudnn_attention_backward_flash_attention_backward)rgr$r%r&r`rr'r(r)r*r+rLr,runusedrHdqdkdvs r_varlen_attn_backwardrosD")5K [[5<< 0F G"3ELL4F4F"GI 67 q>R ;q>R#7f YY^^==              > B$ @AYY^^==              (^)!n#> B& r2:rc[U 5n [R"U5n[R"U5n[R"U5nXU4$)z> Fake implementation for meta tensor computation and tracing. )rrArQ)rgr$r%r&r`rr'r(r)r*r+rLr,r grad_querygrad_key grad_values r_varlen_attn_backward_fakert\sI()5K!!%(J$H!!%(J  ++rgrad_lsegrad_rngc(URupEpgppURn URn URnURnUR n[ RRRUUUUU U UUU U UU UU5unnnUUUSSSSSSS4 $rd) saved_tensorsr)r*r+r,rrArBr^ro)rargrurvr$r%r&r'r(r`rrLr)r*r+r,rrlrmrns r _backwardryysBEARAR>E IIE IIE I IIE//K%%;;        JBB r2tT4tT4 ??r) setup_context)FNN)NN) rlogging functoolsrtypingrrrA getLoggerrr>__all__r_intrr rrlibrary custom_opTensorrVtuplerO register_faker[rrfrortryregister_autogradrrrrs" ! , 'S D(8T#Y 1CD 3"E$(B+ <<B+ B+ <<B+ll B+ ll B+  B+ B+B+ 4<B+cT!B+ 5<<u|| 34B+FB+J$((( <<(( (( <<((ll (( ll ((  (( (((( 4<((cT!(( 5<<u|| 34((((h%)#+] <<] ] <<]ll ] ll ]  ] ]T!] 4<]sCx] \\E%,, 455]@""U38_"c"d"0<2N$(AllA <<A A << A  A  AllAllA A AA||A 4<AcT!A 5<<u|| 34AOAH$$$(,ll, <<, , << ,  ,  ,ll,ll, , ,,||, 4<,cT!, 5<<u|| 34,%,8@ @ @05 @HM @ 5<<$  #$@< yGr