ZjzSSKrSSKJr SSKJr S\R R S\RS\RS\RS \RS-S \S \RS \R\\ \R4-S \ S\ \\ \ 4-S\RS-S\ \RS44Sjr \RRS\R R S\RS\RS\RS \S \RS\ \ \ 4S\RS\R4Sj5rg)N)PagedAttentionCache)!lazy_import_paged_flash_attentionmoduleqkvattention_maskcache cu_seq_lens_q cu_seq_lens_k max_seqlen_q max_seqlen_k block_tablereturnc B[URR5up[USS5(dSOURS- S4nUS:XaSOSn[ U[ 5(aXnXn U Gc URUUURU S U S S 9up#S U ;aS U RS 50O0nU "URSS 5RS5R5UR5UR5UR[R5UR[R5R!5UU 4UR"SUS.UD6n[ U[$5(aUSnUS4$S U ;aS U S 0O0n['XX#XWXU 4 0UD6nUS4$)a Performs the forward pass of attention with paged key-value cache. This function handles the cache updates and performs the attention computation. For decode-only batches (when block_table is provided), uses `flash_attn_with_kvcache` for fused attention + cache update. Otherwise uses `flash_attn_varlen_func`. See the [paged attention guide](https://huggingface.co/docs/transformers/en/paged_attention) for more details. Args: q: (1, nheads, total_q, headdim), where total_q = total number of query tokens in the batch. k: (1, nheads_k, total_k, headdim), where total_k = total number of key tokens in the batch. v: (1, nheads_k, total_k, headdim), where total_k = total number of key tokens in the batch. cu_seq_lens_q: (batch_size + 1,), dtype torch.int32. The cumulative sequence lengths of the sequences in the batch, used to index into q. cu_seq_lens_k: (batch_size + 1,), dtype torch.int32. The cumulative sequence lengths of the sequences in the batch, used to index into kv. max_seqlen_q: int. Maximum query sequence length in the batch. max_seqlen_k: int. Maximum key sequence length in the batch. block_table: (num_groups, batch_size, max_blocks_per_seq), dtype int32. Block table for paged KV cache. If provided, uses flash_attn_with_kvcache for fused attention + cache update. For each request, the block table is a vector of size (max_blocks_per_seq,) with indices indicating the physical location of the cache to read from and write to. The kernel, using the cache_seqlens for that request, knows how much cache to read and dispatches the read using the block table. Same for the write. If a request has fewer than max_blocks_per_seq blocks, the block table is padded with -1s to indicate that the block is not allocated. sliding_windowF)rrfull_attentionsliding_attentionN read_index write_index) key_states value_states layer_idxrrs_auxrT) softmax_scalecausal window_size)rconfig_attn_implementationgetattrr isinstancedictupdaterget transposesqueeze contiguoustotorchint32clonescalingtuple_paged_decode_forward)rrrr r r r r rrrkwargsflash_attn_varlen_funcflash_attn_with_kvcacher layer_type custom_kwargs attn_output flash_kwargss v/root/GenerationalWealth/GenerationalWealth/venv/lib/python3.13/site-packages/transformers/integrations/flash_paged.pypaged_attention_forwardr:sJ7X **73 &-V5Eu%M%MXTZTiTilmTmopSqN%3x%?!EXJ-&&%1 #/ ||&&l+}-  ;BV:K&**W"56QS , KK1  % %a ( 3 3 5 LLN LLN   U[[ )   U[[ ) / / 1    !..&     k5 ) )%a.K   6=5F1B + qU>\g kw   rc 6URURupURU RSURUR UR 5n URU RSURUR UR 5n URSSSS5R5nURSSSS5R5nURSSSS5R5nURS5nUSUS-USU- S- R[R5nXXRU5'U"S UU U UUUURSUS. U D6n[!U["5(aUSnUR%S5$) zaDecode fast path using flash_attn_with_kvcache. Disabled because FA3 has issue with tracing this.rrrrNT) rk_cachev_cacherr cache_seqlensrrr )layer_index_to_group_indicesr key_cacheview block_sizenum_key_value_headshead_dim value_cachepermuter*sizer+r,r-get_block_table_keyr/r$r0r))rrrr r r rr4rr8 group_idxlayer_idx_in_groupr>r? batch_sizer@r7s r9r1r1[s%*$F$FvGWGW$X!Ioo0166r5;K;KUMfMfhmhvhvwG 2388 E  e77G !Q1((*A !Q1((*A !Q1((*AJ"1zA~6{ 9SSVWW[[\a\g\ghMGRG]L**+BCD)   #nn"   K+u%%!!n   q !!r;)r,generation.continuous_batchingrmodeling_flash_attention_utilsrnnModuleTensorr%strintr0r:compilerdisabler1rAr;r9rXs @NQ HHOOQ ||Q ||Q || Q LL4' Q  Q<<Q<<$sELL'8"99QQS#X&Q$Q 5<< Qh." HHOO." ||." ||." || ."  ." << ."#s(O."." \\."."r;