ZjoSSKrSSKJrJrJr SSKJr SSKrSSKJ r SSK J r SSK J r SSKJrJr S S KJrJrJrJr S S KJrJrJrJr S \ S \\\\\\44Sjr\"5"SS55r "SS5r!g)N)floorgcdsqrt)Any)PreTrainedConfig)ContinuousBatchingConfig)is_flash_attention_requested) attach_tracertraced) BlockManagerCacheAllocatorFullAttentionCacheAllocatorSlidingAttentionCacheAllocator) RequestState RequestStatusget_device_and_memory_breakdownloggerconfigreturnc6[USS5nUc6[USS5bSOSn[UR5Vs/sHo2PM nn0n[U5HupVUR U/5U/-XF'M [ UR 5Vs/sHn[U5PM sn6n/n UR5H7upg[S[U5U5HnU RXuXX-5 M M9 U V s/sH oU SPM n n X4$s snfs snfs sn f)a| Group layers depending on the attention mix, according to VLLM's hybrid allocator rules: - Layers in each group need to have the same type of attention - All groups have the same number of layers For a model with the following layer types: ["sliding", "full", "full", "sliding", "full", "full", "full", "full"] We would get four groups: [0, 3], [1, 2], [4,5] and [6,7]. layer_typesNsliding_windowsliding_attentionfull_attentionr) getattrrangenum_hidden_layers enumerategetrvalueslenitemsappend) rr attn_type_ layer_countsi layer_typeindices group_size layer_groupslg group_typess ڂ/root/GenerationalWealth/GenerationalWealth/venv/lib/python3.13/site-packages/transformers/generation/continuous_batching/cache.pygroup_layers_by_attn_typer1s/&-6K+26;KT+R+^'dt */0H0H*IJ*IQy*I JL";/ #/#3#3J#Cqc#I  0<3F3F3HI3Hs7|3HIJJL+113 q#g, 3A   AN ; <4 41== "r!u% K=  $$#KJ>s D  D6Dc\rSrSrSr\R S4S\S\S\R\ -S\RS\ S-S S4 S jjr S \ S \ S \4S jr\S\ S\ S \ S \ S-4Sj5r\S\ S S4Sj5rS \ 4Sjr\S\ S\ S\ S\\\ S-S\\\ S S4 Sj5rS\ S\ S\ S\R,S S4 Sjr\S\ S\ S \\ \ 44Sj5r\S\R,S\R,S\ S\\R,S\\R,S \\R,\R,44 Sj5rS\S \ 4S jrS\ S!\\ S \ 4S"jrS#\S$\ S S4S%jr S&\\ S'\\ S S4S(jr!S)\ S*\\ S \\\ \\ 44S+jr"S.S,jr#S-r$g)/PagedAttentionCache=u Manages the cache for a paged attention mechanism, inspired by VLLM's hybrid allocator. The cache relies on making groups of layers to reduce the complexity of cache management and fragmentation. The cache uses a three-level hierarchy: - Pages: The smallest unit of cache, a page has a size of [num_heads, head_size], which is the space needed to store the key or value states for one token and one layer. For a model with only full-attention layers, to store the KV cache of one token, we need `2 * num_layers` pages: key and values each take `num_layers` pages. Pages are grouped into blocks: - Blocks: A block is a collection of `block_size` pages, serving as the allocation unit to reduce management complexity and fragmentation. Cache is allocated and freed block by block, not page by page. One block is allocated to one layer group, which only has one attention type, like full-attention or sliding-attention. If all layers in the model have the same attention type, then all layers will be in the same group. There is more than one group if and only if the model has a mixed attention types, like layers with full-attention and layers with sliding-attention. - Cache tensors: The physical supports for the cache. There are as many cache tensors as there are layer in a layer group, and the shape of the cache tensor is `[num_blocks * block_size, num_heads, head_size]`. Grouping layers into groups is useful because when we allocate one block to a group N, the block allocated is the same for all layers in group N, equivalently it is allocated across all cache tensors. This allows us to efficiently allocate and free blocks, and to efficiently read and write key and value states. For instance, imagine we have 8 blocks of cache and a model with two layer groups: a full-attention group with 3 layers and a sliding-attention group with 3 layers. At creation time, the physical cache tensors look like this: cache_tensor_0: □ □ □ □ □ □ □ □ cache_tensor_1: □ □ □ □ □ □ □ □ cache_tensor_2: □ □ □ □ □ □ □ □ where □ means the blocks is not allocated to any layer group yet. We have 3 cache tensors because there are 3 layers per group. We allocate 1 block to each group, after allocation, the cache tensors look like this: cache_tensor_0: ✖ ◉ □ □ □ □ □ □ cache_tensor_1: ✖ ◉ □ □ □ □ □ □ cache_tensor_2: ✖ ◉ □ □ □ □ □ □ where ✖ means the block is allocated to the full-attention group, and ◉ means the block is allocated to the sliding-attention group. Now, if we continue to generate, and the sliding window has been reached, we only need to allocate a new block for the full-attention group, and the cache tensors look like this: cache_tensor_0: ✖ ◉ ✖ □ □ □ □ □ cache_tensor_1: ✖ ◉ ✖ □ □ □ □ □ cache_tensor_2: ✖ ◉ ✖ □ □ □ □ □ And after further generation, when we need a new block allocated: cache_tensor_0: ✖ ◉ ✖ ✖ □ □ □ □ cache_tensor_1: ✖ ◉ ✖ ✖ □ □ □ □ cache_tensor_2: ✖ ◉ ✖ ✖ □ □ □ □ This would not have been possible if all layers were in the same group: we would have had to allocate a new block for the sliding-attention group, although it is not needed. Nrcontinuous_batching_configdevicedtypetp_sizerc  XlX@lX0l[USS5nUbUO URUl[USS5nUbUOUR UR-UlURUlURS::a[SUR35e[U5up[US5n [U5Ul 0Ul 0Ul[U5HMupXS:Xa UR OSn [U 5H#upX4URU'XRU'M% MO Ub5US:a/UR U-S:wa[SUR S US 35eURUR -n[#UR5(aSnO SU ;aS nOSnURUR-nSUR S UR$--4nS U-UR U-S U--4n['UUURU UU/US 9nUR(cUR+S S9 UR-UR.UR0UR(URS9unnUUlUUlUR.UR-Ul[4R6"SUR.<SUR<SU<SUR0<SU<3 5 UR8nUcSnUUl/Ul/UlUS -UR-UR UR4UlUR>SS- Ul UR@S- Ul![EU 5Hn[FRH"UR>URURS9n[FRH"UR>URURS9nRJRMU5 [FRJRMU5 UR:ROU5 UR<ROU5 M [4R6"SUR><SUR:SRP<SUR:SRS5<35 URTUl*/Ul+SUl,SUl-SUl.[U 5Hun nUS:Xa4[_XRURTS9nU=RXS- sl,OqUS:Xa][aXRUR UR@URB5nU=RZS- sl-URbUl.O[SU35eURVROU5 M URT=(a U S/:HUl2[gUUR5Ul4SUl5SUl6g)a~Initialize a paged attention cache for efficient memory usage. Also turns in prefix sharing if the model has only full attention layers. Args: config: Model configuration continuous_batching_config: Continuous batching configuration containing cache parameters device: Device for the cache tensors dtype: Data type of the cache tp_size: Tensor parallelism size num_key_value_headsNhead_dimrz%Block size must be positive, but got rr zNumber of key value heads z+ must be divisible by tensor parallel size .)r5 page_size num_groupsr,activation_peaksnum_attention_masksT)has_logit_processors) num_blocksmax_batch_tokensmax_memory_percent cache_dtypez7PagedAttentionCache initialized with self.num_blocks = z, self.block_size = z, page_size = z, self.max_batch_tokens = z num_attention_masks = )r7r6zself.cache_shape = z self.key_cache[0].shape = z self.key_cache[0].numel() = r)allow_block_sharingzInvalid group type: )7rr7r6rnum_attention_headsr: hidden_sizer; block_size ValueErrorr1r#r?sliding_windowslayer_index_to_group_indicesr rr vocab_sizePagedAttentionMemoryHandlerrEresolve_max_memory_percent%infer_num_blocks_and_max_batch_tokensrCrD num_pagesrinfomax_blocks_per_request key_cache value_cache cache_shapesentinel_index trash_indexrtorchempty_dynamomark_static_addressr%shapenumelrGgroup_cache_managersnum_full_attention_groupsnum_sliding_attention_groups%max_sliding_window_blocks_per_requestrr_max_blocks_per_requestuse_prefix_sharingr_block_manager_total_prefix_length_block_table_key)selfrr5r6r7r8kv_headsr;r-r/r,r)grouprjlayerr>rAq_bytes_per_token lm_head_peakattention_peakmemory_handlerrCrDrTr'new_layer_key_cachenew_layer_value_cache group_typecms r0__init__PagedAttentionCache.__init__ws$   6#8$?4<4HfNhNh 6:t4)1)=X6CUCUY_YsYsCs 5?? ??a DT__DUVW W%>f$E! a) l+!,.)!,/HA6AnH[6[V22abN%e,<=611%8.<$$U+-0  7Q;'''1Q6 01I1I0JJuv}u~~AMMD$<$<< ' 4 4"#  K /"# "# #66F   V%6%6!6 6  M   !2 2Q] B  5'A!*N; 3   & 8 8 @ & A AW[ A \'5'['[1<<7HH9LL (\( $ $% 04??: FDOO3GG\$//I]]l`i_mn($$((@*=)A C "@X@XZ^ZgZgh"..q1A5..2z"A"'++d.>.>djjY]YdYd"e $)KK0@0@ [_[f[f$g ! MM - -.A B MM - -.C D NN ! !"5 6    # #$9 : #  *t''++GT^^A->-D-D,HHf$..YZJ[JaJaJcIghi$>#Q#Q :<!)*&,-)562&{3MAz--0OOY]YqYqr..!3.223(=(=t?R?RTXTdTd11Q61=?=W=W: #7 |!DEE  % % , ,R 04#'":":"`{O_N`?`*:tG)*!!%num_requested_blocksallocated_blockscXR-nUR(a4[URU- S5nU[ XA5UR-- nX0R 5:*$)aReturns a boolean indicating if the allocation of (num_requested_blocks) blocks will be successful. The number of newly allocated blocks needed is predicted by the following rules: - for full attention groups: since there is no sliding window for full attention layers, one requested block is always equivalent to one newly allocated block for EACH full attention group - for sliding window groups: because of the sliding window, the number of blocks allocated to a request is capped. Using the number of already (allocated_blocks) we can compute the number of new blocks to actually allocate to the request, which can be lower than the number of requested blocks. That number is the same for all sliding window groups, as only one sliding window size is supported. r)rarbmaxrcminget_num_free_blocks)riryrz needed_blocks blocks_lefts r0will_allocation_be_successful1PagedAttentionCache.will_allocation_be_successfuls`-/M/MM  , ,dHHK[[]^_K SCdFgFgg gM 8 8 :::rxn_blocks request_idcURX5(dgSnURH>nURXUR5nUc[ SUSU35e[ XF5nM@ U$)zAllocate cache blocks across all layer groups for a given request. Actual allocation is done by the cache managers, and this method only returns the maximum number of blocks actually allocated across all managers.NrzFailed to allocate z blocks for request )rr`allocate_blocksrfrKr|)rirrrz max_allocatedrunum_allocated_blockss r0r#PagedAttentionCache.allocate_blocks-sz 11(MM ++B#%#5#5hDL_L_#` #+ #6xj@TU_T`!abb DM , rxc`URHnURXR5 M g)zFree all allocated cache blocks for a given request across all layer groups. Actual deallocation is done by the cache managers.N)r` free_blocksrf)rirrus r0rPagedAttentionCache.free_blocks=s&++B NN:':': ;,rxc.URR$)zHGet the current number of unallocated blocks available for new requests.)rfnum_free_blocks)ris r0r~'PagedAttentionCache.get_num_free_blocksDs""222rx past_length query_length read_index write_indexc [URU5H&upgURURXU55 M( UbA[URU5H&uphURUR XU55 M( gg)a5Retrieve physical cache indices for reading KV states in the cache across all layer groups. This method coordinates with all cache managers to build the complete set of read indices needed for attention computation. When read_index is None, the batch has no cache reads and we only compute the write indices. N)zipr`extendget_write_indicesget_read_indices) rirrrrrru write_indices read_indicess r0extend_read_and_write_indices1PagedAttentionCache.extend_read_and_write_indicesHs~"%T%>%> !L B  !5!5j|!\ ]"M  !$'(A(A:$N ##B$7$7 Q]$^_%O "rx block_tablecj[UR5HupVURXX4U5 M g)N)r r`fill_block_table)rirrrrr)rus r0r$PagedAttentionCache.fill_block_table]s0t889EA    ST~ V:rxc0nURS:aX-US'URS:a(U[XRRS- 5-US'U$)zRetrieve the key sequence length for the given request_id across all layer types. Returns a dictionary of layer types to their corresponding key sequence lengths.rrr r)rarbr}rr)rirr seqlens_ks r0 get_seqlens_k!PagedAttentionCache.get_seqlens_kcs^  ) )A -*5*DI& '  , ,q 0-9C [[MgMgjkMk   ! ! #q (   #4 A   #4 C+ +--i8 Q    #4 A   #4 C$)$6$6wCS$T !&+&8&8!EU&V #"%==%(;(;;FFrJTTUWXD$)$6$6wCS$T ! ! 1 1$ C&+&8&8!EU&V # # 3 3D G   #4 A   #4 C%==rxflash_attn_with_kvcache_fnc8URc[R"U5RR 5nSU;aSUlUR$SU;aSUlUR$[ S[R"U535eUR$)zA function to get the name of the block table key for the given flash_attn_with_kvcache_fn. The function's signature is only inspected once. This is necessary because different version of flash have different names for the block table key.r page_tablezOflash_attn_with_kvcache_fn does not have a block_table or page_table argument: )rhinspect signature parameterskeysrK)rir kwarg_namess r0get_block_table_key'PagedAttentionCache.get_block_table_keys  (!++,FGRRWWYK +(5%$$$ ,(4% $$$!efmfwfwySgTfUV$$$rx prompt_idscnSn/n[[U5UR-5HnX%UR-US-UR-nURR X6SS9nURR R U5nUb.URU5 URRU5 M O U(aC[R"SUS[U5S35 URSnXHRU'[U5UR-n U=RU - sl U $)a Searches for a prefix match in the cache for the given (prompts_ids). If one is found, we reference the matching blocks in the (request_id), increase the reference count of the blocks and return the number of blocks that match. If no prefix match is found, we return 0.Nr r)group_idzFound prefix match for request z with z blocks)rr#rJrf compute_hash _hash_to_idr!r%increase_ref_countrdebugr`rrg) rirr current_hashrzbtokensblock_idru prefix_lengths r0search_prefix_match'PagedAttentionCache.search_prefix_matchs s:$//9:ADOO 3q1u6OPF..;;L[\;]L**66::<HH# ''1##66x@;  LL::,fSQaMbLccjk l**1-B)9NN: &,-?  !!]2!rxstatenum_complete_blocksc,US:XdUR[R:XagURH_nUR(dMUR R UURURURUR-S9 Ma g)aMarks the blocks allocated to a request (state) as complete if they are shareable and they have been computed in the forward pass. A complete block is a block where the KV cache has been fully computed: if the block has enough space to hold the cache for N tokens, the block is marked as complete when the cache data is present for the N tokens. If block sharing is off, this is a no-op.rN)rrzr) statusrFINISHEDr`uses_block_sharingrf!mark_shareable_blocks_as_completerrinitial_tokensgenerated_tokens)rirrrus r0r5PagedAttentionCache.mark_shareable_blocks_as_completes ! #u||}7M7M'M++B$$$##EE(;%'^^E4D4D%E % 4 4u7M7M MF,rxlist_source_blockslist_forked_blocksc[R"XR[RS9n[R"X R[RS9n[ UR UR 5HuupVURSURURUR5nURSURURUR5nXSXT'XcXd'Mw g)z;Copy the cache from the source blocks to the forked blocks.)r6r7rN) rZtensorr6int32rrUrVviewrJr:r;)rirr source_blocks forked_blocksrUrVs r0 copy_cachePagedAttentionCache.copy_caches %7 SXS^S^_  %7 SXS^S^_ &)$..$:J:J&K "I!r4??D>#6#6#8 9,)J   Z (*rx)rfrhrgrGrJrWrr6r7r`r;rUrMrDrTrcrCrar?r:rRrbrXrLrYrerV)rN)%__name__ __module__ __qualname____firstlineno____doc__rZfloat16rr r6strr7intrvboolrr rrr~listrTensorrdictrtuplerrrrrrrrr__static_attributes__rxr0r3r3=s6z#]]" b% b%%=b% s" b% {{ b% t b% b%H;#;Y\;ae;$    PS X[^bXb    LL;>ll;> ;> & ;> %,,' ;> u||U\\) *;> ;>z%c%c% ctCyS4|Z]bf"DT#YDDQTIDZ^D 1c 1DQTI 1Z_`deh`ikopskt`tZu 1)rxr3c\rSrSrSr\R r\Rr Sr Sr S\ S\ S\ S\ S \\\ \ 4S \ S S 4S jr\SS\S \ 4Sjj5rS\\ \ 4S\R(S \\ \ \ \ 44Sjr\S\S\S\S \4Sj5rS\\ \ 4S\ S\ S -S\ S -S\R(S \\ \ 44 SjrS S S\R04S\ S -S\ S -S\S\R(S \\ \ 44 SjjrS\ S\ S\R(S \ 4SjrSrg ) rOi uDetermines the optimal number of pages (N) and max batch tokens (M) for the paged attention cache, given available GPU memory. The relation between N and number of blocks is: num_blocks = N // block_size. The memory footprint is a polynomial in N and M, where each term maps to a tensor allocated in ``ContinuousBatchingIOs._setup_static_tensors`` or ``PagedAttentionCache.__init__``: memory(N, M) = coeff_n · N + coeff_m · M + coeff_nm · N·M + coeff_mm · M² See ``_equation_coefficients`` for the breakdown. All three solving modes (auto, fixed-N, fixed-M) reduce to solving this equation, which is at most quadratic in one variable. iir5r>r?r,r@rArNc4URUlX lX0lX@lXPlX`lUR UlUR cURUlUR(aSOSUl UR(aSUl gSUl g)u>Initialize the memory handler. `activation_peaks` is a list of `(Δcn, Δcm)` pairs giving the activation memory contributions proportional to N (pages) and M (batch tokens) for each peak. Memory must satisfy the constraint at every peak, so we solve each polynomial independently and take the most restrictive result.Nr=r ) rJr>r?r,r@rArTfallback_max_blocks_per_requestreturn_logprobsnum_output_rowsuse_async_batching io_multiplier)rir5r>r?r,r@rAs r0rv$PagedAttentionMemoryHandler.__init__s5??"$$ 0#6 &@&W&W#  & & .*D*d*dD '$>$N$NqTU"<"O"OQUVrxrEcV[5upp4U[XC5- n[XP-5nU$)z^Calculate available GPU memory for cache allocation, accounting for already allocated tensors.)rr|r)rEr'totalreserved allocatedavailable_memorys r0get_available_memory0PagedAttentionMemoryHandler.get_available_memory7s7)H(I%( 3y#;;/DErxpeakrFcURRnURRnURnURnUupxSUR-UR -U-X`R -S--Xt--n X-US-U--X`R-U--X`R -UR-U--X`R -S--X`R -S--n X`R-U-n X`R-U-n XX4$)u{Returns `(coeff_n, coeff_m, coeff_nm, coeff_mm)` for the memory polynomial of a single activation peak. `peak = (Δcn, Δcm)` is the peak-specific activation contribution; the rest of the coefficients are shared across peaks. Each addend is annotated with the tensor it corresponds to in `ContinuousBatchingIOs._setup_static_tensors` (or the forward pass, for activation terms). r=) _input_dtypeitemsize_activation_dtyper r,r>r?rrTrA) rirrFr)ackdelta_ndelta_mcoeff_ncoeff_mcoeff_nmcoeff_mms r0_equation_coefficients2PagedAttentionMemoryHandler._equation_coefficientsAs;    & &  " " + +       $.. 01 4//!A% &k   K!eai &&&* +//!))*,-. . //!A%  & //!A%  & ///!3///!333rxrrrcUS:XaU*U- $US-SU-U-- nUS:a[SUS35eU*[U5-SU-- nUS:a[SUS35eU$)uQLargest positive root of a·x² + b·x + c = 0. Falls back to linear when a == 0.rr=z!No real solution (discriminant = )zNo positive solution (root = )rKr)rrr discriminantroots r0_solve_quadratic,PagedAttentionMemoryHandler._solve_quadraticgs 626M!ta!eai' ! @aPQ QT,''AE2 !82a<PQIY//94;W;WXJ++  %__,A%%c<)9KLA"58T-O-OP ++rxg?cXURU5n[R"SU35 [S5n[S5nURH.nUR XXU5up[ Xi5n[ Xz5nM0 Xgp!URXU5n X:a[SU SU35eX4$)uiSolve for the missing variable(s) in the memory polynomial (see ``_equation_coefficients``). There is one polynomial per activation peak; we solve each independently and take the most restrictive (smallest) result. When both `N` and `M` are unknown, assumes `M = m·N` (m = 0.01, i.e. one batch fills ~1 % of the cache) and solves the resulting quadratic in N. zCache memory: infzMemory footprint z is more than available memory ) rrrSfloatr@r6r}compute_memory_footprint MemoryError) rirCrDrErFr,acc_num_blocksacc_max_batch_tokensrrm_batch_tokensmemory_footprints r0rQAPagedAttentionMemoryHandler.infer_num_blocks_and_max_batch_tokenss--.@A  nYK01u$U|))D'+';';DZkv'w $H :N#&';#L * (6$88Wbc  ' 12B1CCbclbmno o++rxcXR-nUnSnURH;nURXs5uppX-X--X-U--X-U--n [Xl5nM= U$)zaEvaluate the memory polynomial at concrete (N, M) values, taking the max across activation peaks.r)rJr@r#r|) rirCrDrFr5r4max_memory_footprintrr0rur1r2r@s r0r;4PagedAttentionMemoryHandler.compute_memory_footprintsv  (  ))D#::4M BC!v1)g?)rrrrrrZbfloat16rrrr.r/r rrrrv staticmethodr:rr7r#r*r6rrQr;rrrxr0rOrO s ;;L$(!"W$<WW W  W uS#X/ W!W W4    #4#s(O#427++#4 sCc! "#4L E e  %  #,CHo#,#,$J #, * #, [[ #, sCx#,N"&'+$'#(== ,$J,*," , [[ , sCx ,: $3 $# $\a\g\g $lo $rxrO)"rmathrrrtypingrrZconfiguration_utilsrgeneration.configuration_utilsr utils.genericr utils.metricsr r cache_managerrrrrrequestsrrrrrrrrr1r3rOrrxr0rOs!! 3F92ttZZ%&6%5d3iRVWZR[A[;\%BI)I)I)Zu$u$rx