Zj@# rSSKJr SSKJr SSKJrJr SSKJr SSK r SSK J r SSK J r SS KJrJrJrJr "S S 5r\"S S 55rS\ S\4SjrS\S\S\S\4SjrS'S\S\S\S\4SjjrS\S\S\S\4SjrS(S\ R6S\\S\\S\SS4 S jjrS!\S"\S#\S$\S%\S\\4 S&jrg))) OrderedDict) dataclass)ceillog2)AnyN)PretrainedConfig)ContinuousBatchingConfig)FutureRequestState RequestState RequestStatusloggerc\rSrSrSrS\SS4SjrSSjrS\\S 4S\ RRS-4S jr SS \ SS4S jjrS\\S 4S \ RRSS4SjrSrg)CudaGraphBufferz>A fixed-size dict for CUDA graphs with LRU eviction when full.max_sizereturnNcVUS::a[SU35eXl[5Ulg)Nrz#max_size must be positive, but got ) ValueErrorrr_storage)selfrs ڂ/root/GenerationalWealth/GenerationalWealth/venv/lib/python3.13/site-packages/transformers/generation/continuous_batching/utils.py__init__CudaGraphBuffer.__init__s* q=B8*MN N LWM cTURnSUlURSS9 Xlg)Nr T)silent)rplan_for_new_graph)roriginal_max_sizes r__del__CudaGraphBuffer.__del__$s) MM  t,) rkey.cxURRU5nUbURRU5 U$N)rget move_to_endrr"graphs r get_graphCudaGraphBuffer.get_graph*s3 !!#&   MM % %c * rrc.[UR5UR:arURRSS9up#U(d[R "SU<35 UR 5 [UR5UR:aMqgg)NF)lastz!Evicting graph for evicted_key = )lenrrpopitemrinforeset)rr evicted_key evicted_graphs rr"CudaGraphBuffer.plan_for_new_graph0sl$-- DMM1)-)>)>E)>)J &K @+1ABC    ! $-- DMM1rr(c@UR5 X RU'gr$)rrr's r set_graphCudaGraphBuffer.set_graph7s !" cr)rr)rN)F)__name__ __module__ __qualname____firstlineno____doc__intrr tupletorchcuda CUDAGraphr)boolrr5__static_attributes__rrrrsHZZZ * U38_1E1E1L ""$"#U38_#UZZ5I5I#d#rrc@\rSrSr%SrSr\\S'Sr\\S'S Sjr Sr g) WorkloadHints=zRA tiny dataclass containing hints to help choose good continuous batching defaultsrmax_prompt_lengthmax_generated_lengthNcUR(aiUR(aWURcIURUR-n[[ X!R - 55S-nX3S--Ulgggg)z*Resolves the config using the given hints.Nr )rGrHmax_blocks_per_requestr<r block_size)r cb_configmax_sequence_lengthblocks_per_requests rresolve_using_hints!WorkloadHints.resolve_using_hintsEsw  ! !d&?&?//7&*&<&= (size), capped at (max_value).r)rmin)rWrXrYpaddeds rpad_to_intervalr]Ts5;?!8T$& '- 7F v !!rvalue min_valuec[U[SU55nS[[[U555-n[ X15$)zReturn the smallest power of 2 >= (value), capped at (max_value). If a minimum value is provided, the value is at least padded to that value.r rJ)maxr<rrr[)r^rYr_r\s r pad_to_pow2rb\s: s1i( )E #d4;'( (F v !!rx divide_byalign_tocV[[X- 55nX-(a XX-- - nU$r$)r<r)rcrdres raligned_dividergds, D  A|  && Hrattention_maskcumulative_seqlens_qcumulative_seqlens_ksliding_windowc*[R"UR5Rn[ [ U5S- 5HnXS-X- nX%S-X%- nXg:aUS:aXv- S-nOSn[ XXS-5n [ X%X%S-5n [R"USX4RUURURS9n [R"XS9n US:aXv- U- n U [R"XS9- n XSX4'M g)u~Builds an attention mask inplace using the cumulative seqlens of the query and key. If given a sliding window, it will also apply a sliding window mask on top. The attention mask is not boolean, it uses zeroes and -inf (or its equivalent) so it's more of an attention score bias tensor. The attention mask is a block-diagonal matrix, with each block an attention mask for a single query-key pair. Each of those block is built from a causal mask and, if there is a sliding window, a sliding window mask. An example is represented below, with seqlen_k = 8, seqlen_q = 4 and sliding_window = 6: CAUSAL MASK: █ █ █ █ █ ░ ░ ░ █ █ █ █ █ █ ░ ░ █ █ █ █ █ █ █ ░ █ █ █ █ █ █ █ █ SLIDING WINDOW MASK: ┌──────────────────────── seqlen_k - seqlen_q - sliding_window = 8 - 4 - 6 = -2 offset to the left <─┴─> ░ █ | █ █ █ █ █ █ █ █ ░ ░ | █ █ █ █ █ █ █ █ ░ ░ | ░ █ █ █ █ █ █ █ ░ ░ | ░ ░ █ █ █ █ █ █ ATTENTION MASK (sum of causal and sliding window masks): █ █ █ █ █ ░ ░ ░ █ █ █ █ █ █ ░ ░ ░ █ █ █ █ █ █ ░ ░ ░ █ █ █ █ █ █ Another example with seqlen_k = 5, seqlen_q = 3 and sliding_window = 2: CAUSAL MASK: █ █ █ ░ ░ █ █ █ █ ░ █ █ █ █ █ SLIDING WINDOW MASK: ┌──────────────────────── seqlen_k - seqlen_q - sliding_window = 5 - 3 - 2 = 0 offset to the left <┴> | ░ █ █ █ █ | ░ ░ █ █ █ | ░ ░ ░ █ █ ATTENTION MASK (sum of causal and sliding window masks): ░ █ █ ░ ░ ░ ░ █ █ ░ ░ ░ ░ █ █ r .)dtypedevice)diagonalN) r>finformr[ranger-slicefullshaperntriutril)rhrirjrkr_iseqlen_qseqlen_kcausal_diagonal query_range key_range minus_infmaskedsliding_diagonals rbuild_attention_maskrks-t N00155I 3+,q0 1'A.1E1HH'A.1E1HH  8q=&1A5OO035Ia%5PQ .13GA3NO JJ 3 6 7 = =  &&!((  I@ A '2^C  ejjF FF6 transformers.configuration_utilsr+transformers.generation.configuration_utilsr requestsr r r rrrErArVr<r]rbrgTensorlistrrrCrrrsQ$! =PMM##D aa a"H 0HTH "#"c"c"c""s"s"s"3" c c S S  Q=LLQ=s)Q=s)Q= Q=  Q=h        r