ZjXSSKrSSKJrJr SSKJr SSKJrJr SSK J r SSK J r J r JrJr "S S \5r\"5"S S \55r\"5"S S\55r\\S.rg)N)ABCabstractmethod)deque) attach_tracertraced)PagedAttentionCache)FutureRequestState RequestState RequestStatusloggerc$\rSrSrSrS\4SjrS!Sjr\S\ 4S j5r \ S \ S \ S\ \\S-\\ \ 44S j5r\S\4S j5r\S\SS4Sj5rS\ \\ 44Sjr\S\S\\ 4Sj5r\S\4Sj5r\S\\ 4Sj5r\S\S\4Sj5r\S\ S\ S\4Sj5rS\ S\\S\\ 4SjrS\ S\\ S \ S\\SS4 SjrS"S\\ S \ S \ S\\S\S\ \\\\\ \ 44 SjjrS\\ 4Sjr S\\SS4Sjr!S r"g)# Schedulera1 Abstract base class for scheduling requests in the continuous batch processor. Schedulers manage the lifecycle of requests from when they are added to the waiting queue to when they are scheduled for processing. Different schedulers implement different strategies for prioritizing and batching requests. cachecLXl[R"5UlURR(aSOURR R UlURRURR-Ul UR5 g)N) r threadingLock_cancellation_locknum_full_attention_groupsconfigsliding_windowread_cache_limitmax_blocks_per_request block_sizemax_decode_fast_path_lengthreset)selfrs چ/root/GenerationalWealth/GenerationalWealth/venv/lib/python3.13/site-packages/transformers/generation/continuous_batching/scheduler.py__init__Scheduler.__init__si "+.."2(, (L(LRVR\R\RcRcRrRr+/::+L+LtzzOdOd+d( returnNcx0Ul0Ul[5Ul[ 5Ul/UlSUlg)z0Reset scheduler state for a new generation loop.FN)active_requestswaiting_requestsrwaiting_requests_orderset_requests_to_cancel_requests_to_forkblock_new_requestsrs r rScheduler.reset's68:9;27'#-0U 57"'r#statec~XRUR'URRUR5 g)z#Adds a request to the waiting list.N)r' request_idr(append)rr/s r add_waiting_requestScheduler.add_waiting_request0s238e../ ##**5+;+;$$U%5%56266u7G7GH,77 &&.JJ$$U%5%56266u7G7GH||}777*/*H*H'13. -55 8||}4449>$$U%5%56,77 266u7G7GH-;M-JE *&4]l&CE #r# candidates safety_marginc /nSnURRS:nXPRR-n X#pUGHn URR5n X:nU(aFU(a?U R[ R :wa![R"SU <SU <35 GOCURX5n[[U5U5nUS:H=(a U RUR:nU R5nURb[UUR5nU(aU(d UU:aGMUR!U U5nU(dLSnU S:XaAU R"UR$;a'[R"SU R"S 35 GOLGMiUR'XX$5 [U R(5nUUS:H=(a U RUR:-nUU-nUU-nURR*(aFU R5URR,-nUU-nUURR,-nOSnU R.(+nUR1[3U UUU55 U R"nUR$R5US5SLnU(aUR7U5 US:XdUS::dGMU(aGM O X- nX- nXgUUU4$) aSchedules candidate requests for the current batch. This method contains the common logic shared by all schedulers: it checks token and cache budgets, allocates cache blocks if needed, updates request states, and tracks which waiting requests should be removed from the waiting queue. FrzJOutside safety margin, breaking out of scheduling loop. num_free_blocks = z safety_margins = r NTzBreaking mid-loop for request z because the cache is full)rr num_blocksget_num_free_blocksrkr rsrinforwrqr=rrrr`rrfr1r'r{rtallow_block_sharingrror2r rBrQ)rr}r5r6rhr~scheduled_requestsone_allocation_faileddecode_fast_pathsafety_marginsoriginal_token_budgetoriginal_cache_budgetr/num_free_blocksoutside_safety_marginrv request_lenis_decode_eligibleread_cache_neededallocation_successfultokens_in_current_blocktokens_after_forwardcomplete_blocks has_new_tokenreq_id was_waiting num_q_tokens max_kv_reads r _process_candidatesScheduler._process_candidatess  %::<)>>7C4E"jj<<>O$3$D !$); P]PfPf@f aNbbudrcvw"77bNc.1<@K "-!1!ne6K6KdNnNn6n  % 1 1 3 $$0$'(94;P;P$Q!$);PaAa%)$C$CE;$W !)(,%#a'E,<,<@U@U,UKK"@AQAQ@RRl mn  " "5, ke556K  q 0 mU5J5JTMmMm5m m  K 'L - -Lzz--*/*;*;*= @U@U*U''>'L$"6$**:O:O"O"#!& > >>M  % %&8 `k&l m%%F//33FDAMK266v>q \Q%6?O?OM P-; +: !:JLZeeer#c[5n[5nURH6nURUnUR(aUOUR U5 M8 /nU(dU(a^U(aUR UR 55 U(aUR UR 55 U(aMUU(aM^U$)zReturns waiting requests in priority order. Since CPU-offloaded requests are cheaper to restore than fresh requests, they get priority, but we interleave them with fresh request to not saturate new batches with only offloaded requests.)rr(r'rmr2popleft)r offloadedfreshrr/ordereds r _get_waiting_candidates!Scheduler._get_waiting_candidates.s*/ %*W11F))&1E00Ye C CE J2')5y0023u}}/ i55 r#cr[URVs/sH o"U;dM UPM sn5Ulgs snf)z8Removes processed requests from the waiting queue order.N)rr()rrhrs r _cleanup_waiting_queue Scheduler._cleanup_waiting_queue?s2&+"&"="= r"=OqAqV"= r' # rs 44) rr*r+r&r,rrrr'r()r$N))#__name__ __module__ __qualname____firstlineno____doc__r r!rrr r3rinttuplelistr r<r9r>strrDrJrNrRrYr\rfr)rwr{floatrrr__static_attributes__r8r#r rrs 1( == = mm/2m t&'$.c3> ?mm MdM M 333 3@eC,=&>@ CDI   535 5   $|*<   "  s t     sW[ $<]`ad]ejnorjs2$D$DS $D $D -0H $D  $DX # ^f&^f^f ^f -0H ^f  ^f t&'tS#= >^f@l);" S VZ r#rc v^\rSrSrSrS S\S\4U4Sjjjr\S\ S\ S\ \ \ S -\ \ \ 44S j5rS rU=r$) FIFOScheduleriGaCThis scheduler processes requests in the order they arrive, meaning decoding requests has priority over prefilling requests. Additionally, it includes a safety margin mechanism to prevent cache exhaustion. By default, when 80% of the cache is full, new requests will not be scheduled to prioritize decoding active requests.rr~c0>[TU]U5 X lg)aCInitializes the FIFO scheduler. The safety margin is the percentage of free blocks under which we stop scheduling new prefill requests, so safety_margin = 0.1 means that when there is less than 10% of free blocks, or equivalently when more than 90% of blocks are already allocated, we stop scheduling new prefill requests. N)superr!r~)rrr~ __class__s r r!FIFOScheduler.__init__Ms *r#r5r6r$Nc /n/nURR5HenUR[R:XaUR U5 M4UR[R :XdMTUR U5 Mg UR(dURUR55 X4-n[5nURUUUUURS9uppn URU5 U(d U (aSU SS4$XX4$)Nr~r)r&valuesrkr rsr2rpr,extendrr)rr~r rr5r6priority_statessecond_priority_statesr/r}rhrrrrrs r r9FIFOScheduler.schedule_batchUs/157))002E||}555&&u-!9!99&--e4 3&& " ) )$*F*F*H I$= -0U*  $ $2"00 %  _3CS^ ##$FG"&;)1a/ /!\NNr#r)g?)rrrrrr rr!rrrrr r<r9r __classcell__)rs@r rrGslq+1+%++ #O#O/2#O t&'$.c3> ?#O #Or#rc R\rSrSrSr\S\S\S\\\ S-\ \\44Sj5r Sr g) PrefillFirstScheduleri~zScheduler that prioritizes split prefill requests over decoding requests. This scheduler ensures that split prefill requests (which are continuations of partially processed prompts) are completed before processing new decoding requests.r5r6r$Nc /n/nURR5HenUR[R:XaUR U5 M4UR[R :XdMTUR U5 Mg UR(dURUR55 X4-n[5nURUUUUSS9uppn URU5 U(d U (aSU SS4$XX4$)Nrrr) r&rrkr rpr2rsr,rrr)rrrs r r9$PrefillFirstScheduler.schedule_batchs/157))002E||}777&&u-!7!77&--e4 3&& " ) )$*F*F*H I$= -0U*  $ $2! %  _3CS^ ##$FG"&;)1a/ /!\NNr#r8) rrrrrrrrrr r<r9rr8r#r rr~sQ $O$O/2$O t&'$.c3> ?$O $Or#r)fifo prefill_first)rabcrr collectionsr utils.metricsrrrr requestsr r r rrrrSCHEDULER_MAPPINGr8r#r rs}#2&MMl l ` 1OI1O1Ol*OI*O*O\ *r#