ZjNSrSSKrSSKrSSKrSSKrSSKrSSKJrJr SSK J r SSK J r SSK Jr SSKJr SSKJr \(a2SSKrSSKrSSKrSS KJrJrJrJrJr SS KJr SS KJr SS K J!r! S SK"J#r# \RH"\%5r&Sr'"SS\RP5r)"SS5r*"SS\+5r,"SS\-5r.SSSSSSSS.S .S!.0r/S"S#S$\0S-4S%jr1S&\0S$\04S'jr2S(\0S$\3\0S-4S)jr4"S*S+5r5S,\ S-\6S$\74S.jr8"S/S05r9"S1S25r:S3\;S$S4S4jr"S8S9\5r?"S:S;\?5r@"S<S=\?5rA"S>S?5rB"S@SA5rCg)Cz? Shared types, constants, and utilities for the serving layer. N)ABCabstractmethod)Callable)Future)Queue) TYPE_CHECKING)logging)ContinuousBatchingConfigGenerationConfigPreTrainedModelPreTrainedTokenizerFastProcessorMixin)ContinuousBatchingManager)GenerationOutput) Scheduler) ModelManagerz x-request-idc(\rSrSrSrSrSrSrSrSr g) Modality9LLMVLM MULTIMODALSTTTTSN) __name__ __module__ __qualname____firstlineno__rrrrr__static_attributes__ro/root/GenerationalWealth/GenerationalWealth/venv/lib/python3.13/site-packages/transformers/cli/serving/utils.pyrr9s C CJ C Cr"rc&\rSrSrSrS\4SjrSrg) _StreamErrorAz5Sentinel to signal an error from the generate thread.msgcXlgNr')selfr's r#__init___StreamError.__init__Dsr"r*N)rrrr __doc__strr,r!rr"r#r%r%As?Cr"r%c\rSrSrSrSrg)_GenerationCancelledHzERaised inside ``DirectStreamer.put()`` to abort ``model.generate()``.rNrrrr r.r!rr"r#r1r1HsOr"r1c\rSrSrSrSrg)CBWorkerDeadErrorLzRaised when a request is submitted to a CB worker that has died. Surfaced as 503 by the FastAPI exception handler. Carries the original error message that killed the worker so the client knows why the server is in this state. rNr3rr"r#r5r5Lsr"r5qwenz z z(.*?)arrayobjectjson)typezx-parser)zx-regex-iteratorr;items)stcetcschemamodelr returncj^[USU5n[USS5n[USS5n[USS5nU(aU(aU(a USSnO:[U4Sj[R55S5nUcgUS US US pdnUR U5nUR U5n XhU S .$) aRReturn tool call config for the model, or ``None`` if tool calls are not supported. Returns a dict with: - ``schema`` (`dict`): Schema to pass to ``tokenizer.parse_response(block, schema)``. - ``stc_id`` (`int`): Token ID of the start-of-tool-call delimiter. - ``etc_id`` (`int`): Token ID of the end-of-tool-call delimiter. tokenizer stc_tokenN etc_tokenresponse_schema properties tool_callsc3d># UH%upUTRR;dM!Uv M' g7fr))config model_type).0kvr@s r# 'get_tool_call_config..ws(d'CtqqELLLcLcGc'Cs 0 0r=r>r?)r?stc_idetc_id)getattrnext_TOOL_CALL_FALLBACKSr<convert_tokens_to_ids) processorr@rCr=r>rFr?fallbackrQrRs ` r#get_tool_call_configrYes ; :I )[$ /C )[$ /Ci):DAO s .|<d';'A'A'Cdfjk  #E?HUOXh=O&  , ,S 1F  , ,S 1F& AAr" tool_callcURSU5nURS05nUS[U[5(d[R"U5S.$US.$)aNormalize a parsed tool call to ``{"name": str, "arguments": str}``. Different models return different structures from ``parse_response``: - Gemma: ``{"function": {"name": ..., "arguments": {...}}}`` (nested, arguments as dict) - Qwen: ``{"name": ..., "arguments": {...}}`` (flat, arguments as dict) The OpenAI API expects ``arguments`` as a JSON **string**, so we ``json.dumps`` it. function argumentsname)r^r])get isinstancer/r:dumps)rZr\r]s r#_normalize_tool_callrbs[}}Z3H ["-I 2``). schema: The tool call schema (from ``response_schema`` or ``_TOOL_CALL_FALLBACKS``). Returns a list of ``{"name": str, "arguments": str}`` dicts, or ``None`` if none found. N)parse_responser`listrb)rW generated_idsr?parsedrZrHs r#parse_tool_callsrhs[ % %m Update a bar's current byte count and emit aggregate progress.Nru)r+rrryrss r#updateDownloadAggregator.updates$, & r"cgr)r)r+rrs r#closeDownloadAggregator.close r"cb[SURR555nXR:XagXlURR5VVs/sH up#UcM UPM nnnU(a [U5OSnUR SUR SXS.S.5 gs snnf)Nc3*# UH upUv M g7fr)r)rLc_s r#rO+DownloadAggregator._emit..s;(:!(:loadingdownloadryrsstatusr@stageprogress)sumrovaluesrprlr@)r+ agg_currentrttotals agg_totals r#rvDownloadAggregator._emits; (8(8(:;; 33 3 $/! $ 0 0 2D 2a! 2D#)CKt  ##(3H   Es  B++B+)rorlrpr@rAN)rrrr r.rr/r,intrwr{r~rvr!rr"r#rjrjsu 55C5 s3:$ S3sTzd  C D  r"rjcallbackrmcP^^^SSKJn [TT5m"UUU4SjSU5nU$)uCreate a tqdm subclass that routes progress to a callback. Bars with ``unit="B"`` are download bars — aggregated via ``DownloadAggregator``. Other bars (e.g. "Loading weights") emit ``weights`` stage events. Args: callback (`callable`): Called with a dict payload ``{"status": "loading", "model": ..., "stage": ..., "progress": ...}``. model_id (`str`): The model ID (included in progress payloads). Returns: A tqdm subclass that forwards progress to *callback*. r)tqdmc^>^\rSrSrUU4SjrSUUU4SjjrUUU4SjrUU4SjrSrU=r $).make_progress_tqdm_class..ProgressTqdmc>URS5=(d SUlSUS'[TU] "U0UD6 SUlSUlURS:Xa7[ U5UlTRURUR5 gg)NunititTdisablerB) r_sse_unitsuperr,n last_emittedid_bar_idrwrs)r+argskwargs __class__download_aggregators r#r,7make_progress_tqdm_class..ProgressTqdm.__init__sw"JJv.6$DM $F9  G d -f -DF "D }}#!$x #,,T\\4::F$r"c^>UcSnU=RU- slURS:Xa2TRURURUR5 gURUR :wa6URUlT"STSURURS.S.5 ggNrrrweightsrr)rrr{rrsr)r+rrrrms r#r{5make_progress_tqdm_class..ProgressTqdm.updatesy FFaKF}}##**4<<L4,,,$(FF!"+!)!*04$L -r"c 3># URHnU=RS- slURS:Xa2TRURURUR 5 OOURUR :wa5URUlT"STSURUR S.S.5 Uv M g7fr)iterablerrr{rrsr)r+itemrrrms r#__iter__7make_progress_tqdm_class..ProgressTqdm.__iter__s ! ==C''..t||TVVTZZPVVt000(,D%&/%-%.48FFTZZ(P  &sB?Ccx>URS:XaTRUR5 [TU]5 g)Nr)rr~rr)r+rrs r#r~4make_progress_tqdm_class..ProgressTqdm.closes*}}##))$,,7 GMOr")rrrr)r) rrrr r,r{rr~r! __classcell__)rrrrms@r# ProgressTqdmrs" G  " "  r"r) tqdm.autorrj)rrm base_tqdmrrs`` @r#make_progress_tqdm_classrs/,,Xx@00y0d r"c \rSrSrSrSSSS\R S\RS\S \ S-4 S jjr SS jr SS jr SS jr Srg)DirectStreameriarStreamer for ``model.generate()`` (used by :class:`GenerateManager`). Implements the ``put``/``end`` protocol that ``model.generate()`` expects: generate calls ``put(token_tensor)`` after each decode step, and ``end()`` when generation is complete. Tokens are decoded incrementally via the Rust ``DecodeStream`` (O(1) per token) and pushed as text to an asyncio.Queue. NrCtokenizers.Tokenizerloopqueueskip_special_tokens tool_configcSSKJn XlX lX0lU"/U5UlU(aUSOSUlU(aUSOSUlSUlSUl [R"5Ul SUl /Ulg)aK Args: tokenizer: The Rust tokenizer (``tokenizer._tokenizer``). loop (`asyncio.AbstractEventLoop`): The event loop to push decoded text to. queue (`asyncio.Queue`): The queue that receives decoded text chunks. skip_special_tokens (`bool`, *optional*, defaults to `True`): Whether to strip special tokens during decoding. tool_config (`dict`, *optional*): Tool call config from ``get_tool_call_config``. When set, tokens between stc/etc delimiters (inclusive) are suppressed from the queue so tool call markup is never streamed to the client. r DecodeStreamrQNrRFT)tokenizers.decodersr _tokenizer_loop_queue_decode_stream_stc_id_etc_id_inside_tool_call_first threadingEvent _cancelled total_tokensgenerated_token_ids)r+rCrrrrrs r#r,DirectStreamer.__init__"ss& 5#  *2/BC0;{8, 0;{8, !& #//+.0 r"cnURR5(a [5eUR(aSUlgUR 5HnU=R S- slUR RU5 X R:XaSUl OX R:XaSUl URRURU5nUcMUR(aMX R:wdMURRUR R"U5 M g)zHCalled by ``model.generate()`` after each decode step with new token(s).FNrT)ris_setr1rtolistrrappendrrrrsteprrcall_soon_threadsafer put_nowait)r+valuetoken_idtexts r#putDirectStreamer.putCs ?? ! ! # #&( ( ;;DK  H    "   $ $ + +H 5<<')-&\\)).&&&++DOOXFD(>(>(>8||C[ // 0F0FM'r"cdURRURRS5 g)z;Called by ``model.generate()`` when generation is complete.N)rrrrr+s r#endDirectStreamer.endXs  '' (>(>Er"c8URR5 g)zWSignal cancellation. The next ``put()`` call will raise and abort ``model.generate()``.N)rsetrs r#cancelDirectStreamer.cancel\s r") rrrrrrrrrrr)TN)r torch.TensorrANr)rrrr r.asyncioAbstractEventLooprbooldictr,rrrr!rr"r#rrsd%)#' 1)1''1}} 1 " 1 D[ 1BN*Fr"rc\rSrSrSrSSSS\SSS \RS \RS \ S-4 S jjr SS jr SSjr SSjr Srg) CBStreameriaapStreamer for continuous batching (used by :class:`CBGenerateManager`). Same ``put``/``end`` protocol as :class:`DirectStreamer`, but called manually by :class:`CBGenerateManager` instead of by ``model.generate()``: ``put(output)`` receives a CB ``GenerationOutput``, decodes new tokens, and pushes text to the asyncio.Queue. ``end()`` signals the stream is complete. N cb_managerr request_idrCrrrrcSSKJn XlX lX@lXPlX0lU"/S5UlU(aUSOSUlU(aUSOSUl SUl SUl SUl /Ul g)a Args: cb_manager (`ContinuousBatchingManager`): The CB manager instance. request_id (`str`): The request ID to track in the CB scheduler. tokenizer: The Rust tokenizer (``tokenizer._tokenizer``). loop (`asyncio.AbstractEventLoop`): The event loop to push decoded text to. queue (`asyncio.Queue`): The queue that receives decoded text chunks. tool_config (`dict`, *optional*): Tool call config (see ``DirectStreamer``). rrTrQNrRF)rr_cb _request_idrrrrrrr _prev_lenrr)r+rrrCrrrrs r#r,CBStreamer.__init__jso$ 5%  #*2t40;{8, 0;{8, !&.0 r"c URURSn[UR5UlUHnU=RS- slURR U5 X0R :XaSUlOX0R:XaSUlURRURU5nUcMUR(aMX0R:wdMURRU5 M g)zLDecode new tokens from a CB ``GenerationOutput`` and push text to the queue.NrTF)generated_tokensrlenrrrrrrrrrrr)r+output new_tokensrrs r#rCBStreamer.puts,,T^^-=> V445"H    "   $ $ + +H 5<<')-&\\)).&&&++DOOXFD(>(>(>8||C[ &&t,#r"c:URRS5 g)zSignal end of stream.N)rrrs r#rCBStreamer.ends t$r"cNURRUR5 g)zCancel the CB request.N)rcancel_requestrrs r#rCBStreamer.cancels  0 01r") rrrrrrrrrrrrr))rrrANr)rrrr r.r/rrrrr,rrrr!rr"r#rrasi$(1/11* 1 '' 1 }} 1D[1B-"%2r"rseedc0SSKnUR"U5 g)z8Set the PyTorch random seed for reproducible generation.rN)torch manual_seed)rrs r#set_torch_seedrs dr"cSSKnURR5(aURR5 gg)z+Empty the CUDA cache if a GPU is available.rN)rcuda is_available empty_cache)rs r#reset_torch_cacher s- zz    !r"cX\rSrSrSrSrS SjrS\4SjrS\ R4Sjr S r g) InferenceThreadizPersistent thread for ``model.generate()`` calls. ``torch.compile`` with CUDA graphs stores state in thread-local storage. All inference must run on the same thread to avoid corrupted graph state. c[5Ul[R"URSS9UlUR R 5 g)NT)targetdaemon)rrrThread_run_threadstartrs r#r,InferenceThread.__init__s3"W  ''tyyF  r"rANcRURR5upp4nU"U0UD6nUbURURU5 OURU5 M[![a?nUb!URUR U5 SnAN0UR U5 SnANFSnAff=fr))rr_r set_result Exception set_exception)r+fnrrfuturerresultes r#rInferenceThread._runs-1[[__-> *Bfd ,T,V,#--f.?.?H%%f- ,#--f.B.BAFF((++  ,s#'A A B&'B! B!!B&cV[5nURRXX4S45 U$)ESubmit a callable to the inference thread. Returns a blocking Future.N)rrr)r+rrrrs r#submitInferenceThread.submits% 6489 r"c[R"5nUR5nURR XX5U45 U$zOSubmit a callable to the inference thread. Returns an awaitable asyncio.Future.)rget_running_loop create_futurerr)r+rrrrrs r# async_submitInferenceThread.async_submits:'')##% 6489 r")rrr) rrrr r.r,rrr#rr)r!rr"r#rrs-  ,V 7>>r"rc\rSrSrSrSS jr\SSSS S S \SSS \S\S-S\ \ RS44Sjj5r \SSS S S \SSS \S\ \\ \\ 44 Sj5r\SSj5rSrg)BaseGenerateManageriuBase class for generation managers. Subclasses: - :class:`GenerateManager` — sequential ``model.generate()`` on a persistent thread. - :class:`CBGenerateManager` — continuous batching with paged attention. r@r gen_configr rANcg)z:Initialize continuous batching. No-op for non-CB managers.Nrr+r@r-s r#init_cbBaseGenerateManager.init_cbr"rW(ProcessorMixin | PreTrainedTokenizerFastinputsrrzDirectStreamer | CBStreamercg)aStart streaming generation. Args: model (`PreTrainedModel`): The loaded model. processor: The processor or tokenizer for decoding. inputs (`dict`): Tokenized inputs (tensors for sequential, lists for CB). gen_config (`GenerationConfig`): Generation parameters. request_id (`str`): Unique request identifier. tool_config (`dict`, *optional*): Tool call config from ``get_tool_call_config``. When set, tool call tokens (between stc/etc) are suppressed from output. Returns: `tuple[asyncio.Queue, DirectStreamer | CBStreamer]`: A ``(queue, streamer)`` pair where *queue* yields ``str | _StreamError | None`` and *streamer* exposes ``.total_tokens`` and ``.cancel()``. Nr)r+r@rWr4r-rrs r#generate_streaming&BaseGenerateManager.generate_streamingr2r"c # g7f)aRun generation to completion. Args: model (`PreTrainedModel`): The loaded model. processor: The processor or tokenizer for decoding. inputs (`dict`): Tokenized inputs (tensors for sequential, lists for CB). gen_config (`GenerationConfig`): Generation parameters. request_id (`str`): Unique request identifier. Returns: `tuple[str, int, list[int]]`: ``(text, input_len, generated_ids)``. Nr)r+r@rWr4r-rs r#generate_non_streaming*BaseGenerateManager.generate_non_streamingsscg)z/Stop the generation manager and free resources.Nrrs r#stopBaseGenerateManager.stopr2r"rr@r r-r rANr)r)rrrr r.r0rrr/tuplerrr6rrer9r<r!rr"r#r,r,sI$(   >   '    D[  w}};; <  4   >   '     sCc" #  *>>r"r,c\rSrSrSrSrSSSSSS \S S S \S \S-S\\ R\ 44Sjjr SSSSS \S S S \S\\\ S44 SjrS\S\4SjrS\S\ R 4SjrSSjrSrg)GenerateManagerizFSequential generation via ``model.generate()`` on a persistent thread.c"[5Ulgr))rrrs r#r,GenerateManager.__init__!s &( r"Nr@r rWr3r4r-r rrrAc"^^ ^ ^ [R"5m [R"5m [USU5Rn[ UT T US9n0UEXUS.Em [ TS5(aST S'SU U UU 4Sjjn URU 5 T U4$) zLStart streaming generation via ``model.generate()`` on the inference thread.rCr)streamergeneration_configrC has_talkerrgeneration_modec >TR"S0TD6 g![a TRTRS5 g[a8nTRTR[ [ U555 SnAgSnAff=f)Nr)generater1rrrr%r/)r gen_kwargsrr@rs r#r0GenerateManager.generate_streaming.._run7sk R,,' B))%*:*:DA R))%*:*:LQ  '   D[ w}}n, -<. .>. . ' .  . sC' (.(88v8>x>W^^> r"rAc\rSrSrSrSSSjjrSS jrS \4S jrS \ S S4S jr SSSSSS\ SSS \ S\ S-S \ \ R\44SjjrSSSSS\ SSS \ S \ \ \\\44 Sjr\SSj5rSSjrSrg)CBGenerateManageribaQContinuous batching generation via paged attention. Translates between the handler's text-level asyncio.Queue and CB's token-level interface. Per-request: ``max_new_tokens``, ``eos_token_id``. The CB manager is initialized lazily on the first request via :meth:`ensure_initialized`, using that request's ``gen_config`` for shared sampling params (temperature, top_p, do_sample). .. todo:: Remove :meth:`init_cb` when CB supports per-request generation config. At that point, ``gen_config`` can be passed directly to ``add_request`` and the CB manager no longer needs a shared config. NcSUlXlgr)r _cb_config)r+ cb_configs r#r,CBGenerateManager.__init__qs#r"r@r r-r rAcURbgURX RS9UlURR5 g)aLInitialize the CB manager on first call with the request's generation config. .. todo:: Remove when CB supports per-request generation config. Args: model (`PreTrainedModel`): The loaded model (must support ``init_continuous_batching``). gen_config (`GenerationConfig`): Generation config used for shared sampling params. N)rGcontinuous_batching_config)rinit_continuous_batchingrerr/s r#r0CBGenerateManager.init_cbus? 88  11(__2  r"c\URSL=(d URRSL$)zJWhether the CB worker is healthy. ``True`` before ``init_cb()`` is called.N)r fatal_errorrs r#is_aliveCBGenerateManager.is_alives%xx4?488#7#74#??r"rcURb=URRb%[SUSURR35egg)uRaise :class:`CBWorkerDeadError` if the CB worker has died. Called at request entry to fail fast — submitting to a dead worker would otherwise enqueue the request into a void where it never gets processed. Nz,CB worker is dead and cannot accept request : )rrmr5)r+rs r# _check_aliveCBGenerateManager._check_alivesN 88 DHH$8$8$D#>zl"TXXMaMaLbc %E r"rWr3r4rc ^ ^ URnUc [S5eURU5 [R"5n[R "5m USn UR U USURURS9n[USU5Rn [URXZUT US9m U U 4Sjn URX[5 T T 4$)zFStart streaming CB generation. Registers a per-request output handler.3CB manager not initialized. Call `init_cb()` first.rRT)r streamingmax_new_tokens eos_token_idrCrEcn>TRU5 URb5TR[UR55 TR 5 gUR 5(aTR 5 gg![ a-nTR[[U555 SnAgSnAff=fr))rerrorrr%r is_finishedrr/)rrrF text_queues r# _on_output8CBGenerateManager.generate_streaming.._on_outputs < V$<<+)),v||*DELLN''))LLN* <%%l3q6&:;; TR5(dTRU5 ggr))doner)rrs r# _on_result._on_results!;;==!!&)!r"F)rrwrvrxzCB worker died during request rqzCB generation failed for TrS)rrrrrrr'r(rrrwrxrzrmr5rrU)r+r@rWr4r-rrrRrXrrrrfrrs @r#r9(CBGenerateManager.generate_non_streamings0XX :TU U *%;'  N '')##% * ""::  !%44#00   << #~~)'*H TVW]WcWcVd(eff!::,bWX X//  4H --sB$D'D(A5DcrURc [S5eURRR$)z*The CB scheduler (for testing/monitoring).zCB manager not initialized.)rrbatch_processor schedulerrs r#rCBGenerateManager.schedulers0 88 <= =xx''111r"cTURbURRSSS9 gg)NT)blocktimeout)rr<rs r#r<CBGenerateManager.stops% 88  HHMMaM 0 r"rdr))rfContinuousBatchingConfig | Noner>)rArr)rrrr r.r,r0rrnr/rrrr?rrrr6rrer9propertyrr<r!rr"r#rbrbbs $"@$@ s t $$(/$ /$>/$ /$ ' /$  /$D[/$ w}}j( )/$b/. /.>/. /. ' /.  /. sCc" #/.b22 1r"rbc\rSrSrSrSS\S\SS4SjjrS S S \S \4S jrSS\ S\S \ 4Sjjr SSjr S \4Sjr Srg)GenerationStateiaShared generation state across all handlers. Manages per-model :class:`GenerateManager` instances (each with its own :class:`InferenceThread` so different models can run concurrently while ``torch.compile`` / CUDA graphs require same-model-same-thread) and a single :class:`CBGenerateManager` for continuous batching. Args: continuous_batching (`bool`, *optional*, defaults to `False`): Whether to use continuous batching with paged attention instead of sequential ``model.generate()`` calls. Ncontinuous_batchingcompilerfrcRXlX lX0l0UlSUlSUlgr))_continuous_batching_compilere_generate_managers _cb_manager _cb_model_id)r+rrrfs r#r,GenerationState.__init__s, %8! #>@59(,r"r@r modalityrAcUR(dg[US5=(a U[R:HnU(d,[R UR RS35 U$)a'Check if continuous batching can be used for this model and modality. Args: model (`PreTrainedModel`): The loaded model. modality (`Modality`): The detected model modality (LLM, VLM, etc.). Returns: `bool`: ``True`` if CB is enabled and the model supports it, ``False`` otherwise. FrjzM does not support continuous batching. Falling back to sequential generation.)rrNrrlogger warning_oncerr)r+r@rcans r#use_continuous_batching'GenerationState.use_continuous_batchings]((e78UX=U   ??++,-99  r"rmuse_cbcdU(auURU:wa.URb!URR5 SUlURc[URS9UlXlUR$XR ;a[ 5UR U'UR U$)a6Return a per-model generation manager, lazily created on first request. Args: model_id (`str`): The model ID in ``'model_id@revision'`` format. use_cb (`bool`): Whether to return a CB manager or a sequential one. Returns: `BaseGenerateManager`: Either a `GenerateManager` or `CBGenerateManager`. N)rf)rrr<rbrerrA)r+rmrs r# get_managerGenerationState.get_manager2s   H,##/$$))+'+D$'#4t#O $,!## # 22 20?0AD # #H -&&x00r"cbURb"URR5 SUlgg)z$Stop any active generation managers.N)rr<rs r#shutdownGenerationState.shutdownIs-    '    ! ! ##D  (r"c`URSL=(d URR5$)zTWhether the CB worker is healthy. ``True`` if CB is disabled or not yet initialized.N)rrnrs r# is_cb_aliveGenerationState.is_cb_aliveOs'4'F4+;+;+D+D+FFr")rerrrrr)FFNFr)rrrr r.rr,rrr/r,rrrr!rr"r#rrs} %*7; -! - -5 -->(W[(1C11BU1.$ GTGr"rc \rSrSr%SrSr\S-\S'\"5r \\ \S'SSS\ 4S jr S \ S S4S jr\S SS \ 4Sj5rS \ S \\ SS44SjrSS \ SSS\S S4Sjjr\S\\ S\S \\ 4Sj5rSrg) BaseHandleriTaShared logic for chat completion and responses handlers. Provides model resolution, generation config building, and SSE formatting. Generation is delegated to the shared :class:`GenerationState`. Args: model_manager (`ModelManager`): Handles model loading, caching, and lifecycle. generation_state (`GenerationState`): Shared state managing per-model generation managers. N_valid_params_class_unused_fields model_managerrgeneration_statecXlX lgr))rr)r+rrs r#r,BaseHandler.__init__ds + 0r"bodyrAcSSKJn [UR55nURb3U[ URS[55- nU(a U"SSU3S9eX0R -nU(a[RSU35 gg) zMValidate request fields against the handler's params class and unused fields.r HTTPExceptionN__mutable_keys__iz"Unexpected fields in the request:  status_codedetailz,Ignoring unsupported fields in the request: ) fastapirrkeysrrSrrr)r+rr input_keys unexpectedunuseds r#_validate_requestBaseHandler._validate_requestls)%  # # /#gd.F.FHZ\_\a&bbJ#>`ak`lW3X Y  $ .E F  //7;L;[;[^b;b37!0 88M " .,1$}2E,F  )T-()S0.3!+ 88G  (&+DM&:  # 88F  ' 4< (  ) ).?.T.T.\5=  2 */  '! r"messagesrc v/nUGH/nUS/S.nSU;aUSUS'SU;aUSUS'SU;a/OURS5=(d /n[U[5(aSUS./nUGHnUSnUS ;aUSRSUSS.5 M+US ;a\U[R [R 4;a8US n[U[5(aUS nUSRS US.5 MUS:XafU[R :XaRUSn [U [5(aU RSS5OSn U Sn USRSSU SU 3S.5 MUS:XaDU[R [R 4;a USRSUSS S.5 GMCUS:XdGMLU[R :XdGMcUSRSUSS S.5 GM U[R:XaSRSUS55US'URU5 GM2 U$)aConvert OpenAI-format messages to the format expected by HF processors. All modalities extract text. VLM additionally handles ``image_url`` and ``video_url``. MULTIMODAL handles all of the above plus ``input_audio`` and ``audio_url``. For LLMs, the content parts are collapsed into a plain text string. Args: messages (`list[dict]`): OpenAI-format chat messages. modality (`Modality`): The model modality (LLM, VLM, or MULTIMODAL). Returns: `list[dict]`: Processor-compatible messages. role)rcontentrH tool_call_idrr)r;rr;)r input_text output_text) image_url input_imagerurlimage)r;r input_audioformatwavdataaudioz data:audio/z;base64, video_urlvideo audio_url c3*# UH oSv M g7f)rNr)rLrs r#rOABaseHandler.get_processor_inputs_from_messages..s,R@Q1vY@Qr) r_r`r/rrrrrrjoin) rrprocessor_inputsmessagerg raw_contentr content_typerrfmt audio_b64s r#"get_processor_inputs_from_messages.BaseHandler.get_processor_inputs_from_messagessOG%fo"=Fw&'.|'<|$()0)@~&!- 7"gkk)>T>ZXZK+s++(. DE &&v #HH9%,,fgfo-VW!%AAhS[S_S_aiatatRuFu!+.C!#t,,!%j9%,,gc-JK!]2x8CVCV7V")-"8K>HVZ>[>[+//(E:afC +F 3I9%,,g TWSXX`aj`kFl-mn![0X(,,PXPcPcAd5d9%,,ggkFZ[`Fa-bc![0XATAT5T9%,,ggkFZ[`Fa-bc-'28<<'$'HH,Ry@Q,R$Ry!  # #F +W X r")rrr)rrrr r.rr;__annotations__rrr/rr,rr staticmethodrr?rrrrerrr!rr"r#rrTs (,+"uNCH$1%1*1 Yd Yt YG6G3GG *4*E#7HJt2t,u*,W\0!0!3E0!OS0! 0!d< T$Z< 8< X\]aXb< < r"rr)Dr.rrenumr:rabcrrcollections.abcrconcurrent.futuresrrrtypingrtransformers.utilsr pydantic tokenizersrrr r r r r:transformers.generation.continuous_batching.continuous_apir4transformers.generation.continuous_batching.requestsr5transformers.generation.continuous_batching.schedulerrrr get_loggerrr X_REQUEST_IDEnumrr%rr1rr5rUrrYrbrerhrjr/r;rrrrrr rr,rArbrrrr"r#rs #$% &eUO+   H % tyyP9P   ?&F;   B+<BB8DT".t.T T@Q.*( ( VDxD3D4DNEEPA2A2H!&&R>>#>>BA )A H^1+^1BNGNGbp p r"