ZjSrSSKrSSKJrJrJrJrJr SSKJ r SSK J r SSK J r \ R"\5rSS S .r"S S \ 5rS /rg) z Tokenization class for model T5.N) Tokenizerdecoders normalizerspre_tokenizers processors)Unigram)TokenizersBackend)loggingz spiece.modelztokenizer.json) vocab_filetokenizer_filec~^\rSrSrSr\rSS/r\r S S\ \ \ \ \ 4-S-4U4SjjjrSrS rS rU=r$) T5TokenizeraN Construct a T5 tokenizer (backed by HuggingFace's *tokenizers* library). Based on [Unigram](https://huggingface.co/docs/tokenizers/python/latest/components.html?highlight=unigram#models). This tokenizer inherits from [`TokenizersBackend`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: vocab_file (`str`, *optional*): [SentencePiece](https://github.com/google/sentencepiece) file (generally has a *.spm* extension) that contains the vocabulary necessary to instantiate a tokenizer. eos_token (`str`, *optional*, defaults to `""`): The end of sequence token. When building a sequence using special tokens, this is not the token that is used for the end of sequence. The token used is the `sep_token`. unk_token (`str`, *optional*, defaults to `""`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. pad_token (`str`, *optional*, defaults to `""`): The token used for padding, for example when batching sequences of different lengths. extra_ids (`int`, *optional*, defaults to 100): Add a number of extra ids added to the vocabulary for use as sentinels. These tokens are accessible as "" where "{%d}" is a number between 0 and extra_ids-1. These tokens can be retrieved by calling get_sentinel_tokens method and token ids can be by calling get_sentinel_token_ids method additional_special_tokens (`list[str]`, *optional*): Additional special tokens used by the tokenizer. vocab (`str`, `dict` or `list`, *optional*): Custom vocabulary dict. If not provided, a minimal vocabulary is created using the special tokens. input_idsattention_maskNvocabc d>X`lUb~UV s/sHn S[U 5;dMU PM n n [U 5S:a$U[U5V s/sH n SU S3PM sn - nOIUS:a!U[U 5:wa[ SUSUS35eO![U5V s/sH n SU S3PM n n U nUbXlOd[U5S4[U5S4[U5S4S /Ul[US- S S 5H$n UR R SU S3S45 M& [[UR S S S 95Ul Ub%[R"U5URl [R"[R"5[R "SSSS9/5URl[$R "SSSS9URl[(T U]T"SUUUUUS.UD6 [,R."SS//SQSUR04/S9URlgs sn fs sn fs sn f)Nz rzBoth extra_ids (z!) and additional_special_tokens (zk) are provided to T5Tokenizer. In this case the additional_special_tokens must include the extra_ids tokensg)▁gF)unk_id byte_fallbackralwaysT) replacementprepend_schemesplit) eos_token unk_token pad_token extra_idsadditional_special_tokens$A)r%r&z$Br&)singlepairspecial_tokens) _extra_idsstrlenrange ValueError _vocab_scoresappendrr _tokenizerr Precompiled normalizerrSequenceWhitespaceSplit Metaspace pre_tokenizerrdecodersuper__init__rTemplateProcessing eos_token_idpost_processor) selfrr r!r"_spm_precompiled_charsmapr#r$kwargsx extra_tokensi __class__s w/root/GenerationalWealth/GenerationalWealth/venv/lib/python3.13/site-packages/transformers/models/t5/tokenization_t5.pyr;T5Tokenizer.__init__Gsa$ % 0'@['@!LTWXYTZDZA'@L[< 1$)yIY-ZIYA 1#Q.?IY-ZZ)Q9L0A#A &yk1RSlRmn 8=Y7GH7G!j1-7GLH(4 %  !& Y%Y%Y% "D  9q="b1""))Zs!+-**+* &m\-ZIsH#H# H(H-cT[[[SUR555$)zQGet the list of sentinel tokens (extra_id tokens) from additional_special_tokens.cF[[R"SU55SL$)Nz)boolresearch)rBs rF1T5Tokenizer.get_sentinel_tokens..sbii0A1&E!Fd!R)listsetfilterr$)r?s rFget_sentinel_tokensT5Tokenizer.get_sentinel_tokenss& RTXTrTrs t  rOcjUR5Vs/sHoRU5PM sn$s snf)z&Get the token IDs for sentinel tokens.)rSconvert_tokens_to_ids)r?tokens rFget_sentinel_token_ids"T5Tokenizer.get_sentinel_token_idss.?C?W?W?YZ?Ye**51?YZZZs0)r+r2r0)Nr&zzNdN)__name__ __module__ __qualname____firstlineno____doc__VOCAB_FILES_NAMESvocab_files_namesmodel_input_namesrmodelr,rPtuplefloatr;rSrX__static_attributes__ __classcell__)rEs@rFrrsx"H*$&67 E7;"&"&K T%U +,,t3K K Z [[rOr)r_rK tokenizersrrrrrtokenizers.modelsrtokenization_utils_tokenizersr utilsr get_loggerr[loggerr`r__all__r*rOrFrosU' SS%>   H %#1EUV~[#~[B /rO