ZjZ<SrSSKrSSKrSSKrSSKJr SSKJrJr SSK J r \ R"\ 5r SS0r"S S 5r"S S \5rS /rg) z"Tokenization class for model MyT5.N) defaultdict) AddedTokenPreTrainedTokenizer)logging vocab_filezbyte_maps.jsonc\rSrSrSrSrS\\\\4-4SjrS\\\\ \-4S\S\4S jr S\\\4S \\\\ \-44S jr S \ \S S \ \-4Sjr SS\ \S \ \4Sjjr Srg ) ByteRewriteraB Byte rewriter class for MyT5 tokenizer. This class is used to rewrite bytes using a hash tree. The hash tree is constructed from a set of rewriting rules. Args: rewriting_rules (`str` or `dict[str, str]`): A path to a json file containing the rewriting rules or a dictionary containing the rewriting rules. z[LEAF]rewriting_rulesc[U[5(a,[US5n[R"U5nSSS5 O,[U[ 5(d[ S[U535eURU5Ul UR5VVs0sHup4XC_M nnnURU5Ul g!,(df  N_=fs snnf)NrzDrewriting_rules should be either a path to json file or a dict, got ) isinstancestropenjsonloaddict TypeErrortypeconstruct_hash_tree hash_treeitemsreverse_hash_tree)selfr fkvreverse_rewriting_ruless {/root/GenerationalWealth/GenerationalWealth/venv/lib/python3.13/site-packages/transformers/models/myt5/tokenization_myt5.py__init__ByteRewriter.__init__,s os + +os+q"&))A,,+OT22VW[\kWlVmn 11/B4C4I4I4K"L4KDA144K"L!%!9!9:Q!R,+#MsB?C? C rbyte_in_sequencebyte_out_sequencecURS5nURS5nUnUHnXv;a0Xg'XgnM XVUR'g)z< Add a leaf with the output byte sequence to the hash tree.  N)splitLEAF)rrr#r$ byte_in_list byte_out_list tree_pointerbs r add_leafByteRewriter.add_leaf9sU(--c2 )//4  A$"$ '?L #0TYYreturnc[[5nS[S55HnU/X#UR'M UR 5HupEUR X$U5 M U$)z5 Construct a hash tree for rewritten byte sequences. c3(# UHoSv M g7f)02xN).0xs r 3ByteRewriter.construct_hash_tree..Ms1jsG*js)rrranger(rr-)rr rr, in_sequence out_sequences r r ByteRewriter.construct_hash_treeHsb % 1eCj1A'(cIL #2*9)>)>)@ %K MM), ?*Ar/ byte_sequenceNc^URnUHnX2;aX#nM g X R$)zG Search the hash tree and return the rewritten byte sequence if found. N)rr()rr>r+r,s r search_hash_treeByteRewriter.search_hash_treeUs6~~ A +   II&&r/in_bytescp/nSnSnU[U5:aU(d URO URn[U[U55HAnXnX;aXhnOXt:XaU/n Un O( O&URU;dM1X`Rn UnMC UR W 5 US-nU[U5:aMU$)z Rewrite a sequence of bytes using the hash tree. Args: in_bytes (`list[str]`): A list of bytes to be rewritten. reverse (`bool`): If True, decoding is performed with the reverse hash tree. Returns: `list[str]`: The rewritten byte sequence. r)lenrrr:r(extend) rrBreverse out_bytesb_startb_endr+jr,cur_leafs r rewrite_bytesByteRewriter.rewrite_bytesbs H %184>>d>T>TL7CM2K$#/?L\ !sHE99 ,+II6HE3   X &aiG!H %$r/)rr)F)__name__ __module__ __qualname____firstlineno____doc__r(rrr!listr-rr@rM__static_attributes__r4r/r r r s D Sd38n(< S 0$sD49,<'<"= 0QT 0il 0 4S> d3PTW[\_W`P`K`Fa  'd3i 'D49"`): The end of sequence token. unk_token (`str`, *optional*, defaults to `""`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. pad_token (`str`, *optional*, defaults to `""`): The token used for padding, for example when batching sequences of different lengths. extra_ids (`int`, *optional*, defaults to 125): Add a number of extra ids added to the end of the vocabulary for use as sentinels. These tokens are accessible as "" where "{%d}" is a number between 0 and extra_ids-1. Extra tokens are indexed from the end of the vocabulary up to beginning ("" is the last token in the vocabulary like in ByT5 preprocessing see [here](https://github.com/google-research/text-to-text-transfer-transformer/blob/9fd7b14a769417be33bc6c850f9598764913c833/t5/data/preprocessors.py#L2117)). additional_special_tokens (`list[str]`, *optional*): Additional special tokens used by the tokenizer. input_idsattention_maskNr0c >US:a#Uc [U5Vs/sH nSUS3PM nnONUS:aHUbE[U5S:a6[[[SU555n X:wa[ SUSUS35e[ U[ 5(a [USSS 9OUn[ U[ 5(a [USSS 9OUn[ U[ 5(a [USSS 9OUnXBUS .Ul[UR5Ul S Ul [R"[US 55Ul[URS 5Ul[URS5Ul[$T U]L"SUUUSUS.UD6 gs snf)Nrz c0[S[U5;5$)Nextra_id)boolr)r6s r (MyT5Tokenizer.__init__..sDs1v9M4Nr/zBoth extra_ids (z!) and additional_special_tokens (zm) are provided to MyT5Tokenizer. In this case the additional_special_tokens must include the extra_ids tokensT)lstriprstrip)rrDr9r decompose_map merge_map) eos_token unk_token pad_token extra_idsadditional_special_tokensr4)r:rEsetfilter ValueErrorrrr_added_tokens_decoderoffset_utf_vocab_sizerrr byte_mapsr decompose_rewritermerge_rewritersuperr!) rrrgrhrirjrkkwargsi extra_tokens __class__s r r!MyT5Tokenizer.__init__s q=6>DI)DT(UDTq:aS):DT %(U % ]8DMfIgjkIks6*NPi#jklL( &yk1RSlRmn(( HRR[]`GaGaJydCgp GQR[]`GaGaJydCgp GQR[]`GaGaJydCgp )2Y%O"$445 #4 C#89".t~~o/N"O*4>>++FG  &?    3)VsFcUR$N)rq)rs r vocab_sizeMyT5Tokenizer.vocab_sizes###r/c[URUR-5Vs0sHoRU5U_M nnUR UR 5 U$s snfr|)r:r}rpconvert_ids_to_tokensupdateadded_tokens_encoder)rrwvocabs r get_vocabMyT5Tokenizer.get_vocabsX;@SWS^S^A^;_`;_a++A.1;_` T../ asA token_ids_0 token_ids_1already_has_special_tokensc>U(a[TU]XSS9$UcS/[U5-S/-$S/[U5-S/-S/[U5--S/-$)ad Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`list[int]`): List of IDs. token_ids_1 (`list[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `list[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. T)rrrrrD)ruget_special_tokens_maskrE)rrrrrys r r%MyT5Tokenizer.get_special_tokens_masksw$ &72']a3   C#k**qc1 1c+&&1#-!s;7G1GHA3NNr/ token_idsc[U5S:a9USUR:Xa&[R"SURS35 U$XR/-$)z.Do not add eos again if user already added it.rzThis sequence already has zQ. In future versions this behavior may lead to duplicated eos tokens being added.)rE eos_token_idwarningswarnrg)rrs r _add_eos_if_not_present%MyT5Tokenizer._add_eos_if_not_presents[ y>A )B-43D3D"D MM,T^^,<=++   1 122 2r/crUR/nUc[X-5S/-$[X-U-U-5S/-$)ay Create a mask from the two sequences passed to be used in a sequence-pair classification task. MyT5 does not make use of token type ids, therefore a list of zeros is returned. Args: token_ids_0 (`list[int]`): List of IDs. token_ids_1 (`list[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `list[int]`: List of zeros. r)rrE)rrreoss r $create_token_type_ids_from_sequences2MyT5Tokenizer.create_token_type_ids_from_sequencessL   !  {()QC/ /;${2S89QC??r/cXURU5nUcU$URU5nX-$)a" Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. A sequence has the following format: - single sequence: `X ` - pair of sequences: `A B ` Args: token_ids_0 (`list[int]`): List of IDs to which the special tokens will be added. token_ids_1 (`list[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `list[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens. )r)rrrs r build_inputs_with_special_tokens.MyT5Tokenizer.build_inputs_with_special_tokenss9&22;?   66{CK, ,r/textc vURS5Vs/sHo3SPM nnURU5nU$s snf)zTake as input a string and return a list of strings (tokens) for words/sub-words. Represents tokens in two character hex formatutf-8r3)encodemorphological_encode)rrrvrwtokenss r _tokenizeMyT5Tokenizer._tokenize4sA'+kk'&:;&:sG*&:;**62 ?#dF_F_B`` E%5005==//  9 r/save_directoryfilename_prefixc [RRU5(a6[RRX(aUS-OS[S-5nOU(aUS-OSU-n[ USSS9nUR [R"URSSS 95 SSS5 U4$!,(df  U4$=f) N-rwr)encodingrdF)indent ensure_ascii) ospathisdirjoinVOCAB_FILES_NAMESrwriterdumpsrr)rrrrwriters r save_vocabularyMyT5Tokenizer.save_vocabularyns 77== ( (/3!6rUfgsUt tJ4C/C/n\J *cG 4 LLDNN15Q R5}5 4}s 70B22 C)rorqrrrsrtrp)zzz}N)r0N)NFr|)rOrPrQrRrSmodel_input_namesrvocab_files_namesr!propertyr}rrTrr_rrrrrrrrrrrtuplerrU __classcell__)rys@r rWrWs4%&67) "&,  , , \$$puO9O379t3COhlO cOO8 3c 3tCy 3GK@9@379t3C@ c@0GK-9-379t3C- c-4cS  DI$s) DI$s) . c C$J Z_`cZd  r/rW)rSrrr collectionsrtokenization_pythonrrutilsr get_loggerrOloggerrr rW__all__r4r/r rsc) #B   H %"#34ccLr'rj  r/