Zj$SrSSKrSSKrSSKrSSKJr SSKJr SSKJ r J r J r \ "5(aSSK r \ "5(aSSK r\ R"\5rSS0rS r"S S \5rS /rg) zTokenization class for VITS.N)Any)PreTrainedTokenizer)is_phonemizer_availableis_uroman_availablelogging vocab_filez vocab.jsonc\[R"S5nURU5nUSLnU$)Nz [^\x00-\x7F])recompilesearch) input_stringnon_roman_patternmatch has_non_romans {/root/GenerationalWealth/GenerationalWealth/venv/lib/python3.13/site-packages/transformers/models/vits/tokenization_vits.pyhas_non_roman_charactersr$s3 ?3  $ $\ 2E%M c ^\rSrSrSr\rSS/rSSU4Sjjjr\ S5r S r S r S r SS \S \S\S-S\\\\\444SjjrS \S\\4SjrS\\S\4SjrSrSrSS\S\S-S\\S-4SjjrSrU=r$) VitsTokenizer.a| Construct a VITS tokenizer. Also supports MMS-TTS. This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: vocab_file (`str`): Path to the vocabulary file. language (`str`, *optional*): Language identifier. add_blank (`bool`, *optional*, defaults to `True`): Whether to insert token id 0 in between the other tokens. normalize (`bool`, *optional*, defaults to `True`): Whether to normalize the input text by removing all casing and punctuation. phonemize (`bool`, *optional*, defaults to `True`): Whether to convert the input text into phonemes. is_uroman (`bool`, *optional*, defaults to `False`): Whether the `uroman` Romanizer needs to be applied to the input text prior to tokenizing. input_idsattention_maskNreturnc f>[USS9n [R"U 5UlSSS5 URR 5V V s0sHupX_M sn n UlX@lXPlX`lXpl Xl [T U]0"SUUUUUUUSS.U D6 g!,(df  Nz=fs sn n f)Nutf-8encodingnone) pad_token unk_tokenlanguage add_blank normalize phonemize is_uromanspecial_tokens_pattern) openjsonloadencoderitemsdecoderr"r#r$r%r&super__init__)selfr r r!r"r#r$r%r&kwargs vocab_handlekv __class__s rr0VitsTokenizer.__init__Gs*w /<99\2DL0*.););)=>)=)=>  """"  #)  0 /?sB B- B*c,[UR5$N)lenr,)r1s r vocab_sizeVitsTokenizer.vocab_sizejs4<<  rc[UR5Vs0sHoRU5U_M nnURUR5 U$s snfr9)ranger;convert_ids_to_tokensupdateadded_tokens_encoder)r1ivocabs r get_vocabVitsTokenizer.get_vocabnsL;@;QR;Qa++A.1;QR T../ SsAc[URR55[URR55-nSnSnU[ U5:agSnUH-nXU[ U5-U:XdMX6- nU[ U5- nSn O U(dX1UR 5- nUS- nU[ U5:aMgU$)zfLowercase the input string, respecting any special token ids that may be part or entirely upper-cased.rFT)listr,keysrAr:lower)r1rall_vocabulary filtered_textrB found_matchwords rnormalize_textVitsTokenizer.normalize_textssdll//12T$:S:S:X:X:Z5[[ #l##K&AD M2d:!)MTNA"&K 'a!6!6!88 Q#l##rcJURS:XaURSS5nU$)z4Special treatment of characters in certain languagesronuțuţ)r"replace)r1texts r_preprocess_charVitsTokenizer._preprocess_chars# ==E !<<d+D rrUis_split_into_wordsr$c ^UbUO TRnU(aTRU5nTRU5n[U5(a\TR(aK[ 5(d[ RS5 O&[R"5nURU5nTR(aN[5(d [S5e[R"USSSSSS9n[R "SSU5nXT4$U(a6S R#[%['U4S jU555R)5nXT4$) am Performs any necessary transformations before tokenization. This method should pop the arguments from kwargs and return the remaining `kwargs` as well. We test the `kwargs` at the end of the encoding process to be sure all the arguments have been used. Args: text (`str`): The text to prepare. is_split_into_words (`bool`, *optional*, defaults to `False`): Whether or not the input is already pre-tokenized (e.g., split into words). If set to `True`, the tokenizer assumes the input is already split into words (for instance, by splitting it on whitespace) which it will tokenize. normalize (`bool`, *optional*, defaults to `None`): Whether or not to apply punctuation and casing normalization to the text inputs. Typically, VITS is trained on lower-cased and un-punctuated text. Hence, normalization is used to ensure that the input text consists only of lower-case characters. kwargs (`dict[str, Any]`, *optional*): Keyword arguments to use for the tokenization. Returns: `tuple[str, dict[str, Any]]`: The prepared text and the unused kwargs. aCText to the tokenizer contains non-Roman characters. To apply the `uroman` pre-processing step automatically, ensure the `uroman` Romanizer is installed with: `pip install uroman` Note `uroman` requires python version >= 3.10Otherwise, apply the Romanizer manually as per the instructions: https://github.com/isi-nlp/uromanzEPlease install the `phonemizer` Python package to use this tokenizer.zen-usespeakT)r"backendstrippreserve_punctuation with_stressz\s+ rGc">UTR;$r9)r,)charr1s r8VitsTokenizer.prepare_for_tokenization..sTT\\=Qr)r$rPrVrr&rloggerwarningurUromanromanize_stringr%r ImportError phonemizerr subjoinrIfilterr\)r1rUrXr$r2rMuromans` rprepare_for_tokenization&VitsTokenizer.prepare_for_tokenizations4"+!6IDNN &&t,D--d3 #M 2 2t~~&((y & 6 6} E >>*,,!"ijj&00  %) MFF63 >M $$ GGD0QS`)a$bciikM$$rc[U5nUR(a-URS5/[U5S-S--nX#SSS2'UnU$)z]Tokenize a string by inserting the `` token at the boundary between adjacent characters.rrHN)rIr#_convert_id_to_tokenr:)r1rUtokens intersperseds r _tokenizeVitsTokenizer._tokenizesOd >> 55a89S[1_q=PQL!'A !F rrtctUR(a[U5S:aUSSS2nSRU5$)NrHrrrG)r#r:rl)r1rts rconvert_tokens_to_string&VitsTokenizer.convert_tokens_to_strings0 >>c&kAoADqD\FwwvrcVXR;aURU$UR$)z0Converts a token (str) in an id using the vocab.)r, unk_token_id)r1tokens r_convert_token_to_id"VitsTokenizer._convert_token_to_ids' LL <<& &   rc8URRU5$)z=Converts an index (integer) in a token (str) using the vocab.)r.get)r1indexs rrs"VitsTokenizer._convert_id_to_tokens||&&rsave_directoryfilename_prefixc [RRU5(d[R SUS35 g[RR X(aUS-OS[ S-5n[USSS9nUR[R"URS S S S 9S -5 SSS5 U4$!,(df  U4$=f)NzVocabulary path (z) should be a directory-rGr wrrrrTF)indent sort_keys ensure_ascii ) ospathisdirrderrorrlVOCAB_FILES_NAMESr)writer*dumpsr,)r1rrr fs rsave_vocabularyVitsTokenizer.save_vocabularysww}}^,, LL,^,<> C)r#r.r,r&r"r$r%)zzNTTTF)rN)FNr9)__name__ __module__ __qualname____firstlineno____doc__rvocab_files_namesmodel_input_namesr0propertyr;rDrPrVstrbooltupledictrrorIrvryr~rsr__static_attributes__ __classcell__)r6s@rrr.s**$&67 !  ! ! F!! *VZ?%?%.2?%GKd{?% sDcN" #?%B c d3i tCyS ! ' c C$J Z_`cZdgkZk  rr)rr*rr typingrtokenization_pythonrutilsrrrrjrnrf get_loggerrrdrrr__all__r(rrrss# 6JJ   H %!<0G'GT  r