Z j S r SSKJr SSKJr SSKJr SSKJr SSK J
r
\R " \ 5 r
S S
0r\" S S 9 " S
S\5 5 rS/rg)z Tokenization class for SpeechT5. )Any )SentencePieceBackend)logging)requires )EnglishNumberNormalizer
vocab_filezspm_char.model)
sentencepiece)backendsc
0 ^ \ rS rSrSr\rSS/rSr SS\ \
\ 4 S- SS4U 4S jjjr SS
jr
\S 5 r\R S 5 rSS\\ 4S
jjr SS\\ S\\ S- S\S\\ 4U 4S jjjr SS\\ S\\ S- S\\ 4S jjrSrU =r$ )SpeechT5Tokenizer a
Construct a SpeechT5 tokenizer. Based on [SentencePiece](https://github.com/google/sentencepiece).
This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to
this superclass for more information regarding those methods.
Args:
vocab_file (`str`):
[SentencePiece](https://github.com/google/sentencepiece) file (generally has a *.spm* extension) that
contains the vocabulary necessary to instantiate a tokenizer.
bos_token (`str`, *optional*, defaults to `""`):
The begin of sequence token.
eos_token (`str`, *optional*, defaults to `""`):
The end of sequence token.
unk_token (`str`, *optional*, defaults to `""`):
The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
token instead.
pad_token (`str`, *optional*, defaults to `""`):
The token used for padding, for example when batching sequences of different lengths.
normalize (`bool`, *optional*, defaults to `False`):
Whether to convert numeric quantities in the text to their spelt-out english counterparts.
sp_model_kwargs (`dict`, *optional*):
Will be passed to the `SentencePieceProcessor.__init__()` method. The [Python wrapper for
SentencePiece](https://github.com/google/sentencepiece/tree/master/python) can be used, among other things,
to set:
- `enable_sampling`: Enable subword regularization.
- `nbest_size`: Sampling parameters for unigram. Invalid for BPE-Dropout.
- `nbest_size = {0,1}`: No sampling is performed.
- `nbest_size > 1`: samples from the nbest_size results.
- `nbest_size < 0`: assuming that nbest_size is infinite and samples from the all hypothesis (lattice)
using forward-filtering-and-backward-sampling algorithm.
- `alpha`: Smoothing parameter for unigram sampling, and dropout probability of merge operations for
BPE-dropout.
Attributes:
sp_model (`SentencePieceProcessor`):
The *SentencePiece* processor that is used for every conversion (string, tokens and IDs).
input_idsattention_maskFNsp_model_kwargsreturnc
\ > X`l S U l Ub XxS' [ T U ] " SUUUUUUS.UD6 g )Nr )r
bos_token eos_token unk_token pad_token normalize )r _normalizersuper__init__)
selfr
r r r r r r kwargs __class__s
ڃ/root/GenerationalWealth/GenerationalWealth/venv/lib/python3.13/site-packages/transformers/models/speecht5/tokenization_speecht5.pyr SpeechT5Tokenizer.__init__M sS # &(7$ %
!
c UR SU R 5 nU( a SU- nU( a U R U5 nX4$ )Nr )popr
normalizer)r textis_split_into_wordsr r s r! prepare_for_tokenization*SpeechT5Tokenizer.prepare_for_tokenizationj s; JJ{DNN; :D ??4(D~ r# c R U R c [ 5 U l U R $ N)r r )r s r! r' SpeechT5Tokenizer.normalizerr s% # 6 8D r# c Xl g r- )r )r values r! r' r. x s r# c J Uc XR /- $ X- U R /- $ )z=Build model inputs from a sequence by appending eos_token_id.) eos_token_id)r token_ids_0 token_ids_1s r! build_inputs_with_special_tokens2SpeechT5Tokenizer.build_inputs_with_special_tokens| s1 "3"3!44 4(D,=,=+>>>r# r3 r4 already_has_special_tokensc > U( a [ TU ] XSS9$ S/nUc S/[ U5 - U- $ S/[ U5 - S/[ U5 - - U- $ )NT)r3 r4 r7 r r )r get_special_tokens_masklen)r r3 r4 r7 suffix_onesr s r! r9 )SpeechT5Tokenizer.get_special_tokens_mask ss &72 ']a 3
c C#k**k9 9c+&&A3[1A+AB[PPr# c l U R /nUc [ X- 5 S/- $ [ X- U- 5 S/- $ )a}
Create a mask from the two sequences passed to be used in a sequence-pair classification task. SpeechT5 does not
make use of token type ids, therefore a list of zeros is returned.
Args:
token_ids_0 (`list[int]`):
List of IDs.
token_ids_1 (`list[int]`, *optional*):
Optional second list of IDs for sequence pairs.
Returns:
`list[int]`: List of zeros.
r )r2 r: )r r3 r4 eoss r! $create_token_type_ids_from_sequences6SpeechT5Tokenizer.create_token_type_ids_from_sequences sG ! {()QC/ /;,s23qc99r# )r r )zzzzFN)Fr- )NF)__name__
__module__ __qualname____firstlineno____doc__VOCAB_FILES_NAMESvocab_files_namesmodel_input_namesis_fastdictstrr r r* propertyr' setterlistintr5 boolr9 r? __static_attributes__
__classcell__)r s @r! r r s) (T *$&67G
15
c3h $.
:
! !?QUVYQZ ? pu Q 9 Q379t3C Qhl Q
c Q Q GK: 9:379t3C:
c: :r# r N)rE typingr tokenization_utils_sentencepiecer utilsr utils.import_utilsr number_normalizerr
get_loggerrA loggerrF r __all__r r# r! r[ sg ' D * 6
H %!#34
%&E:, E: 'E:P
r#