Z jM S SK JrJrJrJrJr S SKJr SSKJ r SSK
J r \ R " \
5 rSSS.r " S S
\ 5 rS
/rg ) ) Tokenizerdecoders normalizerspre_tokenizers
processors)BPE )TokenizersBackend)loggingz
vocab.jsonz
merges.txt)
vocab_file merges_filec ^ \ rS rSrSr\rSS/r\r SS\
\ \
\ 4 - S- S\
\
\
- S- S\
S \
S
\
S \
S \
S
\
S- S\
S- 4U 4S jjjrSrU =r$ )HerbertTokenizer a~
Construct a BPE tokenizer for HerBERT (backed by HuggingFace's tokenizers library).
Peculiarities:
- uses BERT's pre-tokenizer: BertPreTokenizer splits tokens on spaces, and also on punctuation. Each occurrence of
a punctuation character will be treated separately.
This tokenizer inherits from [`TokenizersBackend`] which contains most of the methods. Users should refer to the
superclass for more information regarding methods.
Args:
vocab_file (`str`):
Path to the vocabulary file.
merges_file (`str`):
Path to the merges file.
cls_token (`str`, *optional*, defaults to `""`):
The classifier token.
unk_token (`str`, *optional*, defaults to `""`):
The unknown token.
pad_token (`str`, *optional*, defaults to `""`):
The padding token.
mask_token (`str`, *optional*, defaults to `""`):
The mask token.
sep_token (`str`, *optional*, defaults to `""`):
The separator token.
vocab (`str`, `dict` or `list`, *optional*):
Custom vocabulary dictionary.
merges (`str` or `list[str]`, *optional*):
Custom merges list.
input_idsattention_maskNvocabmerges cls_token unk_token pad_token
mask_token sep_tokenr r
c
B > Ub UO [ U5 S0U l U=( d / U l [ [ U R U R S [ U5 SS95 U l [ R " SSSSS9U R
l [ R " 5 U R
l [ R " SS9U R
l [ T U ]@ " S UUUUUS.U
D6 [" R$ " U R& S 4U R( S4S
9U R
l g ) Nr z)r r dropoutr end_of_word_suffixFT) lowercase
strip_accents
clean_texthandle_chinese_chars)suffix)r r r r r )sepcls )str_vocab_mergesr r
_tokenizerr BertNormalizer
normalizerr BertPreTokenizer
pre_tokenizerr
BPEDecoderdecodersuper__init__r BertProcessingr r post_processor) selfr r r r r r r r r
kwargs __class__s ځ/root/GenerationalWealth/GenerationalWealth/venv/lib/python3.13/site-packages/transformers/models/herbert/tokenization_herbert.pyr1 HerbertTokenizer.__init__A s % 0es9~q6I | # kk|| i.#)
&1%?%?5TX\&
" )7(G(G(I%"*"5"5V"D
!
*4)B)B # #*
& )r( r) r' ) NNzzzzzNN)__name__
__module__ __qualname____firstlineno____doc__VOCAB_FILES_NAMESvocab_files_namesmodel_input_namesr modelr& dictintlistr1 __static_attributes__
__classcell__)r6 s @r7 r r s @ *$&67 E .2)- " !%"&+
T#s(^#d*+
d3i$&+
+
+
+
+
+
$J+
4Z+
+
r9 r N)
tokenizersr r r r r tokenizers.modelsr tokenization_utils_tokenizersr
utilsr
get_loggerr: loggerr? r __all__r% r9 r7 rO sN T S ! >
H %#/ M P
( P
f
r9