Z jA S r SSKJrJrJrJrJrJr SSKJ r SSK
J r SSK J
r
\
R " \5 rSS0r " S S
\ 5 rS
/rg ) z Tokenization classes for XGLM. )Regex Tokenizerdecoders normalizerspre_tokenizers
processors)Unigram )TokenizersBackend)loggingtokenizer_fileztokenizer.jsonc ^ \ rS rSrSr\rSS/r\r SS\
\ \ \
\
4 - S- S\
S\
S \
S
\
S \
S \
S
\4U 4S jjjrSrU =r$ )
XGLMTokenizer a
Construct a XGLM tokenizer (backed by HuggingFace's tokenizers library). Based on BPE.
This tokenizer inherits from [`TokenizersBackend`] which contains most of the main methods. Users should
refer to this superclass for more information regarding those methods.
Args:
tokenizer_file (`str`, *optional*):
Path to a tokenizers JSON file containing the serialization of a tokenizer.
bos_token (`str`, *optional*, defaults to `""`):
The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token.
eos_token (`str`, *optional*, defaults to `""`):
The end of sequence token.
sep_token (`str`, *optional*, defaults to `""`):
The separator token, which is used when building a sequence from multiple sequences.
cls_token (`str`, *optional*, defaults to `""`):
The classifier token which is used when doing sequence classification.
unk_token (`str`, *optional*, defaults to `""`):
The unknown token.
pad_token (`str`, *optional*, defaults to `""`):
The token used for padding.
vocab (`str`, `dict` or `list`, *optional*):
Custom vocabulary dictionary. If not provided, a minimal vocabulary is created.
merges (`list[tuple[str, str]]`, *optional*):
Custom merge rules for BPE. If not provided, merges are generated from the vocabulary.
add_prefix_space (`bool`, *optional*, defaults to `True`):
Whether to add a prefix space before encoding.
input_idsattention_maskNvocab bos_token eos_token sep_token cls_token unk_token pad_tokenadd_prefix_spacec > SU l [ U R 5 V
s/ s H n
SU
S3PM n n
U R S/ 5 =( d / U S' U S== U V s/ s H oU S ; d M
U PM sn -
ss' Xl Ub Xl O7[ U5 S4[ U5 S4[ U5 S4[ U5 S4/U l [
[ U R SSS95 U l [ R " [ R " [ S 5 S
5 [ R " 5 [ R " [ S 5 S
5 /5 U R l U( a S OS
n
[ R " SU
S9U R l [$ R " SU
S9U R l [( TU ]T " SUUUUUUUS.U D6 [, R. " U R0 SU R0 3U R0 SU R0 S
U R0 SU R0 3U R2 U R4 4U R0 U R6 4/S9U R l g s sn
f s sn f )N z additional_special_tokensg r
F)r unk_id
byte_fallbackz[\n\r\t] z {2,}alwaysneveru ▁) replacementprepend_scheme)r r r r r r r z $A z $B )singlepairspecial_tokens ) num_madeup_wordsrangegetr _vocabstrr r
_tokenizerr SequenceReplacer NFKC
normalizerr Metaspace
pre_tokenizerr decodersuper__init__r TemplateProcessingr r bos_token_id eos_token_idpost_processor)selfr r r r r r r r kwargsi madeup_wordswordr% __class__s {/root/GenerationalWealth/GenerationalWealth/venv/lib/python3.13/site-packages/transformers/models/xglm/tokenization_xglm.pyr8 XGLMTokenizer.__init__> sG !" 49$:O:O4PQ4Pq+aS*4P Q.4jj9TVX.Y._]_*+*+ )0
)T@[9\-\D\0
+ !1 K Y %Y %Y %Y % DK $G$++aW\$]^%0%9%9##E+$6< "##E(OS9
&
" &67(6(@(@Ucq(r%"*"4"4We"f
-
*4)F)Fnn%T$..)9:NN#4'7q 8HT^^L\] !2!23 !2!23 *
&S R0
s I$ I)
*I)
)r/ r- r r* )N