Z j n S SK r S SKrS SKJrJrJrJrJr S SKJ r SSK
J r SSS.r " S S
\ 5 r
S
/rg) N) Tokenizerdecoders normalizerspre_tokenizers
processors)Unigram )TokenizersBackendz spiece.modelztokenizer.json)
vocab_filetokenizer_filec ^ \ rS rSrSr\rSS/r\r SU 4S jjr
S r S r SS \
\\
- S
\S \S- S \S
\4
U 4S jjjrSrU =r$ )
LasrTokenizer! aP
Construct a LASR tokenizer (backed by HuggingFace's *tokenizers* library). Based on
[Unigram](https://huggingface.co/docs/tokenizers/python/latest/components.html?highlight=unigram#models).
This tokenizer inherits from [`TokenizersBackend`] which contains most of the main methods. Users should
refer to this superclass for more information regarding those methods.
Args:
vocab_file (`str`, *optional*):
[SentencePiece](https://github.com/google/sentencepiece) file (generally has a *.spm* extension) that
contains the vocabulary necessary to instantiate a tokenizer.
eos_token (`str`, *optional*, defaults to `""`):
The end of sequence token.
When building a sequence using special tokens, this is not the token that is used for the end of sequence.
The token used is the `sep_token`.
unk_token (`str`, *optional*, defaults to `""`):
The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
token instead.
pad_token (`str`, *optional*, defaults to `""`):
The token used for padding, for example when batching sequences of different lengths.
extra_ids (`int`, *optional*, defaults to 100):
Add a number of extra ids added to the vocabulary for use as sentinels. These tokens are accessible as
"" where "{%d}" is a number between 0 and extra_ids-1. These tokens can be retrieved by
calling get_sentinel_tokens method and token ids can be by calling get_sentinel_token_ids method
additional_special_tokens (`list[str]`, *optional*):
Additional special tokens used by the tokenizer.
vocab (`str`, `dict` or `list`, *optional*):
Custom vocabulary dict. If not provided, a minimal vocabulary is created using the special tokens.
input_idsattention_maskNc d > XPl Ub~ U V
s/ s H n
S[ U
5 ; d M U
PM n n
[ U 5 S: a$ U[ U5 V s/ s H n SU S3PM sn -
nOIUS: a! U[ U 5 :w a [ SU SU S35 eO![ U5 V s/ s H n SU S3PM n n U nUb Xpl Od[ U5 S4[ U5 S4[ U5 S4S /U l [ US-
S
S
5 H$ n U R
R
SU S3S45 M& [ [ U R
S S S
95 U l Ub% [ R " U5 U R l [ R " [ R " 5 [ R " SSSS9/5 U R l [$ R " SSSS9U R l [( T
U ]T " SUUUUUS.U D6 [, R. " SS// SQSU R0 4/S9U R l g s sn
f s sn f s sn f )Nz
r zBoth extra_ids (z!) and additional_special_tokens (zm) are provided to LasrTokenizer. In this case the additional_special_tokens must include the extra_ids tokensg ) ▁g r F)unk_id
byte_fallbackr alwaysT) replacementprepend_schemesplit) eos_token unk_token pad_token extra_idsadditional_special_tokens$A)r" r# z$Br# )singlepairspecial_tokens )
_extra_idsstrlenrange
ValueError
_vocab_scoresappendr r
_tokenizerr Precompiled
normalizerr SequenceWhitespaceSplit Metaspace
pre_tokenizerr decodersuper__init__r TemplateProcessing eos_token_idpost_processor)selfr r r _spm_precompiled_charsmapr r! vocabr kwargsx extra_tokensi __class__s {/root/GenerationalWealth/GenerationalWealth/venv/lib/python3.13/site-packages/transformers/models/lasr/tokenization_lasr.pyr8 LasrTokenizer.__init__J s_ $ % 0'@['@!LTWXYTZDZA'@L[< 1$)yIY-ZIYA
1#Q.?IY-ZZ)Q 9L0A#A &yk1RSlRm n 8=Y7GH7G!j1 -7GLH(4 % !& Y %Y %Y % "D 9q="b1""))Zs!+-**+ *
&k \-Z Is H#H# H(H-c T [ [ [ S U R 5 5 5 $ )zQGet the list of sentinel tokens (extra_id tokens) from additional_special_tokens.c F [ [ R " SU 5 5 S L$ )Nz)boolresearch)r@ s rD 3LasrTokenizer.get_sentinel_tokens.. s bii0A1&E!Fd!R )listsetfilterr! )r< s rD get_sentinel_tokens!LasrTokenizer.get_sentinel_tokens s& RTXTrTrs t
rM c j U R 5 Vs/ s H oR U5 PM sn$ s snf )z&Get the token IDs for sentinel tokens.)rQ convert_tokens_to_ids)r< tokens rD get_sentinel_token_ids$LasrTokenizer.get_sentinel_token_ids s. ?C?W?W?YZ?Ye**51?YZZZs 0 token_idsskip_special_tokens clean_up_tokenization_spaces group_tokensreturnc > [ U[ 5 ( a U/nU( a( [ R " U5 Vs/ s H ofS PM nnU Vs/ s H owU R :w d M UPM nn[
TU ] " SUUUS.UD6$ s snf s snf )Nr )rX rY rZ r' )
isinstanceint itertoolsgroupby pad_token_idr7 _decode) r< rX rY rZ r[ r? token_grouprU rC s rD rc LasrTokenizer._decode s i % %" I ;D;L;LY;WX;WKQ ;WIX )2P ud>O>O5OU Pw
3)E
Y Qs B
B$B)r( r/ r- )r# zzNd NNN)FNT)__name__
__module__ __qualname____firstlineno____doc__VOCAB_FILES_NAMESvocab_files_namesmodel_input_namesr modelr8 rQ rV r_ rN rH r) rc __static_attributes__
__classcell__)rC s @rD r r ! s "H *$&67 E "&"&K
Z
[ %*48 !
c?
"
'+Tk
rM r )r` rI
tokenizersr r r r r tokenizers.modelsr tokenization_utils_tokenizersr
rl r __all__r' rM rD rv sA * S S % > $2EUV U
% U
p
rM