ZjF0SSKrSSKrSSKrSSKJr SSKJr SSKJr SSK r SSK J r SSK J r SSKJr \ R "\5rS S S S S S.rSr\"SS9"SS\ 55rS\S\\\4S\ R04SjrS\SS4SjrS\S\\-4SjrS/rg)N)Path)copyfile)Any)PreTrainedTokenizer)logging)requiresz source.spmz target.spmz vocab.jsonztarget_vocab.jsonztokenizer_config.json) source_spm target_spmvocabtarget_vocab_filetokenizer_config_fileu▁) sentencepiece)backendsc ^\rSrSrSr\rSS/rS-S\\ \ 4S-SS4U4Sjjjr S r S \ S\ 4S jr S rS \ 4SjrS \ S\\ 4SjrS\S\ 4SjrU4SjrU4SjrS.S\S\S-S\ 4U4SjjjrS\\ S\ 4SjrS/S\\4SjjrSrSr\S\4Sj5rS/S\ S\ S-S\\ 4SjjrS\4S jr S!r!S"r"S\4S#jr#S$\SS4S%jr$S&r%S'r&S0S(\S)\S-S*\S\\4S+jjr'S,r(U=r)$)1MarianTokenizer,a Construct a Marian tokenizer. Based on [SentencePiece](https://github.com/google/sentencepiece). This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: source_spm (`str`): [SentencePiece](https://github.com/google/sentencepiece) file (generally has a .spm extension) that contains the vocabulary for the source language. target_spm (`str`): [SentencePiece](https://github.com/google/sentencepiece) file (generally has a .spm extension) that contains the vocabulary for the target language. source_lang (`str`, *optional*): A string representing the source language. target_lang (`str`, *optional*): A string representing the target language. unk_token (`str`, *optional*, defaults to `""`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. eos_token (`str`, *optional*, defaults to `""`): The end of sequence token. pad_token (`str`, *optional*, defaults to `""`): The token used for padding, for example when batching sequences of different lengths. model_max_length (`int`, *optional*, defaults to 512): The maximum sentence length the model accepts. additional_special_tokens (`list[str]`, *optional*, defaults to `["", ""]`): Additional special tokens used by the tokenizer. sp_model_kwargs (`dict`, *optional*): Will be passed to the `SentencePieceProcessor.__init__()` method. The [Python wrapper for SentencePiece](https://github.com/google/sentencepiece/tree/master/python) can be used, among other things, to set: - `enable_sampling`: Enable subword regularization. - `nbest_size`: Sampling parameters for unigram. Invalid for BPE-Dropout. - `nbest_size = {0,1}`: No sampling is performed. - `nbest_size > 1`: samples from the nbest_size results. - `nbest_size < 0`: assuming that nbest_size is infinite and samples from the all hypothesis (lattice) using forward-filtering-and-backward-sampling algorithm. - `alpha`: Smoothing parameter for unigram sampling, and dropout probability of merge operations for BPE-dropout. Examples: ```python >>> from transformers import MarianForCausalLM, MarianTokenizer >>> model = MarianForCausalLM.from_pretrained("Helsinki-NLP/opus-mt-en-de") >>> tokenizer = MarianTokenizer.from_pretrained("Helsinki-NLP/opus-mt-en-de") >>> src_texts = ["I am a small frog.", "Tom asked his teacher for advice."] >>> tgt_texts = ["Ich bin ein kleiner Frosch.", "Tom bat seinen Lehrer um Rat."] # optional >>> inputs = tokenizer(src_texts, text_target=tgt_texts, return_tensors="pt", padding=True) >>> outputs = model(**inputs) # should work ``` input_idsattention_maskNsp_model_kwargsreturnc >U c0OU Ul[U5R5(d SU35eXl[ U5Ul[ U5UR ;a [S5eU (aL[ U5UlURR5VVs0sHupX_M snnUl /Ul OUR R5VVs0sHupX_M snnUl UR Vs/sH4oRS5(dMURS5(dM2UPM6 snUl XPlX`lX/Ul[#XR5Ul[#X R5UlUR$UlUR UlUR-5 SUl[0TU]d"SUUUUU U URUU S. U D6 gs snnfs snnfs snf)Nzcannot find spm source z token must be in the vocab>>< -=> > "+,=">D -1-@-@-F-F-HI-HTQAD-HIDL,.D )-1\\-?-?-AB-ATQAD-ABDL>Bll2vlll[_N`1efeoeopteu1l2vD )&&$1#:/C/CD":/C/CD??#||  ,1)  ##- 00/+  -JC2vs$G. G4G:"G::G:cSSKJn U"UR5RUlg![ [ 4a! [R"S5 SUlgf=f)Nr)MosesPunctNormalizerz$Recommended: pip install sacremoses.cU$Nr")xs r=3MarianTokenizer._setup_normalizer..sQ) sacremosesr@r normalizepunc_normalizer ImportErrorFileNotFoundErrorwarningswarn)r8r@s r=r4!MarianTokenizer._setup_normalizersM / 7#78H8H#I#S#SD ./ / MM@ A#.D  /s'*.AArCc6U(aURU5$S$)zHCover moses empty string edge case. They return empty list for '' input!)rI)r8rCs r=rHMarianTokenizer.normalizes*+t##A&33rFcpXR;aURU$URUR$rB)r3r)r8tokens r=_convert_token_to_id$MarianTokenizer._convert_token_to_ids6 (( (''. .##DNN33rFtextc/nURS5(a5URS5=nS:waURUSUS-5 XS-SnX!4$)z6Remove language codes like >>fr<< before sentencepiecerrN)r,findappend)r8rVcodeend_locs r=remove_language_code$MarianTokenizer.remove_language_codesW ??4 4&@gR%G KK]w{+ ,! &DzrFclURU5up!URRU[S9nX#-$)N)out_type)r^r2encoder&)r8rVr\piecess r= _tokenizeMarianTokenizer._tokenizes7..t4 !!(((<}rFindexcXR;aURU$UR(a URO URnUR U5nU(aU$UR $)z?Converts an index (integer) in a token (str) using the decoder.)r*r5r0r1 IdToPiecer)r8rf spm_modelpieces r=_convert_id_to_token$MarianTokenizer._convert_id_to_tokensS LL <<& &'+'H'HDOOdoo ##E*u14>>1rFc &>[TU]"U40UD6$)a Convert a list of lists of token ids into a list of strings by calling decode. Args: sequences (`Union[list[int], list[list[int]], np.ndarray, torch.Tensor]`): List of tokenized input ids. Can be obtained using the `__call__` method. skip_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not to remove special tokens in the decoding. clean_up_tokenization_spaces (`bool`, *optional*): Whether or not to clean up the tokenization spaces. If `None`, will default to `self.clean_up_tokenization_spaces` (available in the `tokenizer_config`). use_source_tokenizer (`bool`, *optional*, defaults to `False`): Whether or not to use the source tokenizer to decode sequences (only applicable in sequence-to-sequence problems). kwargs (additional keyword arguments, *optional*): Will be passed to the underlying model specific decode method. Returns: `list[str]`: The list of decoded sentences. )r6 batch_decode)r8 sequencesr9r<s r=rnMarianTokenizer.batch_decodes*w#I888rFc &>[TU]"U40UD6$)a_ Converts a sequence of ids in a string, using the tokenizer and vocabulary with options to remove special tokens and clean up tokenization spaces. Similar to doing `self.convert_tokens_to_string(self.convert_ids_to_tokens(token_ids))`. Args: token_ids (`Union[int, list[int], np.ndarray, torch.Tensor]`): List of tokenized input ids. Can be obtained using the `__call__` method. skip_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not to remove special tokens in the decoding. clean_up_tokenization_spaces (`bool`, *optional*): Whether or not to clean up the tokenization spaces. If `None`, will default to `self.clean_up_tokenization_spaces` (available in the `tokenizer_config`). use_source_tokenizer (`bool`, *optional*, defaults to `False`): Whether or not to use the source tokenizer to decode sequences (only applicable in sequence-to-sequence problems). kwargs (additional keyword arguments, *optional*): Will be passed to the underlying model specific decode method. Returns: `str`: The decoded sentence. )r6decode)r8 token_idsr9r<s r=rrMarianTokenizer.decodes0w~i2622rFskip_special_tokensclean_up_tokenization_spacesc |>UR(+nURSU5Ul[TU]"SUUUS.UD6$)zCInternal decode method that handles use_source_tokenizer parameter.use_source_tokenizer)rsrurvr")r!popr5r6_decode)r8rsrurvr9default_use_sourcer<s r=rzMarianTokenizer._decodesR"&!5!55,2JJ7MOa,b)w  3)E    rFtokenscZUR(a URO URn/nSnUH@nXPR;aXBR U5U-S-- n/nM/UR U5 MB XBR U5- nUR [S5nUR5$)zQUses source spm if _decode_use_source_tokenizer is True, and target spm otherwiserP ) r5r0r1all_special_tokens decode_piecesr[replaceSPIECE_UNDERLINEstrip)r8r}sp_modelcurrent_sub_tokens out_stringrSs r=convert_tokens_to_string(MarianTokenizer.convert_tokens_to_strings&*&G&G4??T__ E///445GH5PSVVV %'""))%0  ,,-?@@ ''(8#> !!rFcJUcXR/-$X-UR/-$)z=Build model inputs from a sequence by appending eos_token_id.) eos_token_id)r8 token_ids_0 token_ids_1s r= build_inputs_with_special_tokens0MarianTokenizer.build_inputs_with_special_tokens&s1  "3"3!44 4(D,=,=+>>>rFcHURUlURUlgrB)r0r2r%r3r8s r=_switch_to_input_mode%MarianTokenizer._switch_to_input_mode-s??#||rFclURUlUR(aURUlggrB)r1r2r!r(r3rs r=_switch_to_target_mode&MarianTokenizer._switch_to_target_mode1s*??   #'#6#6D  rFc,[UR5$rB)lenr%rs r= vocab_sizeMarianTokenizer.vocab_size6s4<<  rFsave_directoryfilename_prefixc[RRU5(d[R SUS35 g/nUR (a[RR UU(aUS-OS[S-5n[RR UU(aUS-OS[S-5n[URU5 [URU5 URU5 URU5 O\[RR X(aUS-OS[S-5n[URU5 URU5 [[S[S/URURUR/5GH$upxn [RR X(aUS-OSU-5n [RR!U5[RR!U 5:waB[RR#U5(a[%X5 URU 5 M[RR#U5(aM['U S 5n U R)5n U R+U 5 SSS5 URU 5 GM' [-U5$!,(df  N/=f) NzVocabulary path (z) should be a directory-rPr r r r wb)ospathisdirloggererrorr!joinVOCAB_FILES_NAMES save_jsonr%r(r[zipr.r0r1abspathisfileropenserialized_model_protowritetuple) r8rr saved_filesout_src_vocab_fileout_tgt_vocab_fileout_vocab_filespm_save_filename spm_orig_pathri spm_save_pathficontent_spiece_models r=save_vocabularyMarianTokenizer.save_vocabulary:sEww}}^,, LL,^,<   1 2   1 2WW\\/3!6rUfgnUo oN dllN 3   ~ .;> | ,.? .M N NN __doo .< 7 i GGLL/3!6rUf fMww}-1OOTVT[T[TbTbcpTqTq6""=1WW^^M22-."+4+K+K+M(HH12/""=1< "[!! /.s "K K( c"UR5$rB) get_src_vocabrs r= get_vocabMarianTokenizer.get_vocabgs!!##rFcB[UR40URD6$rB)dictr%added_tokens_encoderrs r=rMarianTokenizer.get_src_vocabjsDLL>D$=$=>>rFcB[UR40URD6$rB)rr(added_tokens_decoderrs r= get_tgt_vocabMarianTokenizer.get_tgt_vocabmsD''E4+D+DEErFcURR5nUR[R /SQ55 U$)N)r0r1r2rIr )__dict__copyupdaterfromkeys)r8states r= __getstate__MarianTokenizer.__getstate__ps4 ""$ MMm n  rFdc^UTl[TS5(d0Tl[TS5(dSTlU4SjTR5uTlTlTR TlTR5 g)Nrr5Fc3P># UHn[UTR5v M g7frB)r/r).0fr8s r= /MarianTokenizer.__setstate__..s$+fWeRSHQ8L8L,M,MWes#&) rhasattrrr5r.r0r1r2r4)r8rs` r= __setstate__MarianTokenizer.__setstate__wsj t.//#%D t;<<05D -+fW[WeWe+f(??  rFcg)zJust EOSr")r8argsr9s r=num_special_tokens_to_add)MarianTokenizer.num_special_tokens_to_addsrFc[UR5nURUR5 UVs/sH o3U;aSOSPM sn$s snf)Nrr)setall_special_idsremove unk_token_id)r8seqrrCs r=_special_token_mask#MarianTokenizer._special_token_masksHd223t001:=>#Q/)q0#>>>sA rralready_has_special_tokenscU(aURU5$UcURU5S/-$URX-5S/-$)zCGet list where entries are [1] if a token is [eos] or [pad] else 0.r)r)r8rrrs r=get_special_tokens_mask'MarianTokenizer.get_special_tokens_masksQ &++K8 8  ++K8A3> >++K,EF!L LrF)rr5r3r2r*r%rIr!rrr.r0r1r+r(r) NNNzzziNF)FNrB)NF)*__name__ __module__ __qualname____firstlineno____doc__rvocab_files_namesmodel_input_namesrr&rr7r4rHrTr^listrdintrkrnrrboolrzrrrrpropertyrrrrrrrrrrr__static_attributes__ __classcell__)r<s@r=rr,s8 t*$&6715= c3h$.=  = = ~/434344cd3i 2#2#29.3:%*48  " '+Tk    ""tCy"S" ?QUVYQZ?,7 !C!!+"c+"C$J+"Z_`cZd+"Z$4$?Fd !d !t !? fk M M.2Tk M^b M c M MrFrrrrcT[R"S0UD6nURU5 U$)Nr")rSentencePieceProcessorLoad)rrspms r=r/r/s%  . . A ACHHTN JrFcz[US5n[R"XSS9 SSS5 g!,(df  g=f)NwrY)indent)rjsondump)datarrs r=rrs% dCA $!$ s, :c|[US5n[R"U5sSSS5 $!,(df  g=f)Nr)rrload)rrs r=r$r$s" dCAyy| s- ;)rrrLpathlibrshutilrtypingrrtokenization_pythonrutilsrutils.import_utilsr get_loggerrrrrrr&rrr/rrr$__all__r"rFr=r s 6*   H % ,4   %&iM)iM'iMX 3c3hM<`<` %#%$% CD4K  rF