Zj&SrSSKrSSKrSSKrSSKJr SSKJrJ r J r J r J r SSK JrJrJrJrJr SSKJr \"5(aSSKr\R,"\5rSS 0rS rS r\"S S 9"SS\ 55rS/rg)z!Tokenization class for Pop2Piano.N) BatchFeature) AddedToken BatchEncodingPaddingStrategyPreTrainedTokenizerTruncationStrategy) TensorTypeis_pretty_midi_availableloggingrequires_backendsto_numpy)requiresvocabz vocab.jsonc*X - nUb [X!5nU$N)minnumbercutoff_time_idx current_idxs څ/root/GenerationalWealth/GenerationalWealth/venv/lib/python3.13/site-packages/transformers/models/pop2piano/tokenization_pop2piano.pytoken_time_to_noter&s K"+7 ctX0b.X0nXd:a#UnURXgX/5 US:XaSOUnXU'U$XCU'U$)Nr)append) rcurrent_velocitydefault_velocitynote_onsets_readyrnotes onset_idx offset_idx onsets_readys rtoken_note_to_noter$.s[ ,%-  "$J LL)J K#3q#84kL(4f % L%0&! Lr) pretty_miditorch)backendscH^\rSrSrSrSS/r\rS+U4Sjjr\ S5r Sr S\ S \ 4S jrS,S \ 4S jjrS \R"S \ S\ S\ 4SjrS-S \R"S\R"S \ S\ S\ 4 SjjrS.S \R"S\S\S-4SjjrS/S\R"S\R"S\ 4SjjrS.S\S\S-S \\4SjjrS0S\R"\ \R6-S\S-S\ S-S \4SjjrS0S\R"\ \R6-S\S-S\ S-S \4SjjrS1S\R"\ \R6-\ \ \R6-S \ \-\!-S!\ \-\-S\ S-S"\ S-S#\ S-S$\\"-S-S%\ S \4S&jjr#S2S'\$S(\ 4S)jjr%S*r&U=r'$)3Pop2PianoTokenizer=aF Constructs a Pop2Piano tokenizer. This tokenizer does not require training. This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: vocab (`str`): Path to the vocab file which contains the vocabulary. default_velocity (`int`, *optional*, defaults to 77): Determines the default velocity to be used while creating midi Notes. num_bars (`int`, *optional*, defaults to 2): Determines cutoff_time_idx in for each token. unk_token (`str` or `tokenizers.AddedToken`, *optional*, defaults to `"-1"`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. eos_token (`str` or `tokenizers.AddedToken`, *optional*, defaults to 1): The end of sequence token. pad_token (`str` or `tokenizers.AddedToken`, *optional*, defaults to 0): A special token used to make arrays of tokens the same size for batching purpose. Will then be ignored by attention mechanisms or loss computation. bos_token (`str` or `tokenizers.AddedToken`, *optional*, defaults to 2): The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token. token_idsattention_maskc N>[U[5(a [USSS9OUn[U[5(a [USSS9OUn[U[5(a [USSS9OUn[U[5(a [USSS9OUnX lX0l[ US5n [ R"U 5UlSSS5 URR5V V s0sHupX_M sn n Ul [T U]0"SUUUUS.UD6 g!,(df  NX=fs sn n f)NF)lstriprstriprb) unk_token eos_token pad_token bos_token) isinstancestrrrnum_barsopenjsonloadencoderitemsdecodersuper__init__) selfrrr8r1r2r3r4kwargsfilekv __class__s rr@Pop2PianoTokenizer.__init__[sJTT]_bIcIcJyuEir IST]_bIcIcJyuEir IST]_bIcIcJyuEir IST]_bIcIcJyuEir 0 % $99T?DL*.););)=>)=)=>        ?s"D#D! Dc,[UR5$)z-Returns the vocabulary size of the tokenizer.)lenr<rAs r vocab_sizePop2PianoTokenizer.vocab_size}s4<<  rcB[UR40URD6$)z(Returns the vocabulary of the tokenizer.)dictr<added_tokens_encoderrJs r get_vocabPop2PianoTokenizer.get_vocabsDLL>D$=$=>>rtoken_idreturncURRXRS35nURS5nSR USS5[ US5pCX4/$)a Decodes the token ids generated by the transformer into notes. Args: token_id (`int`): This denotes the ids generated by the transformers to be converted to Midi tokens. Returns: `List`: A list consists of token_type (`str`) and value (`int`). _TOKEN_TIME_Nr)r>getr1splitjoinint)rArRtoken_type_value token_typevalues r_convert_id_to_token'Pop2PianoTokenizer._convert_id_to_tokensc <<++H8H 6TU+11#6HH%5ab%9:C@PQR@S[R"XZ4SS9nMV Uc/$U$)a Converts relative tokens to notes which are then used to generate pretty midi object. Args: tokens (`numpy.ndarray`): Tokens to be converted to notes. beat_offset_idx (`int`): Denotes beat offset index for each note in generated Midi. bars_per_batch (`int`): A parameter to control the Midi output generation. cutoff_time_idx (`int`): Denotes the cutoff time index for each note in generated Midi. N) start_idxrr)axis)rangerIrelative_tokens_ids_to_notesnp concatenate) rArerfrgrr index_tokens _start_idx_cutoff_time_idx_notess r"relative_batch_tokens_ids_to_notes5Pop2PianoTokenizer.relative_batch_tokens_ids_to_notess*3v;'EmG(>+AA+EEJ.; 66$ 07F 6{aQ?(" =I rbeatstepc\UcSOUnURUUUUS9nURXbX#S9nU$)a Converts tokens to Midi. This method calls `relative_batch_tokens_ids_to_notes` method to convert batch tokens to notes then uses `notes_to_midi` method to convert them to Midi. Args: tokens (`numpy.ndarray`): Denotes tokens which alongside beatstep will be converted to Midi. beatstep (`np.ndarray`): We get beatstep from feature extractor which is also used to get Midi. beat_offset_idx (`int`, *optional*, defaults to 0): Denotes beat offset index for each note in generated Midi. bars_per_batch (`int`, *optional*, defaults to 2): A parameter to control the Midi output generation. cutoff_time_idx (`int`, *optional*, defaults to 12): Denotes the cutoff time index for each note in generated Midi. r)rerfrgr) offset_sec)ru notes_to_midi)rArerwrfrgrr midis r!relative_batch_tokens_ids_to_midi4Pop2PianoTokenizer.relative_batch_tokens_ids_to_midisP0 /6!O77+)+ 8  !!%h>W!X rNrjc UVs/sHo@RU5PM nnUnSn[[SUR55S-5Vs/sHnSPM n n/n UHZupU S:Xa U S:Xa OKMU S:Xa [ XUS9nM'U S:XaU nM1U S :Xa[ U UUR U UU S 9n MQ[S 5e [U 5HJupUcM UcUS-nO[X>S-5n[Xo5nU RUUXR /5 ML [U 5S:Xa/$[R"U 5n U SS2S4S -U SS2S4-nU UR5n U $s snfs snf) an Converts relative tokens to notes which will then be used to create Pretty Midi objects. Args: tokens (`numpy.ndarray`): Relative Tokens which will be converted to notes. start_idx (`float`): A parameter which denotes the starting index. cutoff_time_idx (`float`, *optional*): A parameter used while converting tokens to notes. rc3B# UHoRS5v M g7f)NOTEN)endswith).0rDs r BPop2PianoTokenizer.relative_tokens_ids_to_notes..s4^Q]AZZ5G5GQ]srWN TOKEN_SPECIAL TOKEN_TIMErTOKEN_VELOCITY TOKEN_NOTE)rrrrrr zToken type not understood!)r_rlsumr<rr$r ValueError enumeratemaxrrIrnarrayargsort)rArerjrrbwordsrrirr r]rpitch note_onsetcutoffr" note_orders rrm/Pop2PianoTokenizer.relative_tokens_ids_to_notess@FFve**51vF +04^QUQ]Q]4^1^ab1b+cd+caT+cd"' J_,Q;|+0!P[ //#) |+*!%5%)%:%:&7 + !!=>>+#(."++ u:?IHHUOEq!ts*U1a4[8J*,,./EL]Ges E* E/r rycT[US/5 [R"SSS9n[R"SS9n/nUH8upxp[R"U U X'U- X(U- S9n UR U 5 M: XelURR U5 UR5 U$)as Converts notes to Midi. Args: notes (`numpy.ndarray`): This is used to create Pretty Midi objects. beatstep (`numpy.ndarray`): This is the extrapolated beatstep that we get from feature extractor. offset_sec (`int`, *optional*, defaults to 0.0): This represents the offset seconds which is used while creating each Pretty Midi Note. r%ig^@) resolution initial_tempor)program)velocityrstartend) r r% PrettyMIDI InstrumentNoterr instrumentsremove_invalid_notes) rAr rwrynew_pmnew_inst new_notesr!r"rrnew_notes rrz Pop2PianoTokenizer.notes_to_midi4s $0''3eL))!4 6; 2I5"''!)J6(:5 H   X &7<#!!(+##% rsave_directoryfilename_prefixc[RRU5(d[R SUS35 g[RR X(aUS-OS[ S-5n[US5nUR[R"UR55 SSS5 U4$!,(df  U4$=f)aE Saves the tokenizer's vocabulary dictionary to the provided save_directory. Args: save_directory (`str`): A path to the directory where to saved. It will be created if it doesn't exist. filename_prefix (`Optional[str]`, *optional*): A prefix to add to the names of the files saved by the tokenizer. zVocabulary path (z) should be a directoryN-rw) ospathisdirloggererrorrZVOCAB_FILES_NAMESr9writer:dumpsr<)rArrout_vocab_filerCs rsave_vocabulary"Pop2PianoTokenizer.save_vocabularyTsww}}^,, LL,^,<= 7.5` (Volta). return_attention_mask (`bool`, *optional*): Whether to return the attention mask. If left to the default, will return the attention mask according to the specific tokenizer's default, defined by the `return_outputs` attribute. [What are attention masks?](../glossary#attention-mask) return_tensors (`str` or [`~file_utils.TensorType`], *optional*): If set, will return tensors instead of list of python integers. Acceptable values are: - `'pt'`: Return PyTorch `torch.Tensor` objects. - `'np'`: Return Numpy `np.ndarray` objects. verbose (`bool`, *optional*, defaults to `True`): Whether or not to print more information and warnings. Returns: `BatchEncoding` containing the token_ids. rr)rrrrrT)r rr)rrrrrrr5) r6rnndarrayndimlist"_get_padding_truncation_strategiesrrpad)rAr rrrrrrrrB is_batchedpadding_strategyrr+s r__call__Pop2PianoTokenizer.__call__sZ)35"**(E(EUZZ1_:V[\]V^`dKe EIDkDkE !!1 E  E Az ,A,IDOd !..$7% I(($7% IHH $!1"7) rfeature_extractor_output return_midic [[US5=(a [US5=(a [US55nU(d!USRSS:a [S5eU(a[ USSS2S4S:H5USRS:wd&USRSUS RS:waA[S URSS USRSS US RS35eUSRSURS:wa.[S USRSSURS35eO^USRSS:wdUS RSS:wa2[SUSRSSUS RSS35eU(a'[ R "USSS2S4S:H5SnOURS/n/n/nSn[U5GHupXU n U SS2S[ R"[ R "U [UR5:H5S5S-24n USU n US U n U(a|USU nUSU nU S[ R"[ R "US:H5S5S-n U S[ R"[ R "US:H5S5S-n [U 5n [U 5n [U 5n URU U URURS-S-S9nURSRHDnU=R U S- slU=R"U S- slUR%U5 MF UR%U5 XS-- nGM U(a ['XgS.5$['SU05$)a This is the `batch_decode` method for `Pop2PianoTokenizer`. It converts the token_ids generated by the transformer to midi_notes and returns them. Args: token_ids (`Union[np.ndarray, torch.Tensor]`): Output token_ids of `Pop2PianoConditionalGeneration` model. feature_extractor_output (`BatchFeature`): Denotes the output of `Pop2PianoFeatureExtractor.__call__`. It must contain `"beatstep"` and `"extrapolated_beatstep"`. Also `"attention_mask_beatsteps"` and `"attention_mask_extrapolated_beatstep"` should be present if they were returned by the feature extractor. return_midi (`bool`, *optional*, defaults to `True`): Whether to return midi object or not. Returns: If `return_midi` is True: - `BatchEncoding` containing both `notes` and `pretty_midi.pretty_midi.PrettyMIDI` objects. If `return_midi` is False: - `BatchEncoding` containing `notes`. r,attention_mask_beatsteps$attention_mask_extrapolated_beatstep beatstepsrrWzattention_mask, attention_mask_beatsteps and attention_mask_extrapolated_beatstep must be present for batched inputs! But one of them were not present.Nextrapolated_beatstepzbLength mistamtch between token_ids, beatsteps and extrapolated_beatstep! Found token_ids length - z, beatsteps shape - z$ and extrapolated_beatsteps shape - z!Found attention_mask of length - z but token_ids of length - zLength mistamtch of beatsteps and extrapolated_beatstep! Since attention_mask is not present the number of examples must be 1, But found beatsteps length - z", extrapolated_beatsteps length - .ri)rerwrgr)r pretty_midi_objectsr )boolhasattrshaperrrnwhererrr[r2rr|r8rr rrrr)rAr+rrattention_masks_present batch_idx notes_listpretty_midi_objects_listrjrpend_idxeach_tokens_idsrrrrpretty_midi_objectnotes r batch_decodePop2PianoTokenizer.batch_decodePs8#' ,.> ? Z02LM Z02XY# '+CK+P+V+VWX+Y\]+]H  #,-=>q!tDIJ+K8>>qAB+K8>>qA+,CDJJ1MN!**3//!*<)==QRjkvRwR}R}~SARBB::RSj:k:q:qrs:t9uw ((89??BiooVWFXX 78PQa8b8h8hij8k7lmHIRIXIXYZI[H\]Y)5;;A>!C+,CDJJ1MQRR 44L[4Y4_4_`a4b3cdFG_`wGxG~G~@GAFBBCD #!9:J!KAqD!QUV!VWXYZI"+,I #%  ' 2NE'':O-a1r266"((?VYZ^ZhZhViCi:jkl:m3nqr3r1r.rsO0=eDI$<=T$UV[$\ !'+CD^+_`e+f(7O:884&&^rxx8PTU8U/VWX/Y(Z]^(^_ (=XbffRXX&Ja&OPQRSTWXX)%'7O +I$,-B$C !!%!G!G&.#}}!%!2a 7 "H" +66q9?? il* IaL(!!$'@ % + +,> ? 1 $IG3J  :!gh hgz233r)r>rr<r8)Mrz-1102)r)rr r)g)NN)FNNNNNT)T)(__name__ __module__ __qualname____firstlineno____doc__model_input_namesrvocab_files_namesr@propertyrKrPr[rr_rcrnrrur|floatrmrzr7tuplerr%rr rrrrrr rrr__static_attributes__ __classcell__)rFs@rr)r)=s2%&67)   D!!?#S#T#$NcN * ** *  *` !!   **      H:2:::%:bgjnbn:x2::QT@!c!C$J!Z_`cZd!4:>!% E4zzD!1!122E40$6E4$J E4  E4T:>!% %EzzD!1!122%E0$6%E$J %E  %ET166:!%)--126uzzD!1!122T${?O?O:P5QQuo-u3J!33 u $J u  $J u $d{uj(4/uu uv! w4#/w4 w4w4rr))rr:rnumpyrnfeature_extraction_utilsrtokenization_pythonrrrrr utilsr r r r rutils.import_utilsrr% get_loggerrrrrr$r)__all__r5rrrs( 4vv__*   H % \   +,I 4,I 4-I 4X r