Zj`2SSKrSSKJr SSKJr SSKJr SSKJr SSKJ r J r SSK J r SS K JrJrJrJr SS KJr SS KJr S S KJrJr S SKJrJr S SKJrJr SSKJ r \RB"\"5r#"SS\5r$"SS\5r%"SS\5r&\"SS9"SS\55r'"SS\5r(\"SS9"SS \55r)/S!Qr*g)"N)nn)ACT2FN)Cache)create_bidirectional_mask)BaseModelOutputWithPoolingCausalLMOutputWithPast)Unpack)TransformersKwargsauto_docstringcan_return_tuplelogging)merge_with_config_defaults)capture_outputs)Qwen2AudioEncoderQwen2AudioPreTrainedModel)VoxtralForConditionalGenerationVoxtralMultiModalProjector)WhisperAttentionWhisperEncoderLayer)AudioFlamingo3Configc\rSrSrSrg)AudioFlamingo3Attention'N__name__ __module__ __qualname____firstlineno____static_attributes__rڊ/root/GenerationalWealth/GenerationalWealth/venv/lib/python3.13/site-packages/transformers/models/audioflamingo3/modular_audioflamingo3.pyrr'r$rc\rSrSrSrg)AudioFlamingo3EncoderLayer+rNrrr$r%r(r(+r&r$r(c\rSrSrSrg)AudioFlamingo3PreTrainedModel/rNrrr$r%r+r+/r&r$r+zT The audio model from AudioFlamingo3 without any head or projection on top.  custom_introc \rSrSrSr\\S.r\\ S S\ RS\ RS-S\ \ -4Sjj55rS rg) AudioFlamingo3Encoder3zQ AudioFlamingo3 encoder: Whisper encoder, average pool (time/2), then LayerNorm. ) hidden_states attentionsNinput_featuresinput_features_maskreturnc URSS- S-S-nURS5nUS- S-S-n[R"XARS9USS2S4:n[ R RURU55n[ R RURU55nURSSS5nX`RR-n[ R RXpRURS9n[UR UUS9nUR"HJn UR=(a# [R$"/5UR&:n U (aMBU "Xx5nML URSSS5nUR)U5RSSS5nUR+U5n[-US 9$) a( Args: input_features (`torch.FloatTensor` of shape `(batch_size, feature_size, sequence_length)`): Log-Mel features extracted from raw audio. Use the processor/feature extractor to compute and pad these features from waveform input. input_features_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding feature indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. rrdeviceNr)ptraining)config inputs_embedsattention_mask)last_hidden_state)shapesumtorcharanger:r functionalgeluconv1conv2permuteembed_positionsweightdropoutr<rr=layersrand layerdrop avg_pooler layer_normr) selfr4r5kwargsseq_leninput_features_lengthsr>r2r?layerdrops r%forwardAudioFlamingo3Encoder.forwardBs("''+a/A59!4!8!8!<"81"<!BQ!F#ll7;P;PQTjklnrkrTss **4::n+EF  **4::m+DE %--aA6 &(<(<(C(CC  --m||VZVcVc-d 2;;'. [[E==DUZZ^dnn%DD4 %m D ! &--aA6  6>>q!QG  6 )+  r$rN)rr r!r"__doc__r(r_can_record_outputsrrrCTensortuplerrXr#rr$r%r0r03sg  4-  483  3 #\\D03 + + 3  3 r$r0c4^\rSrSrSrS\4U4SjjrSrU=r$)!AudioFlamingo3MultiModalProjectorzz Audio adaptor (small MLP) that projects AudioFlamingo3Encoder features to the LLM embedding space so they can replace `` tokens. r=c>[TU]5 [R"URR UR R URS9Ul[URUl [R"UR R UR R URS9Ul g)N)bias) super__init__rLinear audio_config hidden_size text_configprojector_biaslinear_1rprojector_hidden_actactlinear_2rRr= __class__s r%re*AudioFlamingo3MultiModalProjector.__init__s     + +V-?-?-K-KRXRgRg  &556    * *F,>,>,J,JQWQfQf  r$)rmrkrn) rr r!r"r[rrer# __classcell__rps@r%r`r`zs  3  r$r`z The AudioFlamingo3 model which consists of a fine-tuned Whisper encoder, a multi-modal projector and a Qwen2 language model. c^\rSrSrSrSrSrSrU4Sjr\ \ "SS9S\ RS\ RS \\S \\-4S j55r\ \ SS \ R&S-S\ RS-S\ RS-S \ RS-S\ R&S-S\S-S\ RS-S\ R&S-S\S-S\\ R-S \\S \4Sjj55rSS.S\4U4SjjjrSrU=r$)&AudioFlamingo3ForConditionalGenerationTNc$>[TU]U5 grZ)rdreros r%re/AudioFlamingo3ForConditionalGeneration.__init__s  r$zThis method is used to get the audio embeddings from input features (a log mel spectrogram), meaning inferring the audio encoder and the multi-modal projector.r-r4r5rSr6c UR"U4USS.UD6nURUR5nURS5R [ R 5nURRU5upx[ R"URSURS9SSS24USS2S4:n XYR UR5Ul U$)a input_features (`torch.FloatTensor`): Float values of mel features extracted from the raw speech waveform. Raw speech waveform can be obtained by loading a `.flac` or `.wav` audio file into an array of type `list[float]` or a `numpy.ndarray`, *e.g.* via the soundfile library (`pip install soundfile`). To prepare the array into `input_features`, the [`AutoFeatureExtractor`] should be used for extracting the mel features, padding and conversion into a tensor of type `torch.FloatTensor`. See [`~WhisperFeatureExtractor.__call__`] input_features_mask (`torch.Tensor` of shape `(batch_size, feature_sequence_length)`): Mask to avoid performing attention on padded feature indices. T)r5 return_dictr8rr9N) audio_towermulti_modal_projectorr@rBtorClong _get_feat_extract_output_lengthsrDrAr: pooler_output) rRr4r5rS audio_output audio_embeds input_lengths_ post_lengths valid_masks r%get_audio_features9AudioFlamingo3ForConditionalGeneration.get_audio_featuress*''  0CQU Y_ 11,2P2PQ ,//366uzzB **KKMZ\\,"4"4Q"7 @S@STUY[\U\]`lmnptmt`uu %1-- @S@S2T%U "r$ input_idsr? position_idspast_key_valuesr>labels use_cachelogits_to_keepc UcUR5"U5nUbUbURX#SS9Rn XRR:HR S5n UR U RUR5U RUR55nUR"SUUUUUU U S.U D6nU$)a input_features_mask (`torch.Tensor` of shape `(batch_size, feature_sequence_length)`): Mask to avoid performing attention on padding feature indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AudioFlamingo3ForConditionalGeneration, AutoProcessor >>> model_id = "nvidia/audio-flamingo-3-hf" >>> processor = AutoProcessor.from_pretrained(model_id) >>> model = AudioFlamingo3ForConditionalGeneration.from_pretrained(model_id, device_map="auto") >>> conversations = [ >>> [ >>> { >>> "role": "user", >>> "content": [ >>> {"type": "text", "text": "Transcribe the input speech."}, >>> { >>> "type": "audio", >>> "path": "https://huggingface.co/datasets/nvidia/AudioSkills/resolve/main/assets/t_837b89f2-26aa-4ee2-bdf6-f73f0dd59b26.wav", >>> }, >>> ], >>> } >>> ], >>> [ >>> { >>> "role": "user", >>> "content": [ >>> { >>> "type": "text", >>> "text": "This track feels really peaceful and introspective. What elements make it feel so calming and meditative?", >>> }, >>> {"type": "audio", "path": "https://huggingface.co/datasets/nvidia/AudioSkills/resolve/main/assets/FPSbCAANfbJLVSwD.mp3"}, >>> ], >>> } >>> ], >>> ] >>> inputs = processor.apply_chat_template( >>> conversations, >>> tokenize=True, >>> add_generation_prompt=True, >>> return_dict=True, >>> ).to(model.device) >>> outputs = model.generate(**inputs, max_new_tokens=500) >>> decoded_outputs = processor.batch_decode( >>> outputs[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True >>> ) >>> print(decoded_outputs) ["The spoken content of the audio is...", "The track's calming and meditative feel can be attributed to..."] ```T)rzr8)r>r?rrrrrr) get_input_embeddingsrrr=audio_token_id unsqueezemasked_scatterr}r:language_model)rRrr4r5r?rrr>rrrrSraudio_token_maskoutputss r%rX.AudioFlamingo3ForConditionalGeneration.forwards^   557 BM  %)*?22>dh2iwwL!*[[-G-G GRRSUV )88 ##M$8$89URSS5nURSS5n[TU]"U0UD6nU(dURSS5(dUbXFS'UbXVS'U$)Nr4r5rF)poprdprepare_inputs_for_generationget)rRrargsrSr4r5 model_inputsrps r%rDAudioFlamingo3ForConditionalGeneration.prepare_inputs_for_generation"ss$4d;$jj)>Ewrs  ! 6R&RR75cL>   H % .  !4  $=  ? -?  ? D (B " ^-L^  ^B or$