ZjՅ"SrSSKrSSKrSSKJr SSKJr SSKJr SSKJ r J r J r SSK J r SS KJrJr SS KJr SS KJrJr SS KJr SS KJrJr SSKJr \R:"\5r"SS\R@5r!"SS\R@5r""SS\RF5r$"SS\RF5r%"SS\5r&\"SS\55r'"SS\'5r(\"SS9"S S!\'55r)\"S"S9"S#S$\'\ 55r*S$S/r+g)%z/PyTorch TrOCR decoder model (based on RoBERTa).N)nn)CrossEntropyLoss)ACT2FN)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)create_bidirectional_maskcreate_causal_mask)GradientCheckpointingLayer))BaseModelOutputWithPastAndCrossAttentions!CausalLMOutputWithCrossAttentions)PreTrainedModel)auto_docstringlogging) TrOCRConfigc^\rSrSrSrS\S\4U4SjjrS S\RS\S \RS-4U4S jjjr S r U=r $) TrOCRLearnedPositionalEmbedding%zF This module learns positional embeddings up to a fixed maximum size. num_embeddings embedding_dimcL>SUl[TU] XR-U5 gN)offsetsuper__init__)selfrr __class__s y/root/GenerationalWealth/GenerationalWealth/venv/lib/python3.13/site-packages/transformers/models/trocr/modeling_trocr.pyr(TrOCRLearnedPositionalEmbedding.__init__*s"  ++5}EN input_idspast_key_values_length position_idsc>Uc]URSSupE[R"X"U-[RURR S9R US5nOURS5n[TU]%X0R-5$)z3`input_ids' shape is expected to be [bsz x seqlen].Nr)dtypedevicer) shapetorcharangelongweightr*expand unsqueezerforwardr)r r%r&r'bszseq_lenr!s r"r3'TrOCRLearnedPositionalEmbedding.forward0s  $??2A.LC <<&(HPUPZPZcgcncncucufS"o (11!4Lw|kk9::r$)r)rN) __name__ __module__ __qualname____firstlineno____doc__intrr-Tensorr3__static_attributes__ __classcell__r!s@r"rr%sWFsF3Fmq ; ;?B ;V[VbVbeiVi ; ;r$rc r^\rSrSrSrS S\S\S\S\S-4U4SjjjrS \R4U4S jjr S r U=r $) TrOCRScaledWordEmbeddingAzT This module overrides nn.Embeddings' forward by multiplying with embeddings scale. rr padding_idx embed_scaleNc2>[TU]XU5 X@lgN)rrrE)r rrrDrEr!s r"r!TrOCRScaledWordEmbedding.__init__Fs  D&r$r%c<>[TU]U5UR-$rG)rr3rE)r r%r!s r"r3 TrOCRScaledWordEmbedding.forwardJswy)D,<,<<r?r@s@r"rBrBAsJ's'3'S'_dgk_k''===r$rBc ^\rSrSrSrSS\S\S\S-4U4Sjjjr\SS\S\S\S-4S jj5r\ R"5SS \ RS \4S jj5r SS \ RS\S \S-4S jjr SrU=r$)"TrOCRSinusoidalPositionalEmbeddingNzDThis module produces sinusoidal positional embeddings of any length.N num_positionsrrDcv>[TU]5 SUlX lX0lUR XU5Ulgr)rrrrrD get_embeddingweights)r rQrrDr!s r"r+TrOCRSinusoidalPositionalEmbedding.__init__Qs5  *&))- T r$rcUS-n[R"S5US- - n[R"[R"U[R S9R 5U*-5n[R"U[R S9R 5RS5URS5-n[R"[R"U5[R"U5/SS9RUS5nUS-S:Xa,[R"U[R"US5/SS9nUbSXBSS24'UR[R"55$) z Build sinusoidal embeddings. This matches the implementation in tensor2tensor, but differs slightly from the description in Section 3.5 of "Attention Is All You Need". ri'r)r)rdimr+N)mathlogr-expr.int64rMr2catsincosviewzerostoget_default_dtype)rrrDhalf_dimembs r"rS0TrOCRSinusoidalPositionalEmbedding.get_embeddingXs !A%hhuoA.ii XU[[AGGISDPQll>=CCEOOPQRUXUbUbcdUeeii338a@EEnVXY 1  !))S%++na"@AqIC  ""#CQ vve--/00r$r%r&cUR5up4URXRU5RUR5nURS-U-nUR bX`R RS5:a+UR X`RUR5UlUR RSURS55RX4S5R5nU$)Nrrr+) size"create_position_ids_from_input_idsrDrbr*rTrSr index_selectr`detach)r r%r&r4r5r'max_posxs r"r3*TrOCRSinusoidalPositionalEmbedding.forwardks ~~' >>yJZJZ\rsvv    ""Q&0 << 7\\->->q-A#A--g7I7I4K[K[\DL LL % %a):):2)> ? D DSSU V ] ] _r$cURU5R5n[R"USS9R U5U-U-nUR 5U-$)z Replace non-padding symbols with their position numbers. Position numbers begin at padding_idx+1. Padding symbols are ignored. This is modified from fairseq's `utils.make_positions`. rrW)ner<r-cumsumtype_asr/)r r%rDr&maskincremental_indicess r"riETrOCRSinusoidalPositionalEmbedding.create_position_ids_from_input_ids}sW||K(,,.$||Da8@@FI__cgg"'')K77r$)rrrDrTrG)r)r7r8r9r:r;r<r staticmethodrSr-no_gradr=r3rir>r?r@s@r"rOrONsNUcU#UCRVJUU1c1#1CRVJ11$ ]]_s$_` 8 847 8QTW[Q[ 8 8r$rOcl^\rSrSrSrSS\S\S\S-S\S-S\S-S \S-S \S-S \S-S \S-4U4S jjjrSS\ RS\ RS-S\ S-S\ RS-S\S-S\ \ R\ RS-\ \ RS-44 Sjjr SrU=r$)TrOCRAttentionz>Multi-headed attention from 'Attention Is All You Need' paper.N embed_dim num_headskdimvdimdropout is_decoderbiasis_cross_attention layer_idxc 2>[T U]5 X lUbUOUUlUbUOUUlX0lX`lX#-UlURU-UR:Xd[SURSUS35eURS-Ul Xpl Xl [R"URX(S9Ul[R"URX(S9Ul[R"X"US9Ul[R"X"US9Ulg)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).g࿩r)rrr{r}r~r|rhead_dim ValueErrorscalingrrrLineark_projv_projq_projout_proj) r configr{r|r}r~rrrrrr!s r"rTrOCRAttention.__init__s " ,D)  ,D) " !.  )T^^;MdnnM]^;b" }}d* $"ii 9@ ii 9@ ii 4@  )TB r$ hidden_stateskey_value_statespast_key_valuesattention_maskoutput_attentionsreturnc USLnUR5upn URU5UR-n Sn Ub][U[5(aFUR R UR5n U(a URn OURn OUn U(aUOUnU(aQUbNU (aGW RURRnU RURRnOURU5nURU5nURUSUR UR"5R%SS5nURUSUR UR"5R%SS5nUbUW R'UUUR5unnU(a.[U[5(aSUR UR'XR -SUR"4nU RXUR UR"5R%SS5n U R("U6n UR("U6nUR("U6nURS5n[*R,"XR%SS55nUR5XR -U U4:wa.[/SXR -U U4SUR535eUbzUR5USU U4:wa#[/S USU U4SUR535eURXR U U5U-nURXR -U U5n[0R2R5USS 9nU(a=URXR U U5nURXR -U U5nOSn[0R2R7UUR6UR8S 9n[*R,"UU5nUR5XR -XR"4:wa5[/S XR XR"4SUR535eURXR XR"5nUR%SS5nUR)XU 5nUR;U5nUU4$) z#Input shape: Batch x Time x ChannelNFr+rrTz$Attention weights should be of size z , but is z!Attention mask should be of size rWptrainingz `attn_output` should be of size )rhrr isinstancer is_updatedgetrcross_attention_cacheself_attention_cachelayerskeysvaluesrrr`r|r transposeupdatereshaper-bmmrr functionalsoftmaxrrr)r rrrrrkwargsrr4tgt_lenr{ query_statesrcurr_past_key_valuescurrent_states key_states value_states proj_shapesrc_len attn_weightsattn_weights_reshaped attn_probs attn_outputs r"r3TrOCRAttention.forwardsg.T9"/"4"4"6i{{=1DLL@   &/+>??,77;;DNNK %+:+P+P(+:+O+O('6$-?)] /"=*-44T^^DIIJ/66t~~FMML^4J;;~6L#b$..$--PZZ[\^_`J',,S"dnndmmT^^_`bcdL*+?+F+FzS_aeaoao+p( L%*_FY*Z*ZAEO..t~~>NN*B > #((t~~t}}U__`acde #++Z8 ''4 #++Z8 //!$yy/C/CAq/IJ    3#7'"J J6nn8LgW^7_6`a %%'(*   %""$a'(BB 7a'8R7SS\]k]p]p]r\st(,,S..'7SVddL',,S>>-A7GTL}},,\r,B  %1$5$5c>>7T[$\ !055cNN6JGU\]L$( !]]**<4<r?r@s@r"ryrysHH  #"' */!%!C!C !C Dj !C Dj !C!C4K!CTk!C!4K!C$;!C!CL15(,.2). d2||d2 ,,-d2 d2  t+ d2  $; d2 u||U\\D0% 2E2LL Md2d2r$ryc^\rSrSrSS\4U4SjjjrSS\RS\RS-S\RS-S\RS-S \S-S \ S-S \ S-4S jjr S r U=r $)TrOCRDecoderLayeriNrc j>[TU]5 URUl[ UURUR UR SUS9UlURUl[URUl URUl [R"UR5UlUR (am[ UURUR UR"UR"UR SSUS9 Ul[R"UR5Ul[R("URUR*5Ul[R("UR*UR5Ul[R"UR5Ulg)NT)r{r|rrr)r{r|r}r~rrrr)rr hidden_sizer{rydecoder_attention_headsattention_dropout self_attnrractivation_function activation_fnactivation_dropoutr LayerNormself_attn_layer_normrcross_attention_hidden_size encoder_attnencoder_attn_layer_normrdecoder_ffn_dimfc1fc2final_layer_norm)r rrr!s r"rTrOCRDecoderLayer.__init__s= ++' nn44,,  ~~ #F$>$>?"(";";$&LL$@!    ... 88777700#'# !D ,.<<+GD (99T^^V-C-CD99V33T^^D " T^^ <r$rrencoder_hidden_statesencoder_attention_maskrr use_cachec Un URUUUUS9up[RRXRURS9nX-nUR U5nSn Ub^Un UR UUUUUS9up[RRXRURS9nX-nURU5nUn URURU55n[RRXRURS9nURU5n[RRXRURS9nX-nURU5nU4n U(aXU 4- n U $)a^ Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. encoder_hidden_states (`torch.FloatTensor`): cross attention input to the layer of shape `(batch, seq_len, embed_dim)` encoder_attention_mask (`torch.FloatTensor`): encoder attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. past_key_values (`Cache`): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. )rrrrrN)rrrrr) rrrrrrrrrrrrr) r rrrrrrrrresidualself_attn_weightscross_attn_weightsoutputss r"r3TrOCRDecoderLayer.forward<s2!,0>>'+)/ ,:, (  --m||VZVcVc-d  0 11-@ " ,$H040A0A+!65 /"3 1B1 -MMM11-< >Gr$) rrrr{rrrrrrrrG)NNNNFT) r7r8r9r:rrr-r=rrr3r>r?r@s@r"rrs"={"="=N/3596:(,).!%G||G t+G %||d2 G !& t 3 G  G $;G$;GGr$rc.\rSrSr%\\S'SrSrS/rSr g)TrOCRPreTrainedModelirmodelTrN) r7r8r9r:r__annotations__base_model_prefixsupports_gradient_checkpointing_no_split_modulesr>rr$r"rrs &*#,-r$rcR^\rSrSrSrS\4U4SjjrSSjrSrU=r $) TrOCRDecoderiz Transformer decoder consisting of *config.decoder_layers* layers. Each layer is a [`TrOCRDecoderLayer`] Args: config: TrOCRConfig rc |>[TU]U5 URUlURUlUR UlUR(a [R"UR5OSn[URURUR US9Ul UR(a&[UR UR5UlO@[%UR UR -S-URUR 5UlUR&(a&[(R*"UR5UlOSUl[(R,"[/UR05Vs/sH n[3XS9PM sn5UlSUlUR95 gs snf)NrLrKr)rF)rrrdecoder_layerdrop layerdrop pad_token_idrDscale_embeddingrYsqrtrrB vocab_size embed_tokensuse_learned_position_embeddingsrmax_position_embeddingsembed_positionsrOlayernorm_embeddingrr ModuleListrangedecoder_layersrrgradient_checkpointing post_init)r rrEir!s r"rTrOCRDecoder.__init__sT  ~~ 11!..7=7M7Mdii 2 23SV 4   v1143C3CQ\   1 1#B6CaCacicucu#vD #E..1A1AAAE""  $D  % %')||F4F4F'GD $'+D $mmUZ[a[p[pUq$rUqPQ%6v%KUq$rs &+#  %ssF9c UbUOURRnU bU OURRn UbUOURRnU bU OURRn UbUb [ S5eUb"Un UR SU RS5nOUb USS2SS2S4n O [ S5eUR(a/UR(aU(a[RS5 SnU(abUc_UcURR(a.[[URS9[URS95O[URS9nUbUR5OSn UcUR!U5nURR"(aUR%XS 9nOUR%XS 9nXn-nUR&bUR'U5n[(R*R-XR,URS 9n[/URUUUS 9nUbUb[1URUUUS 9nU (aS OSnU(aS OSnU(aUbS OSn[3UR45H}unnU (aUU4- nUR(a([6R8"/5nUUR::aMLU"UUUUUUUS9nUSnU(dMfUUS4- nUcMtUUS4- nM U (aUU4- nU (d[=SXUUU455$[?UUUUUS9$)a Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, encoder_sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the decoder. encoder_attention_mask (`torch.LongTensor` of shape `(batch_size, encoder_sequence_length)`, *optional*): Mask to avoid performing cross-attention on padding tokens indices of encoder input_ids. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. NzTYou cannot specify both decoder_input_ids and decoder_inputs_embeds at the same timer+zEYou have to specify either decoder_input_ids or decoder_inputs_embedsz^`use_cache = True` is incompatible with gradient checkpointing. Setting `use_cache = False`...F)rr)r&r)r inputs_embedsrr)rrrrr)rrrrrrc30# UH nUcMUv M g7frGr).0vs r" 'TrOCRDecoder.forward..ZsrArs )last_hidden_staterr attentionscross_attentions) rroutput_hidden_statesr return_dictrr`r,rrlogger warning_onceis_encoder_decoderr rget_seq_lengthrrrrrrrr r enumeraterr-randrrr)r r%rrrrrrrrrrinputr& embed_posrall_hidden_statesall_self_attnsall_cross_attentionsidx decoder_layerdropout_probability layer_outputss r"r3TrOCRDecoder.forwardsV@2C1N-TXT_T_TqTq$8$D $++JjJj "+!6IDKKst t  "E!r5;;r?;I  &!!Q(+Ede e  & &4==##t" 0)4 8V8V$L $DlZ^ZeZeFfg!5  FUE`!?!?!Afg   --i8M ;; 6 6,,U,bI,,Y,fI%1  # # / 44]CM --m||VZVcVc-d +;;')+   ! ,1G1S%>{{+5&; & "#7BD0d&7r?r@s@r"rrsA{B"#!p p r$ra The TrOCR Model with a language modeling head. Can be used for summarization. This wrapper class is a helper class to correctly load pretrained checkpoints when the causal language model is used in combination with the [`EncoderDecoderModel`] framework. ) custom_introc.^\rSrSrU4SjrSrSrU=r$)TrOCRDecoderWrapperihcd>[TU]U5 [U5UlUR 5 grG)rrrdecoderrr rr!s r"rTrOCRDecoderWrapper.__init__ps&  #F+  r$c&UR"U0UD6$rGr)r argsrs r"r3TrOCRDecoderWrapper.forwardus||T,V,,r$r)r7r8r9r:rr3r>r?r@s@r"rrhs --r$rzy The TrOCR Decoder with a language modeling head. Can be used as the decoder part of [`EncoderDecoderModel`] and cf^\rSrSrSS0rU4SjrSrSrSrSr \ SS \ RS -S \ RS -S \ RS -S \ RS -S\S -S\ RS -S\ RS -S\S -S\S -S\S -S\S -S\\-4Sjj5rSrU=r$)TrOCRForCausalLMiyzoutput_projection.weightz!model.decoder.embed_tokens.weightc>SUlSUl[TU] U5 [ U5Ul[ R"URURSS9Ul UR5 g)NTFr) rrrrrrrrrroutput_projectionrrs r"rTrOCRForCausalLM.__init__sZ $)!  (0 !#6+=+=v?P?PW\!] r$cBURRR$rGrrrr s r"get_input_embeddings%TrOCRForCausalLM.get_input_embeddingsszz!!...r$c8XRRlgrGr)r values r"set_input_embeddings%TrOCRForCausalLM.set_input_embeddingss*/ 'r$cUR$rGrr s r"get_output_embeddings&TrOCRForCausalLM.get_output_embeddingss%%%r$cXlgrGr()r new_embeddingss r"set_output_embeddings&TrOCRForCausalLM.set_output_embeddingss!/r$Nr%rrrrrlabelsrrrrrc HU bU OURRn U bU OURRn U bU OURRn URR UUUUUUUU U U S9 n UR U S5nSnUbF[5nU"URSURR5URS55nU (dU4U SS-nUbU4U-$U$[UUU RU RU RU RS9$)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import ( ... TrOCRConfig, ... TrOCRProcessor, ... TrOCRForCausalLM, ... ViTConfig, ... ViTModel, ... VisionEncoderDecoderModel, ... ) >>> import httpx >>> from io import BytesIO >>> from PIL import Image >>> # TrOCR is a decoder model and should be used within a VisionEncoderDecoderModel >>> # init vision2text model with random weights >>> encoder = ViTModel(ViTConfig()) >>> decoder = TrOCRForCausalLM(TrOCRConfig()) >>> model = VisionEncoderDecoderModel(encoder=encoder, decoder=decoder) >>> # If you want to start from the pretrained model, load the checkpoint with `VisionEncoderDecoderModel` >>> processor = TrOCRProcessor.from_pretrained("microsoft/trocr-base-handwritten") >>> model = VisionEncoderDecoderModel.from_pretrained("microsoft/trocr-base-handwritten") >>> # load image from the IAM dataset >>> url = "https://fki.tic.heia-fr.ch/static/img/a01-122-02.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())).convert("RGB") >>> pixel_values = processor(image, return_tensors="pt").pixel_values >>> text = "industry, ' Mr. Brown commented icily. ' Let us have a" >>> # training >>> model.config.decoder_start_token_id = processor.tokenizer.eos_token_id >>> model.config.pad_token_id = processor.tokenizer.pad_token_id >>> model.config.vocab_size = model.config.decoder.vocab_size >>> labels = processor.tokenizer(text, return_tensors="pt").input_ids >>> outputs = model(pixel_values, labels=labels) >>> loss = outputs.loss >>> round(loss.item(), 2) 5.30 >>> # inference >>> generated_ids = model.generate(pixel_values) >>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0] >>> generated_text 'industry, " Mr. Brown commented icily. " Let us have a' ```N) r%rrrrrrrrrrr+r)losslogitsrrrr)rrrrrrrrr`rrrrrr)r r%rrrrrr/rrrrrrr2r1loss_fctoutputs r"r3TrOCRForCausalLM.forwardsBP2C1N-TXT_T_TqTq$8$D $++JjJj &1%r?r@s@r"rrys? 56YZ /0&0.2.2:>:>(,26*.!%)-,0#'m ##d*m  t+m  %0047 m !& 0 04 7 m  m ((4/m   4'm $;m  $;m #Tkm D[m  2 2m m r$r),r;rYr-rtorch.nnr activationsr cache_utilsrrr generationr masking_utilsr r modeling_layersr modeling_outputsrrmodeling_utilsrutilsrrconfiguration_trocrr get_loggerr7r EmbeddingrrBModulerOryrrrrr__all__rr$r"rGs$6 %!CC)J9l-,,   H %;bll;8 =r|| =9898xJ2RYYJ2Zl2l^.?..W 'W t-.-- H +_H  H V 5 6r$