Z j S r SSKrSSKJr SSKrSSKJr SSKJr SSKJ r
SSK J r SS K
JrJrJr SS
KJr SS KJrJr SS KJr SS
KJr SSKJrJ r J r J r J r SSK J!r!J"r" SSK#J$r$ SSK%J&r&J'r'J(r(J)r)J*r* SSK+J,r, SSK-J.r.J/r/ SSK0J1r1 \*Rd " \35 r4S\Rj S\6S\64S jr7 " S S\Rp 5 r9 " S S \Rp 5 r: S>S \Rv S \Rj S \Rj S!\Rj S"\Rj S- S#\ [ TU ] X5 g N)super__init__)selfr4 r5 __class__s r. r9 -BlenderbotLearnedPositionalEmbedding.__init__F s
7r0 Ninput_ids_shapepast_key_values_length position_idsc > UcB USS u pE[ R " X"U- [ R U R R S9n[
TU ] U5 $ )z3`input_ids_shape` is expected to be [bsz x seqlen].N )dtypedevice)torcharangelongweightrC r8 forward)r: r= r> r? bszseq_lenr; s r. rH ,BlenderbotLearnedPositionalEmbedding.forwardI sX *2A.LC <<&(HPUPZPZcgcncncucu L w|,,r0 )r N)
__name__
__module__ __qualname____firstlineno____doc__intr9 rD SizeTensorrH __static_attributes__
__classcell__r; s @r. r2 r2 A sT 8s 83 8 qu - $zz -CF -Z_ZfZfimZm - -r0 r2 c
r ^ \ rS rSrSrS S\S\S\S\S- 4U 4S jjjrS \R 4U 4S
jjr
S r U =r $ )
BlenderbotScaledWordEmbeddingV zT
This module overrides nn.Embeddings' forward by multiplying with embeddings scale.
r4 r5 padding_idx embed_scaleNc 2 > [ TU ] XU5 X@l g r7 )r8 r9 r\ )r: r4 r5 r[ r\ r; s r. r9 &BlenderbotScaledWordEmbedding.__init__[ s
D&r0 r# c < > [ TU ] U5 U R - $ r7 )r8 rH r\ )r: r# r; s r. rH %BlenderbotScaledWordEmbedding.forward_ s wy)D,<,<<L==(( 6??([L,,|3K ''1-88:K $$r0 c $ ^ \ rS rSrSr SS\S\S\S\S\S \S
\S- S \S- 4U 4S jjjr SS
\
R S\
R S- S\ S- S\
R S- S\
\ S\\
R \
R S- 4 4 S jjrSrU =r$ )BlenderbotAttention z=Multi-headed attention from 'Attention Is All You Need' paperN embed_dim num_headsrj
is_decoderbias is_causalconfig layer_idxc t > [ T U ] 5 Xl X l X0l X- U l Xpl U R
U- U R :w a [ SU R SU S35 eU R
S- U l X@l X`l
Xl Uc> U R ( a- [ R SU R R S35 [ R" " XUS9U l [ R" " XUS9U l [ R" " XUS9U l [ R" " XUS9U l g )Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).rm zInstantiating a decoder z without passing `layer_idx` is not recommended and will lead to errors during the forward call, if caching is used. Please make sure to provide a `layer_idx` when creating this class.r )r8 r9 r r rj head_dimr r+ ri r r r logger warning_oncer; rM r Lineark_projv_projq_projout_proj)
r: r r rj r r r r r r; s
r. r9 BlenderbotAttention.__init__ s "" !.
MMI %$.. 8 MdnnM]$YKr3
}}d* $"" *4>>+B+B*C D, ,
ii 4@ ii 4@ ii 4@ )TB
r0
hidden_stateskey_value_statespast_key_valuesrh rk returnc , USLnUR SS n/ UQSPU R P7nU R U5 R U5 R SS5 n Sn
Ub] [ U[ 5 ( aF UR R U R 5 n
U( a
UR n OUR n OUn U( a UOUn U( aQ UbN U
( aG W R U R R n
U R U R R nOU R U 5 n
U R! U 5 n/ U R SS QSPU R P7nU
R U5 R SS5 n
UR U5 R SS5 nUbS W R# XU R 5 u pU( a. [ U[ 5 ( a SUR U R ' [$ R& " U R( R* [, 5 nU" U U U
UU4U R. ( d SO U R0 U R2 S.UD6u nnUR4 " / UQSP76 R7 5 nU R9 U5 nUU4$ ) z#Input shape: Batch x Time x ChannelNr' r! rA FT )rj ri ) r) r r viewru
isinstancer
is_updatedgetr cross_attention_cacheself_attention_cachelayerskeysvaluesr r updater
get_interfacer _attn_implementationr{ rr rj ri reshaperx r )r: r r r rh rk is_cross_attention input_shape hidden_shape query_statesr curr_past_key_valuescurrent_states
key_states value_stateskv_shapeattention_interfacerz ry s r. rH BlenderbotAttention.forward sc .T 9 $))#2. 88b8$--8 {{=166|DNNqRST
&/+>?? , 7 7 ; ;DNN K
%+:+P+P(+:+O+O('6$-?)] /"=*-44T^^DIIJ/66t~~FMML ^4J ;;~6LF --cr2FBF
FH# 2<(?(M(M KK , ,.E)
%8 %
$}}C$,,LL %
%
! \ "));;;;FFH mmK0 L((r0 )
r rj r r r r r r r r r ri r )r FTFNNNNN)rM rN rO rP rQ rR rc boolr" r9 rD rT r r r tuplerH rU rV rW s @r. r} r} s G *. $%C%C %C %C
%C
%C %C !4'%C :%C %CT 15(,.2 H) ||H) ,,-H) H)
t+ H) -.H)
u||U\\D 00 1H) H)r0 r} c ^ \ rS rSrS\4U 4S jjrS\R S\R S\\ S\R 4S jr
S r U =r $ )
BlenderbotEncoderLayer r c j > [ TU ] 5 UR U l [ U R UR
UR US9U l [ R " U R 5 U l
UR U l [ UR U l UR U l [ R " U R UR" 5 U l [ R " UR" U R 5 U l [ R " U R 5 U l g )N)r r rj r )r8 r9 d_modelr r} encoder_attention_headsattention_dropout self_attnr LayerNormself_attn_layer_normrj r activation_function
activation_fnactivation_dropoutr encoder_ffn_dimfc1fc2final_layer_normr: r r; s r. r9 BlenderbotEncoderLayer.__init__ s
,nn 44,,
%'LL $@! ~~ #F$>$> ?"(";"; 99T^^V-C-CD99V 3 3T^^D " T^^ < r0 r rh rk r c UnU R U5 nU R " SUUS.UD6u p[ R R XR U R
S9nXA- nUnU R
U5 nU R U R U5 5 n[ R R XR U R
S9nU R U5 n[ R R XR U R
S9nXA- nUR [ R :X aC [ R " UR 5 R S-
n[ R " X* US9nU$ )a
Args:
hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)`
attention_mask (`torch.FloatTensor`): attention mask of size
`(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values.
)r rh rp i )minmaxrL )r r r rv rj rr r r r r r rB rD float16finfor clamp)r: r rh rk residual_ clamp_values r. rH BlenderbotEncoderLayer.forward
sD ! 11-@
>>
')
--m||VZVcVc-d
0
--m<
**488M+BC
--m?V?Vaeanan-o
/
--m||VZVcVc-d
0
%-- / ++m&9&9:>>EK !KK
<[ YM r0 ) r r rj r r r r r r )
rM rN rO rP r" r9 rD rT r r rH rU rV rW s @r. r r sR = / =$" ||" " +, "
" " r0 r c ^ \ rS rSrSS\S\S- 4U 4S jjjr SS\R S\R S- S\R S- S \R S- S
\ S- S \
S- S \ \ S
\R 4S jjr
SrU =r$ )BlenderbotDecoderLayeri0 Nr r c
> [ TU ] 5 UR U l [ U R UR
UR SSUUS9U l UR U l [ UR U l UR U l [ R " U R 5 U l [ U R UR
UR SUUS9U l [ R " U R 5 U l [ R$ " U R UR& 5 U l [ R$ " UR& U R 5 U l [ R " U R 5 U l g )NT)r r rj r r r r )rj r r r )r8 r9 r r r} decoder_attention_headsr r rj r r r r r r r encoder_attnencoder_attn_layer_normr decoder_ffn_dimr r r )r: r r r; s r. r9 BlenderbotDecoderLayer.__init__1 s
,nn 44,,
~~ #F$>$> ?"(";"; $&LL $@! / NN * *,,
(*||DNN'C$99T^^V-C-CD99V 3 3T^^D " T^^ < r0 r rh encoder_hidden_statesencoder_attention_maskr use_cacherk r c UnU R U5 nU R " SUUUS.UD6u p[ R R XR U R
S9nX- nUbb UnU R
U5 nU R " SUUUUS.UD6u p[ R R XR U R
S9nX- nUnU R U5 nU R U R U5 5 n[ R R XR U R
S9nU R U5 n[ R R XR U R
S9nX- nU$ )a
Args:
hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)`
attention_mask (`torch.FloatTensor`): attention mask of size
`(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values.
encoder_hidden_states (`torch.FloatTensor`):
cross attention input to the layer of shape `(batch, seq_len, embed_dim)`
encoder_attention_mask (`torch.FloatTensor`): encoder attention mask of size
`(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values.
past_key_values (`Cache`): cached past key and value projection states
)r r rh rp )r r rh r rL )
r r r rv rj rr r r r r r r r )
r: r rh r r r r rk r r s
r. rH BlenderbotDecoderLayer.forwardP s * ! 11-@
>>
' +)
--m||VZVcVc-d
0
! ,$H 8 8 GM # 0 0 +!6 5 /
M MM 1 1-< [ TU ] U5 [ U[ 5 ( a! [ R
" UR 5 g g r7 )r8
_init_weightsr "BlenderbotForConditionalGenerationinitzeros_final_logits_bias)r: rd r; s r. r 'BlenderbotPreTrainedModel._init_weights s6
f% f @ A A KK00 1 Br0 c U R R n[ R " / SQSSSSU//U R S9nUR U5 UUS.nU$ )N)r
rA r rA rC )rh r# decoder_input_ids)r r$ rD tensorrC ne)r: pad_tokenr# dummy_inputss r. r &BlenderbotPreTrainedModel.dummy_inputs sZ KK,, LL"2Q2q)4L!MVZVaVab 'll9 5"!*
r0 rL )rM rN rO rP r" __annotations__base_model_prefix supports_gradient_checkpointing_supports_flash_attn_supports_sdpa_supports_flex_attn_can_compile_fullgraphr propertyr rU rV rW s @r. r r sE &*# N !2
r0 r c ^ \ rS rSrSr\\S.rS\4U 4S jjr \
\ \ SS\
R S- S\
R S- S \
R S- S
\\ S \4
S jj5 5 5 rS
rU =r$ )BlenderbotEncoderi z
Transformer encoder consisting of *config.encoder_layers* self attention layers. Each layer is a
[`BlenderbotEncoderLayer`].
Args:
config: BlenderbotConfig
embed_tokens (nn.Embedding): output embedding
)r
attentionsr c > [ TU ] U5 UR U l UR U l UR
nUR U l UR U l UR ( a [ R " U5 OSn[ UR X R US9U l [! UR U5 U l [$ R& " [) UR* 5 Vs/ s H n[- U5 PM sn5 U l [$ R0 " UR
5 U l SU l U R7 5 g s snf )Nrb ra F) r8 r9 rj encoder_layerdrop layerdropr r$ r[ max_position_embeddingsmax_source_positionsscale_embeddingmathsqrtrY
vocab_size embed_tokensr2 embed_positionsr
ModuleListrangeencoder_layersr r r
layer_normgradient_checkpointing post_init)r: r r r\ r r; s r. r9 BlenderbotEncoder.__init__ s
~~ 11NN !..$*$B$B!.4.D.Ddii *# 9 y*:*:
D * *
mmUSYShShMi$jMi%;F%CMi$jk ,,v~~6&+# %ks 0ENr# rh
inputs_embedsrk r c D US L US L- ( a [ S5 eUc U R U5 nUR 5 S S nU R U5 nX6- n[ R
R
XpR U R S9n[ U R UUS9n[ U R 5 HR u pSn
U R ( a' [ R " / 5 n XR : a Sn
U
( a MH U " UU40 UD6nMT U R U5 n[! US9$ )Nz:You must specify exactly one of input_ids or inputs_embedsr' rp )r r rh FT)last_hidden_state)r+ r rs r r rv rj rr r
r enumerater rD randr r r ) r: r# rh r rk r embed_posr idx
encoder_layerto_dropdropout_probabilitys r. rH BlenderbotEncoder.forward s
-t"; < YZ Z - -i 8M#((*3B/ ((5 %1
--m||VZVcVc-d
2;;')
#,DKK"8 CG}}&+jjn#& 7 "G7 -!"! !
#9
6
+
r0 ) rj r r r
r r r r r[ r )rM rN rO rP rQ r r} _can_record_outputsr" r9 r r r rD
LongTensorrT FloatTensorr r r rH rU rV rW s @r. r r s 0)
/ 4 .2.226 ,
##d*,
t+,
((4/ ,
+, ,
,
,
r0 r c J ^ \ rS rSrSr\\" \SSS9\" \SSS9S.rS\ 4U 4S jjr
\ \ \
SS \R S
- S \R S
- S
\R" S
- S\R S
- S\S
- S\R" S
- S\S
- S\\ S\4S jj5 5 5 rSrU =r$ )BlenderbotDecoderi z
Transformer decoder consisting of *config.decoder_layers* layers. Each layer is a [`BlenderbotDecoderLayer`]
Args:
config: BlenderbotConfig
embed_tokens (nn.Embedding): output embedding
r! r )index
layer_namer )r r cross_attentionsr c
> [ TU ] U5 UR U l UR U l UR
U l UR U l UR ( a [ R " UR 5 OSn[ UR UR U R US9U l [! UR UR 5 U l [$ R& " [) UR* 5 Vs/ s H n[- XS9PM sn5 U l [$ R0 " UR 5 U l SU l U R7 5 g s snf )Nrb ra )r F) r8 r9 rj decoder_layerdropr r$ r[ r max_target_positionsr r r r rY r r r2 r r r r
decoder_layersr r r r r
r )r: r r\ ir; s r. r9 BlenderbotDecoder.__init__ s
~~ 11!..$*$B$B!393I3Idii/s 9 v~~t/?/?[
D * * NN
mmBGH]H]B^ _B^Q
#F
8B^ _
,,v~~6&+#
`s E Nr# rh r r r r r rk r c US L US L- ( a [ S5 eUc U R U5 nU( ab Uc_ Uc U R R ( a. [ [ U R S9[ U R S95 O[ U R S9nUR
5 S S u pUb UR 5 OSn [ R " XR S9U - n Uc2 [ 5 ( d# X- n
[ R " XUR S9n[ U[ 5 ( a UR OUn[ U R UUUS9n[! U R UUUS9nU R# X4XS9n Xl- n[$ R& R) UU R( U R* S 9n[- U R. 5 Hj u nnU R* ( a( [ R0 " / 5 nUU R2 : a M? U" UUU4UUUS
.UD6n[ U[4 5 ( a US OUnMl U R7 U5 n[9 UUS 9$ ) NzTYou cannot specify both decoder_input_ids and decoder_inputs_embeds at the same time)r r' r r )r r rh r )r r rh r )r? rp )r r r )r r ) r+ r r is_encoder_decoderr r
rs get_seq_lengthrD rE rC r onesr r r r
r r rv rj rr r r r r r r r )r: r# rh r r r r r rk
batch_size
seq_lengthr> r? mask_seq_lengthself_attn_cache causal_maskr r
decoder_layerr
layer_outputss r. rH BlenderbotDecoder.forward+ sS
-t"; < st t - -i 8M 0 )4 8V8V $L $DlZ^ZeZeFfg!5
"/!3!3!5cr!:
ETE` !?!?!Afg ||J7K7KLOee !*B*D*D 4 AO "ZZ
ML`L` aN /+>??
0 0 );;') +
";;;'1"7 "
++
$&< ,
&4
--mt||VZVcVc-d
"+DKK"8 C }}&+jjn#& 7 ) % (> /#
M 1;=%0P0PM! ,VcM! #9&
6
8 + +
r0 ) rj r r r
r r r r% r[ )NNNNNNN)rM rN rO rP rQ r r r} r r" r9 r r r rD r rT r r r r r r rH rU rV rW s @r. r r s 0$%8kZ *+>aTb c / 2 .2.2:>:>(,26!%U
##d*U
t+U
% 0 04 7 U
!& 0 04 7 U
U
((4/U
$;U
+,U
3U
U
r0 r c Z ^ \ rS rSrSSS.rS\4U 4S jjrS rS r\ \
SS \ R S- S
\ R S- S \ R S- S \ R S- S
\S- S\S- S\ R S- S\ R S- S\S- S\\ S\4S jj5 5 rSrU =r$ )BlenderbotModeli z
shared.weight)zencoder.embed_tokens.weightzdecoder.embed_tokens.weightr c J > [ TU ] U5 UR UR p2UR ( a [
R " UR 5 OSn[ X1R X$S9U l [ U5 U l [ U5 U l
U R 5 g )Nrb ra )r8 r9 r$ r r r r r rY sharedr encoderr decoderr )r: r r[ r r\ r; s r. r9 BlenderbotModel.__init__ sv
"("5"5v7H7HZ393I3Idii/s 3JP[u (0 (0
r0 c U R $ r7 )r8 r: s r. get_input_embeddings$BlenderbotModel.get_input_embeddings s {{r0 c | Xl U R U R l U R U R l g r7 )r8 r9 r r: r: rg s r. set_input_embeddings$BlenderbotModel.set_input_embeddings s' $(KK !$(KK !r0 Nr# rh r decoder_attention_maskencoder_outputsr r decoder_inputs_embedsr rk r c
Uc U R " S UUUS.U
D6nOK[ U[ 5 ( d6 [ US [ U5 S: a US OS[ U5 S: a US OSS9nU R " S UUUS UUUU S.U
D6n [ U R U R U R U R U R UR UR UR S9$ )
a
decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*):
Indices of decoder input sequence tokens in the vocabulary.
Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and
[`PreTrainedTokenizer.__call__`] for details.
[What are decoder input IDs?](../glossary#decoder-input-ids)
Blenderbot uses the `bos_token_id` as the starting token for `decoder_input_ids` generation. If
`past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see
`past_key_values`).
decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*):
Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also
be used by default.
Example:
```python
>>> from transformers import AutoTokenizer, BlenderbotModel
>>> model = BlenderbotModel.from_pretrained("facebook/blenderbot-400M-distill")
>>> tokenizer = AutoTokenizer.from_pretrained("facebook/blenderbot-400M-distill")
>>> inputs = tokenizer("Studies have been shown that owning a dog is good for you", return_tensors="pt")
>>> decoder_input_ids = tokenizer("Studies show that", return_tensors="pt").input_ids # Batch size 1
>>> outputs = model(input_ids=inputs.input_ids, decoder_input_ids=decoder_input_ids)
>>> last_hidden_states = outputs.last_hidden_state
>>> list(last_hidden_states.shape)
[1, 6, 1280]
```N)r# rh r r r! rA )r r r r# rh r r r r r )r r decoder_hidden_statesdecoder_attentionsr" encoder_last_hidden_stater encoder_attentionsrL ) r9 r r lenr: r r r r r r" ) r: r# rh r rD rE r r rF r rk decoder_outputss r. rH BlenderbotModel.forward s ` "/3|| 0# - +0 0O O_== -"1!"4474H14Loa 0RV14_1E1I?1-t O FJ\\ F
'1"1!"4#1 +/ F
F
" - ? ? + ; ;"1"?"? . 9 9 , = =&5&G&G"1"?"? . 9 9
r0 )r: r9 r8 ) NNNNNNNNN)rM rN rO rP _tied_weights_keysr" r9 r> rB r r rD r rT r r r r r r r rH rU rV rW s @r. r6 r6 s/ (7'6
/
0
.2.259:>26(,26:>!%P
##d*P
t+P
!++d2 P
!& 0 04 7 P
)4/
P
P
((4/P
% 0 04 7P
$;P
+,P
P
P
r0 r6 z\
The Blenderbot Model with a language modeling head. Can be used for summarization.
) custom_introc ^ \ rS rSrSrS/rSS0rS\4U 4S jjr S S \ S
\ S- S \
S \ R 4U 4S
jjjr
S \ S S4S jr\\ S S\R$ S- S\R& S- S\R$ S- S\R$ S- S\S- S\S- S\R, S- S\R, S- S\R$ S- S\
S- S\\ S \4S jj5 5 rSrU =r $ ) r i r r lm_head.weightzmodel.shared.weightr c v > [ TU ] U5 [ U5 U l U R S[
R " SU R R R 45 5 [ R " UR U R R R SS9U l U R 5 g )Nr r! Fr )r8 r9 r6 r register_bufferrD zerosr8 r4 r r r lm_headr r s r. r9 +BlenderbotForConditionalGeneration.__init__ s
$V,
0%++q$**BSBSBbBb>c2deyy 1B1B1Q1QX]^
r0 Nnew_num_tokenspad_to_multiple_of
mean_resizingr c x > [ TU ] XU5 nU R UR R S 5 U$ )Nr )r8 resize_token_embeddings_resize_final_logits_biasrG r) )r: rY rZ r[ new_embeddingsr; s r. r] :BlenderbotForConditionalGeneration.resize_token_embeddings s< 8 ]jk &&~'<'<'B'B1'EF r0 c , U R R S nX:: a U R S S 2S U24 nON[ R " SX-
4U R R S9n[ R
" U R U/SS9nU R
SU5 g )Nr' r! r rn r )r r) rD rV rC catrU )r: rY old_num_tokensnew_bias
extra_biass r. r^ >> from transformers import AutoTokenizer, BlenderbotForConditionalGeneration
>>> mname = "facebook/blenderbot-400M-distill"
>>> model = BlenderbotForConditionalGeneration.from_pretrained(mname)
>>> tokenizer = AutoTokenizer.from_pretrained(mname)
>>> UTTERANCE = "My friends are cool but they eat too many carbs."
>>> print("Human: ", UTTERANCE)
Human: My friends are cool but they eat too many carbs.
>>> inputs = tokenizer([UTTERANCE], return_tensors="pt")
>>> reply_ids = model.generate(**inputs)
>>> print("Bot: ", tokenizer.batch_decode(reply_ids, skip_special_tokens=True)[0])
Bot: That's unfortunate. Are they trying to lose weight or are they just trying to be healthier?
>>> REPLY = "I'm not sure"
>>> print("Human: ", REPLY)
Human: I'm not sure
>>> NEXT_UTTERANCE = (
... "My friends are cool but they eat too many carbs.
That's unfortunate. "
... "Are they trying to lose weight or are they just trying to be healthier? "
... " I'm not sure."
... )
>>> inputs = tokenizer([NEXT_UTTERANCE], return_tensors="pt")
>>> next_reply_ids = model.generate(**inputs)
>>> print("Bot: ", tokenizer.batch_decode(next_reply_ids, skip_special_tokens=True)[0])
Bot: I see. Well, it's good that they're trying to change their eating habits.
```
NzJThe `use_cache` argument is changed to `False` since `labels` is provided.F)rh r rE rD r r rF r r r' ) losslogitsr rI rJ r" rK r rL )r warningr/ r r$ r% r rW r torC r r r r r rI rJ r" rK r rL )r: r# rh r rD rE r r rF rg r rk outputs lm_logitsmasked_lm_lossloss_fcts r. rH *BlenderbotForConditionalGeneration.forward sd H kl I (-B-J$6KK 4 4dkk6X6X%! '+jj '
) / +#9 +'"7 '
'
LL, 6 6 9 9):J:J KK YYy / /0F')H %innR9O9O&PRXR]R]^`Ra bN # 3 3")"?"? & 9 9 $ 5 5&-&G&G")"?"? & 9 9
r0 rW r )NT)
NNNNNNNNNN) rM rN rO rP r _keys_to_ignore_on_load_missingrP r" r9 rR r r Embeddingr] r^ r r rD r rT r r r r r r rH rU rV rW s @r. r r s ':&;#/ / ae ! 7:Tz Y] < < < .2.259:>26(,26:>*.!%k
##d*k
t+k
!++d2 k
!& 0 04 7 k
)4/
k
k
((4/k
% 0 04 7k
4'k
$;k
+,k
k
k
r0 r c 2 ^ \ rS rSrSrU 4S jrS rSrU =r$ )BlenderbotDecoderWrapperi z
This wrapper class is a helper class to correctly load pretrained checkpoints when the causal language model is
used in combination with the [`EncoderDecoderModel`] framework.
c d > [ TU ] U5 [ U5 U l U R 5 g r7 )r8 r9 r r: r r s r. r9 !BlenderbotDecoderWrapper.__init__ s&
(0 r0 c & U R " U0 UD6$ r7 r: )r: argsrk s r. rH BlenderbotDecoderWrapper.forward s ||T,V,,r0 rz ) rM rN rO rP rQ r9 rH rU rV rW s @r. rv rv s
- -r0 rv c j ^ \ rS rSrSS0rU 4S jrS rS r\\ SS\
R S- S \
R S- S
\
R S- S \
R S- S \S- S
\
R S- S\
R S- S\S- S\\
R - S\\ S\\- 4S jj5 5 rSrU =r$ )BlenderbotForCausalLMi rS z!model.decoder.embed_tokens.weightc > SUl SUl [ TU ]
U5 [ U5 U l [ R " UR UR SS9U l
U R 5 g )NTFr ) r r* r8 r9 rv r r r hidden_sizer rW r r s r. r9 BlenderbotForCausalLM.__init__ sX $)!
-f5
yy!3!3V5F5FUS
r0 c B U R R R $ r7 r r: r r= s r. r> *BlenderbotForCausalLM.get_input_embeddings s zz!!...r0 c 8 XR R l g r7 r rA s r. rB *BlenderbotForCausalLM.set_input_embeddings s */
'r0 Nr# rh r r r r rg r logits_to_keeprk r c
U R R " SUUUUUUUS.U
D6n U S n [ U [ 5 ( a
[ U * S5 OU n
U R U SS2U
SS24 5 nSnUba UR
UR 5 n[ 5 nU" UR SU R R 5 UR S5 5 n[ UUU R U R U R U R S9$ )a
labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*):
Labels for computing the masked language modeling loss. Indices should either be in `[0, ...,
config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored
(masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`.
Example:
```python
>>> from transformers import AutoTokenizer, BlenderbotForCausalLM
>>> tokenizer = AutoTokenizer.from_pretrained("facebook/blenderbot-400M-distill")
>>> model = BlenderbotForCausalLM.from_pretrained("facebook/blenderbot-400M-distill")
>>> assert model.config.is_decoder, f"{model.__class__} has to be configured as a decoder."
>>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
>>> outputs = model(**inputs)
>>> logits = outputs.logits
>>> expected_shape = [1, inputs.input_ids.shape[-1], model.config.vocab_size]
>>> list(logits.shape) == expected_shape
True
```rH r Nr' )ri rj r r r r" rL )r r: r rR slicerW rl rC r r r r r r r r r" )r: r# rh r r r r rg r r rk rm r
slice_indicesrj ri rp s r. rH BlenderbotForCausalLM.forward s L >BZZ=O=O >
)"7#9 +' >
>
8B>SV8W8W~ot4]k
mA}a,? @A YYv}}-F')HFKKDKK,B,B CV[[QS_UD0 # 3 3!// )) $ 5 5
r0 rr ) NNNNNNNNr )rM rN rO rP rP r9 r> rB r r rD r rT r r r rR r r r r rH rU rV rW s @r. r~ r~ s/ = /0 .2.2:>;?(,26*.!%-.A
##d*A
t+A
% 0 04 7 A
!& 1 1D 8 A
A
((4/A
4'A
$;A
ell*A
+,A
2 2A
A
r0 r~ )r~ r r6 r )Nr )IrQ r collections.abcr rD r torch.nnr r r activationsr cache_utilsr r
r
generationr
masking_utilsr
r modeling_flash_attention_utilsr modeling_layersr modeling_outputsr r r r r modeling_utilsr r processing_utilsr utilsr r r r r
utils.genericr utils.output_capturingr r configuration_blenderbotr"
get_loggerrM r rT rR r/ rt r2 rY Modulerc r{ r} r r r r r r6 r rv r~ __all__rL r0 r. r s $ % & ! C C ) J B 9 G & l l 7 E 6
H %
%,,
c
[^
-2<< -*
=BLL
=( !% II% <<%
% << %
LL4' % T\
% % '(%:r)")) r)l5 7 5 rZ 7 Z z 4X
1 X
v@
1 @
F l
/ l
l
^
M
)BO M
M
b - 8 - Y
5 Y
x r0