Zj+SrSSKJr SSKJr SSKJr SSKJr \"SS9\"S S \555r \"SS9\"S S \555r S /r g )zMpt configuration)Literal)strict)PreTrainedConfig)auto_docstringzmosaicml/mpt-7b) checkpointc\rSrSr%SrSrSr\S\S'Sr \ \S'S r \ \S 'S r \S -\S 'S r\S -\S 'Sr\\S'Sr\\S'Sr\\S'Sr\\S'Sr\ \S'Srg )MptAttentionConfiga attn_type (`str`, *optional*, defaults to `"multihead_attention"`): type of attention to use. Options: `"multihead_attention"`, `"multiquery_attention"`. attn_pdrop (`float`, *optional*, defaults to `0.0`): The dropout probability for the attention layers. attn_impl (`str`, *optional*, defaults to `"torch"`): The attention implementation to use. One of `"torch"`, `"flash"`, or `"triton"`. clip_qkv (`float`, *optional*): If not `None`, clip the queries, keys, and values in the attention layer to this value. softmax_scale (`float`, *optional*): If not `None`, scale the softmax in the attention layer by this value. If `None`, will default to `1/sqrt(hidden_size)`. prefix_lm (`bool`, *optional*, defaults to `False`): Whether the model should operate as a Prefix LM. This requires passing an extra `prefix_mask` argument which indicates which tokens belong to the prefix. Tokens in the prefix can attend to one another bi-directionally. Tokens outside the prefix use causal attention. qk_ln (`bool`, *optional*, defaults to `False`): Whether to apply layer normalization to the queries and keys in the attention layer. attn_uses_sequence_id (`bool`, *optional*, defaults to `False`): Whether to restrict attention to tokens that have the same token_type_ids. When the model is in `train` mode, this requires passing an extra *token_type_ids* argument which indicates which sub-sequence each token belongs to. Defaults to `False` meaning any provided *token_type_ids* will be ignored. alibi (`bool`, *optional*, defaults to `True`): Whether or not to use the alibi bias instead of positional embedding. alibi_bias_max (`int`, *optional*, defaults to 8): The maximum value of the alibi bias. attn_configmultihead_attention)r multiquery_attention attn_typer attn_pdroptorch attn_implNclip_qkv softmax_scaleF prefix_lmqk_lnattn_uses_sequence_idTalibialibi_bias_max)__name__ __module__ __qualname____firstlineno____doc__base_config_keyrr__annotations__rintrstrrfloatrrboolrrrr__static_attributes__rz/root/GenerationalWealth/GenerationalWealth/venv/lib/python3.13/site-packages/transformers/models/mpt/configuration_mpt.pyr r s8$OH]IwDE]JIs!Hedl!"&M54<&ItE4"'4'E4NCr(r c^\rSrSr%SrSrS\0rSSSS.rS r \ \ S'S r \ \ S'S r \ \ S'S r\ \ S 'S r\ \ S'Sr\ \ S'Sr\\ -\ S'Sr\\ S'Sr\\ -\ S'Sr\\ S'Sr\\-S-\ S'Sr\\ S'Sr\\-S-\ S'Sr\\ S'Sr\\ S'Sr\\ S 'S!r\\ S"'S#r \\ S$'Sr!\\ S%'Sr"\ S-\ S&'Sr#\ S-\ S''Sr$\ \%\ -S-\ S('U4S)jr&S*r'U=r($)+ MptConfigEa expansion_ratio (`int`, *optional*, defaults to 4): The ratio of the up/down scale in the MLP. max_seq_len (`int`, *optional*, defaults to 2048): The maximum sequence length of the model. layer_norm_epsilon (`float`, *optional*, defaults to 1e-05): The epsilon to use in the layer normalization layers. learned_pos_emb (`bool`, *optional*, defaults to `True`): Whether to use learned positional embeddings. attn_config (`dict`, *optional*): A dictionary used to configure the model's attention module. init_device (`str`, *optional*, defaults to `"cpu"`): The device to use for parameter initialization. Defined for backward compatibility logit_scale (`float`, *optional*): If not None, scale the logits by this value. no_bias (`bool`, *optional*, defaults to `True`): Whether to use bias in all linear layers. embedding_fraction (`float`, *optional*, defaults to 1.0): The fraction to scale the gradients of the embedding layer by. norm_type (`str`, *optional*, defaults to `"low_precision_layernorm"`): Type of layer norm to use. All MPT models uses the same layer norm implementation. Defined for backward compatibility. Example: ```python >>> from transformers import MptConfig, MptModel >>> # Initializing a Mpt configuration >>> configuration = MptConfig() >>> # Initializing a model (with random weights) from the configuration >>> model = MptModel(configuration) >>> # Accessing the model configuration >>> configuration = model.config ``` mptr n_headsd_modeln_layers)num_attention_heads hidden_sizenum_hidden_layersir expansion_ratio max_seq_leni vocab_sizeg resid_pdropgh㈵>layer_norm_epsilon emb_pdropTlearned_pos_embNcpu init_device logit_scaleno_biasg?embedding_fractionlow_precision_layernorm norm_typeF use_cacheg{Gz?initializer_rangetie_word_embeddings pad_token_id bos_token_id eos_token_idc >URc[5UlO9[UR[5(a[S0URD6Ul[TU]"S0UD6 g)Nr)r r isinstancedictsuper __post_init__)selfkwargs __class__s r)rNMptConfig.__post_init__sS    #13D  (($ / /1ED4D4DED  ''r()r ))rrrrr model_typer sub_configs attribute_mapr/r#r"r.r0r6r7r8r9r%r:r;r<r&r rLr>r$r?r@rArCrDrErFrGrHrIlistrNr' __classcell__)rQs@r)r+r+EsH%NJ "45K( 'M GSGSHcOSKJ"K" $$ Ius{  OT 48K**T18K&*Kt#*GT ##.Is.It#u# $$#L#*##L#*#+/L#S /D(/((r(r+N) r typingrhuggingface_hub.dataclassesrconfiguration_utilsrutilsrr r+__all__rr(r)r]sx.3#,-()(.(V,-L( L(.L(^ -r(