ZjSrSSKJr SSKJr SSKJr SSKJr \\"SSS 9"S S \555r \\"S SS 9"S S\555r \"SS9\"SS\555r S/r g)zDBRX model configuration)strict)PreTrainedConfig)RopeParameters)auto_docstringz4This config is used to instantiate attention layers.z$transformers-community/dbrx-instruct) custom_intro checkpointcZ\rSrSr%SrSrSr\\-\ S'Sr \\-S-\ S'Sr \\ S 'S r g) DbrxAttentionConfiga^ attn_pdrop (`float`, *optional*, defaults to 0.0): The dropout probability for the attention layers. clip_qkv (`float`, *optional*): If set, clip the queries, keys, and values in the attention layer to this value. kv_n_heads (`int`, *optional*, defaults to 1): For grouped_query_attention only, allow user to specify number of kv heads. attn_config attn_pdropNclip_qkv kv_n_heads) __name__ __module__ __qualname____firstlineno____doc__base_config_keyrfloatint__annotations__rr__static_attributes__r|/root/GenerationalWealth/GenerationalWealth/venv/lib/python3.13/site-packages/transformers/models/dbrx/configuration_dbrx.pyr r s9 $O!J !#'HcEkD 'Jrr z6This config is used to instantiate feedforward layers.c^\rSrSr%SrSrSr\\S'Sr \ S-\S'Sr \\S 'S r \\S 'S r \\S 'Sr\S-\S'Sr\\S'Sr\S-\S'U4SjrSrU=r$) DbrxFFNConfig-a ffn_act_fn (`dict`, *optional*, defaults to `None`): A dict specifying activation function for the FFN. The dict should have a key 'name' with the value being the name of the activation function along with any additional keyword arguments. If `None`, then set to `{"name": "silu"}`. ffn_hidden_size (`int`, *optional*, defaults to 3584): The hidden size of the feedforward network. moe_num_experts (`int`, *optional*, defaults to 4): The number of experts in the mixture of experts layer. moe_top_k (`int`, *optional*, defaults to 1): The number of experts to use in the mixture of experts layer. moe_jitter_eps (`float`, *optional*, defaults to `None`): If not `None`, the jitter epsilon for the mixture of experts layer. moe_loss_weight (`float`, *optional*, defaults to 0.01): The loss weight for the mixture of experts layer. moe_normalize_expert_weights (`float`, *optional*, defaults to 1.0): The normalization factor for the expert weights. ffn_configi hidden_sizeN ffn_act_fniffn_hidden_sizemoe_num_expertsr moe_top_kmoe_jitter_epsg{Gz?moe_loss_weightg?moe_normalize_expert_weightsc >URc SS0UlSHnX!;dM URU5 M [U5S:wa[SU<35e[TU]"S0UD6 g)Nnamesilu) model_typeattn_implementationexperts_implementationtransformers_version _commit_hash torch_dtypedtyperzFound unknown kwargs=r)r%poplen ValueErrorsuper __post_init__)selfkwargsk __class__s rr;DbrxFFNConfig.__post_init__Qsh ?? "%v.DO A{ 1   v;! 5fY78 8 ''r)r%)rrrrrrr$rrr%dictr&r(r)r*rr+r,r;r __classcell__r?s@rr!r!-sv &#OK"Jt "OSOSIs#'NEDL'!OU!14 %$,4((rr!)r c^\rSrSr%SrSr\\S.rSSSSS .r S r \ S -\ S'S r \ S -\ S'S r\ S -\ S'S r\ S -\ S'Sr\ \ S'Sr\S -\ S'Sr\S -\ S'S r\\-S -\ S'S r\\-S -\ S'Sr\\ S'Sr\\ S'Sr\S -\ S'S r\\-S -\ S'S r\ S -\ S'S r\ S -\ S'S r\ \ \ -S -\ S'Sr!\\ S'U4S jr"S!r#S"r$U=r%$)# DbrxConfigfa max_seq_len (`int`, *optional*, defaults to 2048): The maximum sequence length of the model. attn_config (`dict`, *optional*): A dictionary used to configure the model's attention module. ffn_config (`dict`, *optional*): A dictionary used to configure the model's FFN module. Example: ```python >>> from transformers import DbrxConfig, DbrxModel >>> # Initializing a Dbrx configuration >>> configuration = DbrxConfig(n_layers=2, d_model=256, n_heads=8, vocab_size=128) >>> # Initializing a model (with random weights) from the configuration >>> model = DbrxModel(configuration) >>> # Accessing the model configuration >>> configuration = model.config ``` dbrx)r r#n_headsd_modeln_layers max_seq_len)num_attention_headsr$num_hidden_layersmax_position_embeddingsiNi} vocab_sizer resid_pdrop emb_pdropr r#T use_cacheg{Gz?initializer_rangeFoutput_router_logitsrope_parameters pad_token_id bos_token_id eos_token_idtie_word_embeddingsc >URc[5UlO9[UR[5(a[S0URD6UlURc[ 5UlO9[UR[5(a[ S0URD6UlURR Ul[TU]$"S0UD6 g)Nr) r r isinstancerAr#r!rnum_key_value_headsr:r;)r<r=r?s rr;DbrxConfig.__post_init__s    #24D  (($ / /2FT5E5EFD  ?? "+oDO  . .+>doo>DO#'#3#3#>#>  ''rc<UR(a [S5eg)zOPart of `@strict`-powered validation. Validates the architecture of the config.z5tie_word_embeddings is not supported for DBRX models.N)r[r9)r<s rvalidate_architecture DbrxConfig.validate_architectures  # #TU U $r)r r#r^)&rrrrrr0r r! sub_configs attribute_maprIrrrHrJrKrQrRrrSr rAr#rTboolrUrVrWrrXrYrZlistr[r;rarrBrCs@rrErEfs<.J"5]SK( '#0 MGS4ZGS4ZHcDj"Kt"J #K#!Iut|!59K$t+d29.2J $t+2It#u#(-$+-48O^d*T18#L#*##L#*#+/L#S /D(/ %% (VVrrEN) rhuggingface_hub.dataclassesrconfiguration_utilsrmodeling_rope_utilsrutilsrr r!rE__all__rrrrls.31#G5*  "I51($1(  1(hABDV!DVCDVN .r