ZjSSKJr SSKJr SSKrSSKJr SSKJr SSK J r SSK J r SS KJrJr SS KJr SS KJrJr SS KJr SS KJrJr SSKJrJr SSKJrJ r SSK!J"r"J#r# SSK$J%r% SSK&J'r'J(r(J)r)J*r* SSK+J,r,J-r- SSK.J/r/J0r0 SSK1J2r2 \*Rf"\45r5\"S5"SS\Rl55r7"SS\Rl5r8"SS\Rl5r9"SS\Rl5r:"S S!\Rl5r;"S"S#\Rl5rS'\R~S(\@S)\R~4S*jrASES+\RlS,\R~S-\R~S.\R~S/\R~S-S0\BS1\BS2\%\'4S3jjrC"S4S5\Rl5rD"S6S7\5rE\("S8S9\#55rF\("S:S;\F55rGSFS<\R~\H\R~-S-S=\@S-S/\R~S-S)\R~\@-4S>jjrI"S?S@\F\5rJ"SASB\\F5rK/SCQrLg)G)Callable)OptionalN)nn) functional)initialization)ACT2FN)Cache DynamicCache)GenerationMixin)use_kernel_forward_from_hubuse_kernel_func_from_hub)create_causal_mask) GenericForSequenceClassificationGradientCheckpointingLayer)MoeCausalLMOutputWithPastMoeModelOutputWithPast)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tuplelogging)maybe_autocastmerge_with_config_defaults)OutputRecordercapture_outputs) JetMoeConfigRMSNormcx^\rSrSrS S\SS4U4SjjjrS\RS\R4SjrSr S r U=r $) JetMoeRMSNorm0epsreturnNc>[TU]5 [R"[R "U55UlX lg)z, JetMoeRMSNorm is equivalent to T5LayerNorm N)super__init__r Parametertorchonesweightvariance_epsilon)self hidden_sizer' __class__s {/root/GenerationalWealth/GenerationalWealth/venv/lib/python3.13/site-packages/transformers/models/jetmoe/modeling_jetmoe.pyr+JetMoeRMSNorm.__init__2s/ ll5::k#:; # hidden_statescURnUR[R5nUR S5R SSS9nU[R "X0R-5-nURURU5-$)NT)keepdim) dtypetor-float32powmeanrsqrtr0r/)r1r7 input_dtypevariances r4forwardJetMoeRMSNorm.forward:sw#)) %((7  $$Q',,R,>% H?T?T4T(UU {{]--k:::r6c^[URR5SUR3$)Nz, eps=)tupler/shaper0)r1s r4 extra_reprJetMoeRMSNorm.extra_reprAs* ))*+6$2G2G1HIIr6)r0r/)gư>) __name__ __module__ __qualname____firstlineno__floatr+r-TensorrDrI__static_attributes__ __classcell__r3s@r4r%r%0sB$$$$$;U\\;ell;JJr6r%c ^\rSrSr%\R \S'SS\4U4Sjjjr\ SS\S-S\ SS\ S-S \ S \ 44S jj5r\R"5\S 55rS rU=r$)JetMoeRotaryEmbeddingEinv_freqNconfigc>[TU]5 URUlURUlXlUR R SUlURnURS:wa[URnU"UR U5uo@l URSUSS9 URSUR5SS9 g)N rope_typedefaultrWF) persistentoriginal_inv_freq) r*r+max_position_embeddingsmax_seq_len_cachedoriginal_max_seq_lenrXrope_parametersrZcompute_default_rope_parametersrattention_scalingregister_bufferclone)r1rXdevice rope_init_fnrWr3s r4r+JetMoeRotaryEmbedding.__init__Hs "("@"@$*$B$B! 44[A!%!E!E >>Y &.t~~>L+7 V+L(( ZeD 0(..2BuUr6rfz torch.deviceseq_lenr(z torch.Tensorc URSn[USS5=(d URUR-nSnSU[R "SUS[R S9RU[RS9U- -- nXe4$) aH Computes the inverse frequencies according to the original RoPE implementation Args: config ([`~transformers.PreTrainedConfig`]): The model configuration. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin (unused in this type of RoPE). rope_thetahead_dimNg?rr9r<)rfr<) ragetattrr2num_attention_headsr-arangeint64r=rO)rXrfribasedimattention_factorrWs r4rb5JetMoeRotaryEmbedding.compute_default_rope_parametersXs&%%l3fj$/c63E3EIcIc3c U\\!S!5;;?BB&X]XcXcBdgjj k ))r6cLURSSS2S4R5RURSSS5R UR 5nUSS2SSS24R5n[ UR R[5(a0UR RS:waUR ROSn[USS9 UR5UR5-RSS5n[R"Xf4SS 9nUR5UR-nUR5UR-n SSS5 WR UR S 9W R UR S 94$!,(df  N@=f) Nrr:r!mpscpuF) device_typeenabledr9rsrm)rWrOexpandrHr=rf isinstancetypestrr transposer-catcosrcsinr<) r1x position_idsinv_freq_expandedposition_ids_expandedryfreqsembrrs r4rDJetMoeRotaryEmbedding.forwardvsN!MM$4-8>>@GG HZHZ[\H]_acdehhijiqiqr ,QaZ 8 > > @'1!((--'E'E!((--[`J`ahhmmfk  U C&,,.1F1L1L1NNYYZ[]^_E))UN3C'')d444C'')d444C D vvAGGv$cff177f&;;; D Cs BF F#)rcrXr_r`rZNNNN)rKrLrMrNr-rP__annotations__r"r+ staticmethodrintrGrOrbno_gradrrDrQrRrSs@r4rUrUEsllV|VV &*+/"*t#*(*t* ~u$ % **: ]]_ < [TU]5 [R"[R "XU55UlXlX lX0l g)aS Initialize the JetMoeParallelExperts module. The experts weights are stored in [num_experts, output_size, input_size] format. Such that it's compatible with many MoE libraries, such as [Megablock](https://github.com/databricks/megablocks) and [ScatterMoE](https://github.com/shawntan/scattermoe), as well as the [MoE kernel](https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/layers/fused_moe/fused_moe.py) used in vllm. Args: num_experts (int): Number of experts. input_size (int): Size of the input. output_size (int): Size of the output. N) r*r+rr,r-emptyr/rrr)r1rrrr3s r4r+JetMoeParallelExperts.__init__s<" ll5;;{#TU &$&r6cURUSS9n/n[UR5H8nUR[R "X5UR U55 M: [R"USS9nU$)z Forward pass of the JetMoeParallelExperts module. Args: inputs (Tensor): Input tensor. expert_size: Expert size information. Returns: Tensor: Output tensor. rr{) splitrangerappendFlinearr/r-r)r1inputs expert_size input_list output_listiresultss r4rDJetMoeParallelExperts.forwardsh\\+1\5  t''(A   qxx t{{1~F G)))KQ/r6)rrrr/ rKrLrMrNrr+rDrQrRrSs@r4rrs.'C'S's't'.r6rc>^\rSrSrS\S\S\4U4SjjrSrSrU=r$)JetMoeTopKGatingrrtop_kcz>[TU]5 X lXlX0l[ R "XSS9Ulg)z Initialize the top-k gating mechanism. Args: input_size (`int`): Size of the input. num_experts (`int`): Number of experts. top_k (`int`): Number of top experts to select. FbiasN)r*r+rrrrLinearlayer)r1rrrr3s r4r+JetMoeTopKGating.__init__s2 &$ YYzUC r6czURU5R5nURURSS9up4[R "USS9R U5n[R"URS5UR/URURS9nURSUS5nUR5RS5nUR5nUR!5n U R#S5upU R%URSS9n UR!5nX[n XXU4$)Nr!r{rr<rftrunc) rounding_mode)rrOtopkrr-softmaxtype_aszerossizerr<rfscatterlongsumtolistflattensortdiv)r1r7logits top_k_logits top_k_indices top_k_gatesrgatesr top_k_experts_index_sorted_experts batch_index batch_gatess r4rDJetMoeTopKGating.forwards"M*002&,kk$**!k&D# mmLa8@@O    a $"2"2 3;;L;LU`UgUg  a2jjl&&q) "((* &--/ "/"4"4Q"7*..tzz.Q "))+ !7 #+FRRr6)rrrrrrSs@r4rrs-D3DSDD(SSr6rc:^\rSrSrSrS\4U4SjjrSrSrU=r $) JetMoeMoEz A Sparsely gated mixture of experts layer with 1-layer Feed-Forward networks as experts. Args: config: Configuration object with model hyperparameters. rXc<>[TU]5 URUlURUl[ UR Ul[RR[R"UR55Ul [URURURS-5Ul[URURUR5Ul[#URURUR$S9Ulg)Nr9rrr)r*r+r2rintermediate_sizer activation_function activationr-rr,rrrnum_local_experts input_linear output_linearrnum_experts_per_tokrouterr1rXr3s r4r+JetMoeMoE.__init__s  ,,!33 !;!;<HH&&u{{4??'CD 1&2J2JDOO]a]m]mpq]qr263K3KTM]M]_c_n_no&00,,  r6cUR5up#nURSU5nURU5upVpxn Xn URX5n U R SSS9n UR U S5U S-n UR X5n XSS2S4-n [R"X#-UR4U RU RS9nURSXm5nURX#UR5nXR-nU$)z Forward pass of the mixture of experts layer. Args: layer_input (Tensor): Input tensor. Returns: Tensor: Output tensor. Tensor: Router logits. r:r9r{rr!Nr)rreshaperrchunkrrr-rrr<rf index_addviewr)r1 layer_inputbszlengthemb_sizerrrr router_logits expert_inputsr7chunked_hidden_statesexpert_outputsr layer_outputs r4rDJetMoeMoE.forwards!, 0 0 2X!))"h7 BF++kBZ? -#0 ))-E - 3 3A2 3 >(=a(@ADYZ[D\\ ++MG'ag*>> S\4??;>CWCW`n`u`uvq+F #((dooF #ii/ r6)rrr2rrrr) rKrLrMrN__doc__r"r+rDrQrRrSs@r4rrs | r6rcF^\rSrSrSrS\4U4SjjrSrSrSr Sr U=r $) JetMoeMoAiz A Sparsely gated mixture of attention layer with pairs of query- and output-projections as experts. Args: config: Configuration object with model hyperparameters. rXcd>[TU]5 URUlURUlUR UR-UlURUl [RR[R"UR 55Ul[URUR UR5Ul[URURUR 5Ul[%UR URURS9Ulg)Nr)r*r+rrr2r kv_channelsnum_key_value_headsrrr-rr,rrrrrrrrs r4r+JetMoeMoA.__init__'s !33 ,,!--0J0JJ// HH&&u{{4??'CD 1$2B2BDOOUYUeUef243C3CTEUEUW[WfWfg&((**  r6cUR5up#nURSU5nURU5upVpxn XVXx4n Xn URX5n [R "X#-UR -UR4U RU RS9n U RSX\5nURX#UR S5nXU 4$)zq Map inputs to attention experts according to routing decision and compute query projection inside each experts. r:rr) rrrrr-rrr2r<rfrr)r1rrrrrrrrr topo_inforrrrs r4map JetMoeMoA.map9s !, 0 0 2X!))"h7 UYU`U`alUmR;]) Q $0 **=F \DJJ &(8(8 9AUAU^l^s^s q*>O #((djj"E I55r6cUR5up4pVURSU5nUupxpXn URX5n XSS2S4-n [R"X4-UR 4U R U RS9n U RSX5nURX4UR 5nXR-nU$)ze Compute output projection inside each attention experts and merge the outputs of different experts. r:Nrr) rrrr-rrr<rfrrr)r1rrrrkr2rrrrrrrrs r4reduceJetMoeMoA.reducePs'2&6&6&8#Q!))"k: FOC;$9 ++MG(ag*>> S\4??;>CWCW`n`u`uvq+F #((dooF #ii/ r6c[S5e)Nz-This module doesn't support call and forward.)NotImplementedError)r1rs r4rDJetMoeMoA.forwardfs!"QRRr6)rr2rrrrrr) rKrLrMrNrr"r+rrrDrQrRrSs@r4rrs* | $6.,SSr6rcUSSURSS-24nUSURSS-S24n[R"U*U4SS9$)z*Rotates half the hidden dims of the input..Nr:r9r{)rHr-r)rx1x2s r4 rotate_halfrjsZ 3"!''"+"" " #B 3 q " " #B 99rc2YB ''r6rotary_pos_embcURU5nURU5nX-[U5U--nX-[U5U--nXV4$)aIApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. ) unsqueezer)qrrr unsqueeze_dimq_embedk_embeds r4apply_rotary_pos_embrqsS& -- &C -- &Cw;q>C/0Gw;q>C/0G  r6r7n_repr(cURup#pEUS:XaU$USS2SS2SSS2SS24RX#XU5nURX#U-XE5$)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) r!N)rHr|r)r7r batchrslenrls r4 repeat_kvr s_ 2?1D1D.E z!!Qa"23::5W\dlmM  e(CT TTr6modulequerykeyvalueattention_maskscalingdropoutkwargsc [X R5n[X0R5n [R"XR SS55U-n UbX-n [ R RU S[RS9RUR5n [ R RXURS9n [R"X5n U R SS5R5n X4$)Nr9rr:)rsr<)ptrainingr!)r num_key_value_groupsr-matmulrrrrr>r=r<rr contiguous) rrrrrrrr key_states value_states attn_weights attn_outputs r4eager_attention_forwardr s3 ; ;[TU]5 XlX lSUlUc-[ R SURRS35 SUl URUl URUl URUR-UlURUlUR UlURUlUR$S-Ul[)U5Ul[,R.R1UR2URS-SS 9Ulg) z Initialize the JetMoeAttention module. Args: config: Configuration object with model hyperparameters. layer_idx: Index of the layer in the model. TNzInstantiating z without passing a `layer_idx` is not recommended and will lead to errors during the forward call if caching is used. Please make sure to provide a `layer_idx` when creating this class.r!gr9Fr)r*r+rXr# is_causallogger warning_oncer3rKrrrattention_dropoutrrkv_projection_sizero num_headsrlrrexpertsr-rrr2kv_projr1rXr#r3s r4r+JetMoeAttention.__init__s  "      !8!8 9:,,  %&!// !'!9!9"("4"4v7Q7Q"Q#)#=#= 33** }}d*  ( xxv'9'94;R;RUV;V]bc r6r7rposition_embeddingspast_key_valuesr(c URSSn/UQSPURP7nURRU5upn UR U5R SSS9upUR U5RSS5nU R U5RSS5n U R U5RSS5n Uup[XX5upUbURXUR5up[R"URR[5nU R!SUR"SS5n U R!SUR"SS5n U"UUU U U4UR$(dSO UR&UR(S.UD6unnUR "/UQUR"PSP76nURR+UU 5nUR "/UQSP76nUUU 4$)Nr:r9r{r!)rr)rHrlr+rr,rrrrupdater#r get_interfacerX_attn_implementationr repeatrrr(rr)r1r7rr/r0r input_shape hidden_shape query_statesrrrrrrattention_interfacerrs r4rDJetMoeAttention.forwards$))#2. 88b8$--8 151A1A-1P. Y#'<< #>#D#DQB#D#O #((6@@AF __\2<[TU]5 URUl[U5Ul[ UR5Ul[ UR5Ul[X5Ul gr) r*r+r2rmlpr%input_layernormpost_attention_layernormr"self_attentionr-s r4r+JetMoeDecoderLayer.__init__ s[ !--V$,V-?-?@(5f6H6H(I%-f@r6r7rrr0 use_cacher/rr(c UnURU5nUR"SUUUUUUS.UD6un n X-nUnURU5nURU5nX-nU$)N)r7rrr0rEr/)rArCrBr@) r1r7rrr0rEr/rresidualrs r4rDJetMoeDecoderLayer.forwards!,,]; "11 ')%+ 3    q!!0 !55mD /  0 r6)r2rAr@rBrCr)NNNFN)rKrLrMrNr"rr+r-rPr<r boolrGrrrDrQrRrSs@r4r>r>sA|Ad AA/304(,!&HL|| t+&&-    $; #5<<#=>E+, r6r>c^\rSrSr%\\S'SrSrS/rS/r Sr Sr Sr Sr Sr\"\SS 9\"\S S 9/\\"\S S 9S .r\R*"5U4S j5rSrU=r$)JetMoePreTrainedModeli1rXmodelFr>r0Tr9)indexr!)rr7 attentionsc2>[TU]U5 [U[5(a5[R "UR SURRS9 g[U[[-5(a![R"UR5 gg)zInitialize the weights.r2)r@stdN) r* _init_weightsr}rinitnormal_r/rXinitializer_rangerrzeros_r)r1rr3s r4rS#JetMoePreTrainedModel._init_weightsCse f% f3 4 4 LLSdkk6S6S T  I 5 6 6 KK $7r6rG)rKrLrMrNr"rbase_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_supports_sdpa_supports_flex_attn_can_compile_fullgraph_supports_attention_backendrr"rr>_can_record_outputsr-rrSrQrRrSs@r4rLrL1s &+#-.#4"5N""&(BNScklDmn+$_A>  ]]_%%r6rLc^\rSrSrS\4U4Sjjr\\\SS\ RS-S\ RS-S\ RS-S\ S-S \ RS-S \S-S \\S \4S jj555rSrU=r$) JetMoeModeliMrXc *>[TU]U5 URUlURUl[ R "URURUR5Ul[ R"[UR5Vs/sHn[X5PM sn5Ul [URURS9Ul[#US9UlSUlUR(UlUR+5 gs snf)N)r'rXF)r*r+ pad_token_id padding_idx vocab_sizer Embeddingr2 embed_tokens ModuleListrnum_hidden_layersr>layersr% rms_norm_epsnormrU rotary_embgradient_checkpointingr5 post_initr-s r4r+JetMoeModel.__init__Os  !.. ++LL):):F&+#$*$?$?!  esDN input_idsrrr0 inputs_embedsrErr(c BUSLUSL- (a [S5eU(aUc[URS9nUcURU5nUcUUbUR 5OSn[ R "URSURS9U-nURS5n[URUUUUS9n Un URX5n URSURRHn U "U 4U U UUUS.UD6n M URU 5n [U US9$) Nz:You must specify exactly one of input_ids or inputs_embedsrfrr!)rf)rXrvrr0r)r/rr0rEr)last_hidden_stater0) ValueErrorr rXrkget_seq_lengthr-rprHrfrrrqrnrmrpr) r1rurrr0rvrErpast_seen_tokens causal_maskr7r/ decoder_layers r4rDJetMoeModel.forward`sF  -t"; <YZ Z 0*$++>O   --i8M  CRC^==?de  << (;(;A(>}G[G[\_ooL'11!4L(;;')+%  & #oomJ![[)H4;;+H+HIM)$7* /#) MJ -0 %++  r6)r5rkrrrnrprhrqri)NNNNNN)rKrLrMrNr"r+rr rr-r<rPr FloatTensorrJrrrrDrQrRrSs@r4rdrdMs|" .2.204(,26!%5 ##d*5  t+5 &&- 5  5 ((4/ 5 $;5 +,5  5  5 r6rd gate_logitsrcUb[U[5(dg[U[5(aCUSRn[R"UVs/sHoUR U5PM snSS9n[R RRWSS9n[R"XrSS9up[R RRX5n Uc:[R"U R5SS9n [R"USS9n GOURupURSX--nUSSS2SS2SS4RXXU45RSX!5R W5n[R "U R5U-SS9[R "USS9- n USSS2SS2S4RXX45RSU5R U5n[R "UU-SS9[R "USS9- n [R "XR#S5-5nUU-$s snf)ax Computes auxiliary load balancing loss as in Switch Transformer - implemented in Pytorch. See Switch Transformer (https://huggingface.co/papers/2101.03961) for more details. This function implements the loss function presented in equations (4) - (6) of the paper. It aims at penalizing cases where the routing between experts is too unbalanced. Args: gate_logits: Logits from the `gate`, should be a tuple of model.config.num_hidden_layers tensors of shape [batch_size X sequence_length, num_experts]. num_experts: Number of experts top_k: The number of experts to route per-token, can be also interpreted as the `top-k` routing parameter. attention_mask (`torch.Tensor`, *optional*): The attention_mask used in forward function shape [batch_size X sequence_length] if not None. Returns: The auxiliary loss. Nrr{r:)r}rGrfr-rr=rrrrone_hotr@rOrHr|rrr)rrrrcompute_device layer_gateconcatenated_gate_logitsrouting_weightsrselected_experts expert_masktokens_per_expertrouter_prob_per_expert batch_sizesequence_lengthrmexpert_attention_mask router_per_expert_attention_mask overall_losss r4load_balancing_loss_funcrs+:*[%"@"@+u%%$Q..#(99^i-j^iPZmmN.K^i-jpq#r hh))112JPR1SO**_DA((%%--.>LK!JJ{'8'8':B"'O!C&4&:&:# 4::1=*B^_ 4AtT1 2 V&OKX Y WR , R  "IIk&7&7&9[TU]U5 [U5UlURUlUR Ul[ R"URURSS9Ul URUl URUl URUl UR5 g)NFr)r*r+rdrMri aux_loss_coefrrr2lm_headtie_word_embeddingsrrrrsrs r4r+JetMoeForCausalLM.__init__s   (  ++#11yy!3!3V5F5FUS #)#=#= !33#)#=#=  r6Nrurrr0rvlabelsrElogits_to_keepoutput_router_logitsr(c \UR"SUUUUUUU S.U D6n U Rn [U[5(a [ U*S5OUn UR U SS2U SS245nSnUb*UR "UU4SURR0U D6nSnU (aY[U RURURU5nUb*XRURUR5-- n[!UUUU R"U R$U R&U RS9$)N)rurrr0rvrErri)lossaux_lossrr0r7rPrrG)rMrxr}rslicer loss_functionrXrirrrrrr=rfrr0r7rP)r1rurrr0rvrrErrroutputsr7 slice_indicesrrrs r4rDJetMoeForCausalLM.forwardsN+/** + )%+'!5 +  +  11 8B>SV8W8W~ot4]k mA}a,?@A  %% ;;11 D /%%  (( H !**X[[-EEE(#33!//))!//  r6)rrrMrrrri) NNNNNNNrF)rKrLrMrN_tied_weights_keysr+rrr-r<rPr rrJrrrDrQrRrSs@r4rrs*,GH .2.204(,26*.!%-.,19 ##d*9  t+9 &&- 9  9 ((4/ 9   4'9 $;9 ell*9 #Tk9  #9 9 r6rc\rSrSrSrg)JetMoeForSequenceClassificationi;rGN)rKrLrMrNrQrGr6r4rr;s`cr6r)rrdrLr)r!)r2)Nr9N)Mcollections.abcrtypingrr-rtorch.nnrrrrT activationsr cache_utilsr r generationr integrationsr r masking_utilsrmodeling_layersrrmodeling_outputsrrmodeling_rope_utilsrrmodeling_utilsrrprocessing_utilsrutilsrrrr utils.genericrrutils.output_capturingrr configuration_jetmoer" get_loggerrKr&Moduler%rUrrrrrrrPrr rOr r"r>rLrdrGrrr__all__rGr6r4rs*% $&!.)Q/[QKF&RRGE.   H %Y'JBIIJ(J(>