Z j; f S r SSKrSSKrSSKrSSKJr SSKrSSKJr SSK J
r
J r J r SSK
Jr SSKJr \R" " \5 r\ " S S
5 5 r " S S 5 r\ " S
S\5 5 r " S S5 r " S S\5 r " S S\5 r " S S\5 r " S S\5 r " S S\\5 r g)zJ
Callbacks to use with the Trainer class and customize the training loop.
N) dataclass)tqdm )IntervalStrategy SaveStrategy
has_length)TrainingArguments)loggingc \ rS rSr% SrSr\\S' Sr\ \S' Sr
\ \S' Sr \ \S' Sr \ \S ' Sr
\ \S
' S r\ S - \S ' Sr\ \S
' Sr\ \S' Sr\\S' S r\\\\4 \S' S r\S - \S' S r\ S - \S' S r\S - \S' Sr\\S' Sr\\S' Sr \\S' S r \S - \S' S r \\\\- \ - \- 4 S - \S' S r \S S - \S ' S r S \4S jr!\"S \4S j5 r#S! r$S" r%S#r&g )$ TrainerState" a
A class containing the [`Trainer`] inner state that will be saved along the model and optimizer when checkpointing
and passed to the [`TrainerCallback`].
In all this class, one step is to be understood as one update step. When using gradient accumulation, one update
step may require several forward and backward passes: if you use `gradient_accumulation_steps=n`, then one update
step requires going through *n* batches.
Args:
epoch (`float`, *optional*):
Only set during training, will represent the epoch the training is at (the decimal part being the
percentage of the current epoch completed).
global_step (`int`, *optional*, defaults to 0):
During training, represents the number of update steps completed.
max_steps (`int`, *optional*, defaults to 0):
The number of update steps to do during the current training.
logging_steps (`int`, *optional*, defaults to 500):
Log every X updates steps
eval_steps (`int`, *optional*):
Run an evaluation every X steps.
save_steps (`int`, *optional*, defaults to 500):
Save checkpoint every X updates steps.
train_batch_size (`int`, *optional*):
The batch size for the training dataloader. Only needed when
`auto_find_batch_size` has been used.
num_input_tokens_seen (`int`, *optional*, defaults to 0):
When tracking the inputs tokens, the number of tokens seen during training (number of input tokens, not the
number of prediction tokens).
total_flos (`float`, *optional*, defaults to 0):
The total number of floating operations done by the model since the beginning of training (stored as floats
to avoid overflow).
log_history (`list[dict[str, float]]`, *optional*):
The list of logs done since the beginning of training.
best_metric (`float`, *optional*):
When tracking the best model, the value of the best metric encountered so far.
best_global_step (`int`, *optional*):
When tracking the best model, the step at which the best metric was encountered.
Used for setting `best_model_checkpoint`.
best_model_checkpoint (`str`, *optional*):
When tracking the best model, the value of the name of the checkpoint for the best model encountered so
far.
is_local_process_zero (`bool`, *optional*, defaults to `True`):
Whether or not this process is the local (e.g., on one machine if training in a distributed fashion on
several machines) main process.
is_world_process_zero (`bool`, *optional*, defaults to `True`):
Whether or not this process is the global main process (when training in a distributed fashion on several
machines, this is only going to be `True` for one process).
is_hyper_param_search (`bool`, *optional*, defaults to `False`):
Whether we are in the process of a hyper parameter search using Trainer.hyperparameter_search. This will
impact the way data will be logged in TensorBoard.
stateful_callbacks (`list[StatefulTrainerCallback]`, *optional*):
Callbacks attached to the `Trainer` that should have their states be saved or restored.
Relevant callbacks should implement a `state` and `from_state` function.
r epoch global_step max_stepsi
logging_steps
eval_steps
save_stepsNtrain_batch_sizenum_train_epochsnum_input_tokens_seen
total_flos log_history best_metricbest_global_stepbest_model_checkpointTis_local_process_zerois_world_process_zeroFis_hyper_param_search
trial_name trial_paramsTrainerCallbackstateful_callbacksc U R c / U l U R c 0 U l g [ U R [ 5 ( a g 0 nU R H n[ U[ 5 ( d [ S[
U5 35 eUR R nX1; aA [ X [ 5 ( d X /X' X R UR 5 5 M UR 5 X' M Xl g )NzNAll callbacks passed to be saved must inherit `ExportableState`, but received ) r r"
isinstancedictExportableState TypeErrortype __class____name__listappendstate)selfr" callbacknames n/root/GenerationalWealth/GenerationalWealth/venv/lib/python3.13/site-packages/transformers/trainer_callback.py
__post_init__TrainerState.__post_init__t s # !D " " *&(D #
//
6
6 "$ 3 3!(_>>#himnviwhxy ))22- &&8&>EE4F4L3M*0&,33HNN4DE/7~~/?&, 4 '9 # json_pathc [ R " [ R " U 5 SSS9S- n[ USSS9 nUR U5 SSS5 g! , ( d f g= f) zDSave the content of this instance in JSON format inside `json_path`. T)indent sort_keys
wutf-8encodingN)jsondumps dataclassesasdictopenwrite)r. r5 json_stringfs r1 save_to_jsonTrainerState.save_to_json sK jj!3!3D!9!tTW[[
)S7
3q
GGK 4
3
3s A
A"c [ USS9 nUR 5 nSSS5 U " S0 [ R " W5 D6$ ! , ( d f N*= f)z3Create an instance from the content of `json_path`.r< r= N )rC readr? loads)clsr5 rF texts r1 load_from_jsonTrainerState.load_from_json s@ )g
.!668D /&TZZ%&& /
.s A
Ac S HC n[ X S35 nUc M US: a [ R " X$- 5 n[ X S3U5 ME g)zt
Calculates and stores the absolute value for logging,
eval, and save steps based on if it was a proportion
or not.
)r
evalsave_stepsNr )getattrmathceilsetattr)r. argsr step_kind num_stepss r1
compute_stepsTrainerState.compute_steps sN 5I 6&:;I$ q= $ )*? @I 6 2I> 5r4 c UR b- UR b UR UR 5 U l SU l Ub SSKJn U" U5 U l X l X0l UR 5 U l UR 5 U l g)z9
Stores the initial training references needed in `self`
Nr ) hp_params)
hp_name_trialr r transformers.integrationsr_ r r r r )r. trainerr r trialr_ s r1 init_training_references%TrainerState.init_training_references sx ?? &7>>+E &oognn =DO ; )% 0D " 0 %,%B%B%D"%,%B%B%D"r4 )r r r r r r" r r )'r*
__module__ __qualname____firstlineno____doc__r float__annotations__r intr r r r r r r r r r+ r% strr r r r boolr r r r r" r2 rG classmethodrO r\ re __static_attributes__rJ r4 r1 r r " s\ 9v E5KIs M3 JJ#'cDj' c !"3"J*.Kd3:&'. $K $#'cDj'(,3:,"&4&"&4&"'4' !Jd
!?CL$sC%K# - 445<C9= ./$6=96!c ! 's ' ' ?Er4 r c 6 \ rS rSrSrS\4S jr\S 5 rSr g)r& a
A class for objects that include the ability to have its state
be saved during `Trainer._save_checkpoint` and loaded back in during
`Trainer._load_from_checkpoint`.
These must implement a `state` function that gets called during the respective
Trainer function call. It should only include parameters and attributes needed to
recreate the state at a particular time, to avoid utilizing pickle/maintain standard
file IO writing.
Example:
```python
class EarlyStoppingCallback(TrainerCallback, ExportableState):
def __init__(self, early_stopping_patience: int = 1, early_stopping_threshold: Optional[float] = 0.0):
self.early_stopping_patience = early_stopping_patience
self.early_stopping_threshold = early_stopping_threshold
# early_stopping_patience_counter denotes the number of times validation metrics failed to improve.
self.early_stopping_patience_counter = 0
def state(self) -> dict:
return {
"args": {
"early_stopping_patience": self.early_stopping_patience,
"early_stopping_threshold": self.early_stopping_threshold,
},
"attributes": {
"early_stopping_patience_counter": self.early_stopping_patience_counter,
}
}
```returnc [ S5 e)NzInternal class that just calls the list of callbacks in order.c / U l U H nU R U5 M X l X0l X@l XPl S U l S U l [ S U R 5 5 ( d# [ R SU R - 5 g g )Nc 3 B # U H n[ U[ 5 v M g 7fN)r$ DefaultFlowCallback.0cbs r1 +CallbackHandler.__init__.. s P 2:b"566 s zThe Trainer will not work properly if you don't have a `DefaultFlowCallback` in its callbacks. You
should add one before training with `trainer.add_callback(DefaultFlowCallback). The current list ofcallbacks is
:) callbacks add_callbackmodelprocessing_class optimizer lr_schedulertrain_dataloadereval_dataloaderanyloggerwarning
callback_list)r. r r r r r r s r1 __init__CallbackHandler.__init__ s B b !
0 " ( $ # P PPP NN$ $$%
Qr4 c j [ U[ 5 ( a U" 5 OUn[ U[ 5 ( a UO UR nX0R Vs/ s H oDR PM sn; a) [ R SU S3S- U R - 5 U R R U5 g s snf )NzYou are adding a zH to the callbacks of this Trainer, but there is already one. The currentzlist of callbacks is
:)r$ r( r) r r r r r, )r. r/ r cb_classcs r1 r CallbackHandler.add_callback s %h55XZ8 )(D 9 98x?Q?Q ^^<^ ^< < NN#H:-uv+,$$%
b!
=s B0c " [ U[ 5 ( aC U R H2 n[ X!5 ( d M U R R U5 Us $ g U R H' nX!:X d M
U R R U5 Us $ g r r$ r( r remover. r/ r s r1 pop_callback CallbackHandler.pop_callback sk h % %nn b++NN))"- I %
nn>NN))"- I %r4 c [ U[ 5 ( aA U R H0 n[ X!5 ( d M U R R U5 g g U R R U5 g r r r s r1 remove_callback CallbackHandler.remove_callback sQ h % %nn b++NN))"- %
NN ! !( +r4 c F SR S U R 5 5 $ )Nr: c 3 L # U H oR R v M g 7fr )r) r* r s r1 r 0CallbackHandler.callback_list.. s H 2 .. s "$)joinr rw s r1 r CallbackHandler.callback_list s yyH HHHr4 rY r- r c , U R " SXU40 UD6$ )Nr
call_eventr s r1 r CallbackHandler.on_init_end }d7MfMMr4 c : SUl U R " SXU40 UD6$ )NFr )r r r s r1 r CallbackHandler.on_train_begin s# ',$ /gPPPr4 c , U R " SXU40 UD6$ )Nr r r s r1 r CallbackHandler.on_train_end ~tGNvNNr4 c : SUl U R " SXU40 UD6$ )NFr )r r r s r1 r CallbackHandler.on_epoch_begin s# $)! /gPPPr4 c , U R " SXU40 UD6$ )Nr r r s r1 r CallbackHandler.on_epoch_end r r4 c V SUl SUl SUl U R " SXU40 UD6$ )NFr )r r r r r s r1 r CallbackHandler.on_step_begin s2 ""' #WOOOr4 c , U R " SXU40 UD6$ )Nr r r s r1 r %CallbackHandler.on_pre_optimizer_step s 6WWPVWWr4 c , U R " SXU40 UD6$ )Nr r r s r1 r !CallbackHandler.on_optimizer_step s 2DSFSSr4 c , U R " SXU40 UD6$ )Nr r r s r1 r CallbackHandler.on_substep_end s /gPPPr4 c , U R " SXU40 UD6$ )Nr r r s r1 r CallbackHandler.on_step_end r r4 c > SUl U R " SXU4SU0UD6$ )NFr r )r r r s r1 r CallbackHandler.on_evaluate
s( "' }d7^G^W]^^r4 c 0 U R " SXU4SU0UD6$ )Nr r r r s r1 r CallbackHandler.on_predict s |T']7]V\]]r4 c : SUl U R " SXU40 UD6$ )NFr )r r r s r1 r CallbackHandler.on_save s" #y$wI&IIr4 c > SUl U R " SXU4SU0UD6$ )NFr logs)r r )r. rY r- r r r s r1 r CallbackHandler.on_log s' "xgSDSFSSr4 c , U R " SXU40 UD6$ )Nr r r s r1 r "CallbackHandler.on_prediction_step s 3T'TVTTr4 c , U R " SXU40 UD6$ )Nr r r s r1 r CallbackHandler.on_push_begin s WOOOr4 c U R Ha n[ Xa5 " UUU4U R U R U R U R
U R U R S.UD6nUc M_ UnMc U$ )N)r r r r r r )r rU r r r r r r )r. eventrY r- r r r/ results r1 r CallbackHandler.call_event s H X- jj!%!6!6 .. ! . .!%!6!6 $ 4 4 F ! ' r4 )r r r r r r r N) r* rg rh ri rj r r r r propertyr r r r r r r r r r r r r r r r r r r r r rq rJ r4 r1 r r s2 H& "
, I IN 1 N, NQ_ NQ#4 Q\ QTb QO!2 O< OR` OQ#4 Q\ QTb QO!2 O< OR` OP"3 PL PSa P X*; XL X[i XT&7 T TWe TQ#4 Q\ QTb QN 1 N, NQ_ N_ 1 _, _Q_ _^ 0 ^ ^P^ ^J - Jl J^ JT, T\ TN TU'8 U UXf UP"3 PL PSa Pr4 r c D \ rS rSrSrS\S\S\4S jrS\S\S\4S jr Sr
g )
r i3 zp
A [`TrainerCallback`] that handles the default flow of the training loop for logs, evaluation and checkpoints.
rY r- r c z UR S:X a UR ( a SUl UR [ R
:X a$ UR UR - S:X a SUl UR [ R
:X a> UR UR - S:X a! UR UR :: a SUl
UR [ R
:X a4 UR S: a$ UR UR - S:X a SUl UR UR : a SUl UR [ R
:X a> UR UR - S:w a! UR UR :: a SUl
UR [ R
:X a SUl U$ )Nr Tr )r logging_first_stepr logging_strategyr STEPSr
eval_strategyr
eval_delayr
save_strategyr r r r r r s r1 r DefaultFlowCallback.on_step_end8 sm !d&=&=!%G $4$:$: :u?P?PSXSfSf?fjk?k!%G
"2"8"8 8!!E$4$4495#4#44&*G #
,"4"4 4 1$!!E$4$449"&G /+/G ( ""&6&<&<<%%(8(88A=OOu'8'88*.'!!\%7%77&*#r4 c UR [ R :X a SUl UR [ R :X a! UR
UR :: a SUl UR [ R :X a SUl
U$ )NT) r% r EPOCHr r' r( r r r) r r r s r1 r DefaultFlowCallback.on_epoch_end` sp $4$:$: :!%G !1!7!7 7DOOu{{ z/. Consider increasing `max_str_len` if needed.].4gr )
r r2 r{ r$ rn r? r0 rk poprD )
r. rY r- r r r shallow_logsr} r~ _s
r1 r ProgressCallback.on_log s & &4+<+<+H L
a%%#a&43C3C*C?Axs4K[K[J\ ]H H !O 5)))*3LO&'O % t4A # #C $5 6! ,I &r4 c j UR ( a" U R R 5 S U l g g r )r r2 rC r s r1 r ProgressCallback.on_train_end s* & & # # % $D 'r4 )r8 r0 r3 r2 )d r )r* rg rh ri rj rm r r r r r r r r rq rJ r4 r1 r/ r/ p s6
'C '
2
*' ' 7&%r4 r/ c " \ rS rSrSrSS jrSrg)PrinterCallbacki z7
A bare [`TrainerCallback`] that just prints the logs.
Nc UR SS 5 nUR ( aM Ub> UR 5 VVs0 s H! u pxU[ U[ 5 ( a US OU_M# nnn[ U5 g g s snnf )Nr rI )rJ r r{ r$ rk print) r. rY r- r r r rL r} r~ s r1 r PrinterCallback.on_log sh HH\4 ( & & SWS]S]S_`S_41*Q*>*>qgAES_` $K '`s (A2rJ r )r* rg rh ri rj r rq rJ r4 r1 rR rR s r4 rR c T \ rS rSrSrS
S\S\S- 4S jjrS rS r S r
S
\ 4S jr S r
g)EarlyStoppingCallbacki a
A [`TrainerCallback`] that handles early stopping.
Args:
early_stopping_patience (`int`):
Use with `metric_for_best_model` to stop training when the specified metric worsens for
`early_stopping_patience` evaluation calls.
early_stopping_threshold(`float`, *optional*):
Use with TrainingArguments `metric_for_best_model` and `early_stopping_patience` to denote how much the
specified metric must improve to satisfy early stopping conditions. `
This callback depends on [`TrainingArguments`] argument *load_best_model_at_end* functionality to set best_metric
in [`TrainerState`]. Note that if the [`TrainingArguments`] argument *save_steps* differs from *eval_steps*, the
early stopping will not occur until the next save step.
early_stopping_patienceearly_stopping_thresholdNc * Xl X l SU l g )Nr rX rY early_stopping_patience_counter)r. rX rY s r1 r EarlyStoppingCallback.__init__ s '>$(@%/0,r4 c 2 UR ( a [ R O[ R nUR b< U" XBR 5 ( a- [ XBR -
5 U R : a SU l g U =R S-
sl g )Nr r )greater_is_betternpgreaterlessr absrY r\ )r. rY r- r metric_valueoperators r1 check_metric_value(EarlyStoppingCallback.check_metric_value sk !%!7!72::RWW $ \#4#4 5 5L#4#4458U8UU34D 0 0 0A 5 0r4 c UR ( d [ R S5 UR c S5 eUR [
R :w d S5 eg )NzUsing EarlyStoppingCallback without load_best_model_at_end=True. Once training is finished, the best model will not be loaded automatically.zBEarlyStoppingCallback requires metric_for_best_model to be definedzAEarlyStoppingCallback requires IntervalStrategy of steps or epoch)load_best_model_at_endr r metric_for_best_modelr' r NOr s r1 r $EarlyStoppingCallback.on_train_begin sb ** NN^
))5
P
5 !!%5%8%88
O
8r4 c UR nUR S5 ( d SU 3nUR U5 nUc [ R SU S35 g U R XX75 U R U R : a SUl g g )Neval_z@early stopping required metric_for_best_model, but did not find z so early stopping is disabledT) rj
startswithgetr r rf r\ rX r )r. rY r- r r r metric_to_checkrd s r1 r !EarlyStoppingCallback.on_evaluate s 44 ))'22 %o%6 7O {{?3 NNRSbRc d
WC / /43O3O O+/G ( Pr4 rt c R U R U R S.SU R 0S.$ )N)rX rY r\ r r[ rw s r1 r- EarlyStoppingCallback.state s7 ,0+G+G,0,I,I
243W3W
r4 )rX r\ rY )r g )r* rg rh ri rj rm rk r rf r r r% r- rq rJ r4 r1 rW rW s< 1 1SX[_S_ 1 6
0"
t
r4 rW ) rj rA r? rV r numpyr` tqdm.autor
trainer_utilsr r r
training_argsr utilsr
get_loggerr* r r r& r r! r r r/ rR rW rJ r4 r1 r{ s ! E E ,
H % WE WE WEt) )X :
_ :
:
zC C LCo CL:/ :zG% G%T
o
I
O_ I
r4