2VhgddlZddlZddlmZddlmZddlmZddlmZddlm Z ddl m Z ddl m Z dd lmZGd d e Zd Zd ZdZy)N)backend) initializers)ops)learning_rate_schedule)serialization_lib) KerasSaveable)tracking) auto_nameceZdZdZ d0dZedZdZdZe jdZ dZ edZ d Z d1d Z d2d Z d3d Zd ZdZdZdZdZdZd4dZdZdZdZdZdZdZedZej@dZdZ!dZ"dZ#dZ$d Z%d!Z&d"Z'd5d#Z(d$Z)d%Z*d&Z+d'Z,d(Z-d)Z.d*Z/d+Z0e1d4d,Z2fd-Z3d4d.Z4d/Z5xZ6S)6 BaseOptimizeraAbstract optimizer base class. If you intend to create your own optimization algorithm, please inherit from this class and override the following methods: - `build`: Create your optimizer-related variables, such as momentum variables in the SGD optimizer. - `update_step`: Implement your optimizer's variable updating logic. - `get_config`: serialization of the optimizer. Example: ```python class SGD(Optimizer): def __init__(self, **kwargs): super().__init__(**kwargs) self.momentum = 0.9 def build(self, variables): super().build(variables) self.momentums = [] for variable in variables: self.momentums.append( self.add_variable_from_reference( reference_variable=variable, name="momentum" ) ) def update_step(self, gradient, variable, learning_rate): learning_rate = ops.cast(learning_rate, variable.dtype) gradient = ops.cast(gradient, variable.dtype) m = self.momentums[self._get_variable_index(variable)] self.assign( m, ops.subtract( ops.multiply(m, ops.cast(self.momentum, variable.dtype)), ops.multiply(gradient, learning_rate), ), ) self.assign_add(variable, m) def get_config(self): config = super().get_config() config.update( { "momentum": self.momentum, "nesterov": self.nesterov, } ) return config ``` c  Fd|_| jddtjd| rt d| | t |j j} | |_||_ ||_ ||_ ||_ ||_ | |_| |_| r| dk\st d| |r=|dkDs|dkrt d ||r#t!|t"r|dkrt d |||_||_t)d |||fD} | dkDrt d |d |d|d|_g|_g|_t1j2dd|j,fi|_i|_t9j:|j|5t9j<ddddd}ddd|j?||_ t!|tBjDr||_#ytI|r||_#yt!|tJst d|t9j:|j|5t9j<|dt9jLdd}ddd|j?|||_#y#1swYxYw#1swY.xYw)NFdecayz= 2. Received: gradient_accumulation_steps=rzC`ema_momentum` must be in the range [0, 1]. Received: ema_momentum=z]`ema_overwrite_frequency` must be an integer >= 1 or None. Received: ema_overwrite_frequency=c3$K|]}|du ywN).0as S/home/dcms/DCMS/lib/python3.12/site-packages/keras/src/optimizers/base_optimizer.py z)BaseOptimizer.__init__..s ATM szYOnly one of `clipnorm`, `clipvalue` and `global_clipnorm` can be set. Received: clipnorm=z , clipvalue=z, global_clipnorm= variablesc6t|tjSr) isinstancerVariable)xs rz(BaseOptimizer.__init__..sjG,<,<=caller iterationintonly_first_replica)namedtype trainable aggregationzArgument `learning_rate` should be float, or an instance of LearningRateSchedule, or a callable (that takes in the current iteration value and returns the corresponding learning rate value). Received instead: learning_rate= learning_rate)'_lockpopwarningswarn ValueErrorr __class____name__r$ weight_decayclipnormglobal_clipnorm clipvalueuse_emaloss_scale_factorgradient_accumulation_stepsrr" ema_momentumema_overwrite_frequencysumbuilt _variables_trainable_variablesr Tracker_tracker_trainable_variables_indicesr name_scoper_track_variable _iterationsrLearningRateSchedule_learning_ratecallablefloatfloatx)selfr(r0r1r3r2r4r7r8r5r6r$kwargs clip_args_sum iterationss r__init__zBaseOptimizer.__init__Ds ::gt $ 0 MMN  ;F8DE E <T^^445D (  ." !2+F( &.!3 =235 a$ $,i#I   1 ..6Z8&K'9/9JL   $&! ((=OO   -/)    $ 7  )) 0 J  Z(%  1FF #0D  m $"/D mU3 78Eo G##DIId;  ' 0 0!(!..*# 4 !     /"/D I  6  sJ =-J JJ c|jr*tj|j|jS|jSr)r6r floor_dividerBrHs rrKzBaseOptimizer.iterationss=  + +##  $"B"B rc<|jjd|y)Nr)r> add_to_storerHvariables rrAzBaseOptimizer._track_variables "";9rct|ddS)Noverwrite_with_gradientF)getattrrRs r!_overwrite_variable_with_gradientz/BaseOptimizer._overwrite_variable_with_gradientsx!:EBBrcf|jr|j|d|_|jrg|_t |D]\\}}||j |j|<|js1|jj|j|d^|dd|_ d|_ y)Naveragegradient_accumulator)r$T) r4add_optimizer_variables_model_variables_moving_averager6_accumulated_gradients enumerater?_var_keyappendadd_variable_from_referencer<r:)rHrirSs rbuildzBaseOptimizer.builds <<373O3O94D 0  + +*,D '$Y/ KAxIJD - -dmmH.E F//++2244 35 %.aL! rct|Sr)idrRs rr_zBaseOptimizer._var_keys (|rc |jddSr)r;rOs rrzBaseOptimizer.variablessq!!rc>|j|j|Sr)r?r_rRs r_get_variable_indexz!BaseOptimizer._get_variable_indexs00x1HIIrc |jtj|}tj|j |5tj |||d|||}ddd|j|S#1swYxYw)aGAdd a variable to the optimizer. Args: shape: Shape tuple for the variable. Must be fully-defined (no `None` entries). initializer: Initializer object to use to populate the initial variable value, or string name of a built-in initializer (e.g. `"random_normal"`). Defaults to `"zeros"`. dtype: Dtype of the variable to create, e.g. `"float32"`. If unspecified, defaults to the `keras.backend.floatx()`. aggregation: Optional string, one of `None`, `"none"`, `"mean"`, `"sum"` or `"only_first_replica"`. Annotates the variable with the type of multi-replica aggregation to be used for this variable when writing custom data parallel training loops. Defaults to `"none"`. layout: Optional tensor layout. Defaults to `None`. name: String name of the variable. Useful for debugging purposes. Returns: An optimizer variable, in the format of `keras.Variable`. rF) initializershaper%r&r'layoutr$N)_check_super_calledrgetrr@r$rrA)rHrkrjr%r'rlr$rSs r add_variablezBaseOptimizer.add_variables<   ""&&{3    $ 7 ''''H  X&  s A??Bc L|xsd}t|dr#|jjdddz|z}n;t|jjddjdddz|z}|j |j ||j|t|ddS) a]Add an optimizer variable from the model variable. Create an optimizer variable based on the information of model variable. For example, in SGD optimizer momemtum, for each model variable, a corresponding momemtum variable is created of the same shape and dtype. Args: reference_variable: `keras.Variable`. The corresponding model variable to the optimizer variable to be created. name: Optional string. The name prefix of the optimizer variable to be created. If not provided, it will be set to `"var"`. The variable name will follow the pattern `{variable_name}_{reference_variable.name}`, e.g., `momemtum/dense_1`. Defaults to `None`. initializer: Initializer object to use to populate the initial variable value, or string name of a built-in initializer (e.g. `"random_normal"`). If unspecified, defaults to `"zeros"`. Returns: An optimizer variable, in the format of `keras.Variable`. varpath/_:_layoutN)rkrjr%r$rl) hasattrrrreplacestrr$rorkr%rV)rHreference_variabler$rjs rraz)BaseOptimizer.add_variable_from_references2}u %v .%**223FFsCP    $**#$**-y$? !  rc f|}|}t|tr|g}|g}n9t|tst|tjr|gt |z}t |t |k7rt d|d|dt d|D}|D]}|j|sGtt||D].\}\} } ||j|j|| | 0[tt |D]}||jdt|tr|dS|S)aAdd optimizer variables from the list of trainable model variables. Create an optimizer variable based on the information of the supplied model variables. For example, in SGD optimizer momemtum, for each model variable, a corresponding momemtum variable is created of the same shape and dtype. Note that trainable variables with `v.overwrite_with_gradient == True` will insert `None`, into the output list, since the optimizer variable will not be used anyways, and could be wasteful. Args: trainable_variables: `keras.Variable`, the corresponding model variable to the optimizer variable to be created. name: The name prefix(es) of the optimizer variable(s) to be created. Can be a single string or list of strings. If a list of strings, will create an optimizer variable for each prefix. The variable name will follow the pattern `{variable_name}_{trainable_variable.name}`, e.g., `momemtum/dense_1`. initializer: Initializer object(s) to use to populate the initial variable value(s), or string name of a built-in initializer (e.g. `"random_normal"`). If unspecified, defaults to `"zeros"`. Returns: A list of optimizer variables, in the format of `keras.Variable`s. If multiple names are provide, returns a tuple of lists. z]The number of provided names must match the number of provided initializers. Received name='z', initializer=''c3 K|]}gywrr)rrts rrz8BaseOptimizer.add_optimizer_variables..~s#:1B#:s )r$rjNr) rryr Initializerlenr-tuplerWr^zipr`rarange) rHtrainable_variablesr$rj name_listinitializer_listoptimizer_variablesrSrbvar_namevar_inits rr[z%BaseOptimizer.add_optimizer_variablesHsi@ & dC I +} +s+z\550%0=3y>#A y>S!12 2::>@ +}A/ $#: #::+ 8H99(C/8 #340 +A+((*1188$!)(09 s9~.8A'*11$78 8$ dC &q) )""rcj|D].}|j||jvs!td|dy)NzUnknown variable: z. This optimizer can only be called for the variables it was originally built with. When working with a new set of variables, you should recreate a new optimizer instance.)r_r?r-)rHrvs r_check_variables_are_knownz(BaseOptimizer._check_variables_are_knownsF A}}Qt'H'HH (,99 rc&|j|y)aAssign a value to a variable. This should be used in optimizers instead of `variable.assign(value)` to support backend specific optimizations. Note that the variable can be a model variable or an optimizer variable; it can be a backend native variable or a Keras variable. Args: variable: The variable to update. value: The value to add to the variable. N)assignrHrSvalues rrzBaseOptimizer.assigns rc&|j|y)aAdd a value to a variable. This should be used in optimizers instead of `variable.assign_add(value)` to support backend specific optimizations. Note that the variable can be a model variable or an optimizer variable; it can be a backend native variable or a Keras variable. Args: variable: The variable to update. value: The value to add to the variable. N) assign_addrs rrzBaseOptimizer.assign_add E"rc&|j|y)aSubtract a value from a variable. This should be used in optimizers instead of `variable.assign_sub(value)` to support backend specific optimizations. Note that the variable can be a model variable or an optimizer variable; it can be a backend native variable or a Keras variable. Args: variable: The variable to update. value: The value to add to the variable. N) assign_subrs rrzBaseOptimizer.assign_subrrctr)NotImplementedError)rHgradientrSr(s r update_stepzBaseOptimizer.update_steps!!rcTt|\}}|j|||jSr)rapplyrB)rHgrads_and_varsgradsrs rapply_gradientszBaseOptimizer.apply_gradientss-%(.%9"" 5-.rct|dk(ry|s|js tdt|t|jk7r.tdt|dt|jd|j}njt |}|jsBt j|j|5|j|dddd|_|j|t j|j|5|j||\}}|j||\}}tt |dkDrj|j}||Dcgc] }||n||z  }}|j|||D]/}|j|j!|j|1ddd|j"j%d y#1swYxYwcc}w#1swY7xYw) a3Update traininable variables according to provided gradient values. `grads` should be a list of gradient tensors with 1:1 mapping to the list of variables the optimizer was built with. `trainable_variables` can be provided on the first call to build the optimizer. rNzWhen passing `grads` without `variables`, the optimizer must already be built on a list of variables. Call `optimizer.build(trainable_variables)` first. zWhen passing `grads` as a list of gradient tensors, the gradients must match `optimizer.variables` one-to-on. Received a list of z* gradients, but the optimizer is tracking z trainable variables.rTr)rr:r-r?r<listrr@r$rcr_filter_empty_gradients,_overwrite_variables_directly_with_gradientsr5_backend_apply_gradients constraintrrBr)rHrrscalegrSs rrzBaseOptimizer.applys u:?   &:: J 5zS!B!BCC **-e*6--01J1J-K,LM++#'";"; "&':"; ::'' $?4JJ234!  + +,? @    $ 7 G)-)E)E** &E&AA. 'E& 4;!#..$DIJq!)QU:JEJ--e5HI 3GH**6 (;(;H(EFG/ G8 ##A&C44.K% G Gs1=G AG2$G-4%G2"G2 G*-G22G;crjrsjdzjzdk(}Dcgc] }jj|"c}fdt j |fdfdn?j jjjjrcjjjr;jdzjzdk(}t j |fddyyycc}w) arApply method that can be overridden by different backends. JAX overrides it in order to deal with statelessness in gradient accumulation and EMA handling. The below implementation is intended to be generally backend-agnostic, but may not work with all backends. This method does 4 things: - Call the optimizer's update_step() to update trainable variables and optimizer variables. - Update EMA variables, if EMA is configured. - Update gradient accumulators, if gradient accumulation is configured. - Update the iteration counter. rrcj}t|Dcgc] \}}||z|z }}}j|}j|j ||j j ycc}}wr)r6r_clip_gradients_apply_weight_decay_backend_update_stepr($_backend_reset_gradient_accumulators)rrstepsracc_g acc_gradsrHs r_update_step_fnz?BaseOptimizer._backend_apply_gradients.._update_step_fn4s888;E98M,4AuQY%' ,,U3(()<=)).0B0B99;sBcSrr)rrrsrrz8BaseOptimizer._backend_apply_gradients..Fs/BCrc(jSr)(_backend_increment_gradient_accumulators)rrrHsrrz8BaseOptimizer._backend_apply_gradients..GsEE9rc:jjSr)-_overwrite_model_variables_with_average_valuer<rOsrrz8BaseOptimizer._backend_apply_gradients..asDNN11rcyrrrrrrz8BaseOptimizer._backend_apply_gradients..dsrN)r6rBr]rhrcondrrrr(r4&_update_model_variables_moving_averager<r8rK)rHrris_update_steprshould_overwrite_model_varsrrs``` @@rrz&BaseOptimizer._backend_apply_gradientss>  + +  1$001456N-++D,D,DQ,GHI  < HHC ((/E  $ $%8 9  % %*D,>,>  <<  7 7)) ++OOa'00/145/6+/!  , Ms%D4cRt||D]\}}|j|||y)zCollective update_step that can be overridden by the backend. It is overridden by torch for performance reasons, and by TF to support tf.distribute. N)rr)rHrrr(gradrqs rrz"BaseOptimizer._backend_update_stepgs1 U$78 7ID#   T3 6 7rc|jD]?}||jtj|j|j AyN)r%)r]rrzerosrkr%)rHg_accs rrz2BaseOptimizer._backend_reset_gradient_accumulatorsps>00 HE  SYYu{{%++FG Hrct||Dcgc] \}}||z }}}t||D]\}}|j|ycc}}wr)rr)rHrrrr new_g_accsn_g_accrs rrz6BaseOptimizer._backend_increment_gradient_accumulatorsusP25eY2GHhaq5yH H!*i8 "NGU LL ! "IsAc  |j|js#td|jjdt |t |j k7rEtd|jjdt |dt |j dt |t |jk7r.tdt |dt |jdtt|j|tt|j |z}tj| 5}|j|dddg}|jD]8}j|}||j|(|j|:g}|j D]8}j|}||j|(|j|:||fS#1swYxYw) NzTo call `stateless_apply`, zx must be built (i.e. its variables must have been created). You can build it via `optimizer.build(trainable_variables)`.zNArgument `optimizer_variables` must be a list of tensors corresponding 1:1 to z(().variables. Received list with length z, but expected z variables.zArgument `optimizer_variables` must be a list of tensors corresponding 1:1 to the trainable variables list that the optimizer was built with. Received len(trainable_variables) == z& whereas the optimizer was built with ) state_mapping)rmr:r-r.r/rrr<rrrStatelessScoperget_current_valuer`)rHrrrmappingscopernew_vs rstateless_applyzBaseOptimizer.stateless_applyzs   "zz-dnn.E.E-FGOO  " #s4>>': :((,(?(?'@A--01D-E,FG/0 =  " #s4+D+D'E E/033F/G.HI8t0012+ ?  ))+> ? T^^%89 :;  # #' : e JJu  !** .A++A.E #**51#**1-  . ! .A++A.E #**51#**1-  . #$777%  s G::Hc<|j||jzS|S)zScale the loss before computing gradients. Scales the loss before gradients are computed in a `train_step`. This is primarily useful during mixed precision training to prevent numeric underflow. )r5)rHlosss r scale_losszBaseOptimizer.scale_losss&  ! ! -$000 0 rc"|jSr)_get_current_learning_raterOs rr(zBaseOptimizer.learning_rates..00rct|jtjr |j}nd}t|tj r||_n]t |r||_nJt|jtj r td|jj||7t|jtjs|j|yyy)NaThis optimizer was created with a `LearningRateSchedule` object as its `learning_rate` constructor argument, hence its learning rate is not settable. If you need the learning rate to be settable, you should instantiate the optimizer with a float `learning_rate` argument.) rrDrrrrCrE TypeErrorr_untrack_variable)rHr( prev_lr_vars rr(zBaseOptimizer.learning_rates d))7+;+; <--KK  1FF #0D  m $"/D ##%;%P%P K    & &} 5  ":   !1!1,   " "; / , "rc V|js tdt|j|D]y\}}|j|jk7rJtd|j |dt |jdt |jd|j|{y)z!Set the weights of the optimizer.zYou are calling `set_weights()` on an optimizer that has not yet been built. Please call `optimizer.build(trainable_variables)` to create the optimizer weights before calling `set_weights()`.zOptimizer variable z has shape z+ not compatible with provided weight shape .N)r:r-rr;rkr_ryr)rHweightsrSweights r set_weightszBaseOptimizer.set_weightsszzD  !$DOOW = $ Hf~~- )$--*A)B+8>>*+,$$' $5#6a9 OOF # $rcvt|jD]!\}}|j|t|<#y)z'Get the state of this optimizer object.N)r^rnumpyry)rHstorerbrSs rsave_own_variablesz BaseOptimizer.save_own_variabless2$T^^4 -KAx$NN,E#a&M -rc t|jt|jk7rud|jdt|jdt|jd}t|jdk(r|dz }t j |dy t |jD]"\}}|j|t|$y ) z'Set the state of this optimizer object.z)Skipping variable loading for optimizer 'z', because it has z+ variables whereas the saved optimizer has z variables. rzCThis is likely because the optimizer has not been called/built yet.r) stacklevelN) rkeysrr$r+r,r^rry)rHrmsgrbrSs rload_own_variablesz BaseOptimizer.load_own_variabless uzz| DNN 3 3;DII;G""%dnn"5!67++.uzz|+<*=\K  4>>"a'( MM#! , $T^^4 +KAx OOE#a&M * +rct|jtjr|j|jSt |jr|jS|jSr)rrDrrCrBrErOs rrz(BaseOptimizer._get_current_learning_rates^    !7!L!L &&t'7'78 8 d)) *&&( ("""rc tfd|Ds||fSt|}t|}tt|dz ddD]}||||c j s!j rj }j dz|zdk(}jj  tj| fd fd}tj| fd fd tj| fd  fd } j|  j|n j |j||j|||fS) a.Overwrite the variables directly by their gradients. This method is designed for a special case where we want to overwrite the variable directly with its computed gradient. For example, in float8 training, new `scale` and `amax_history` are computed as gradients, and we want to overwrite them directly instead of following the typical procedure such as gradient descent with a learning rate, gradient clipping and weight decaying. After the update, the processed pairs will be filtered out. c3@K|]}j|ywr)rW)rrrHs rrzMBaseOptimizer._overwrite_variables_directly_with_gradients..sK499!<KrrcZtjjjSr)rrrkr%rsrrzLBaseOptimizer._overwrite_variables_directly_with_gradients..1s !'' Arc0tjSrrmaximumrrsrrzLBaseOptimizer._overwrite_variables_directly_with_gradients..2 Au 5rc0tjSrrrsrrzLBaseOptimizer._overwrite_variables_directly_with_gradients..6rrcSrrrsrrzLBaseOptimizer._overwrite_variables_directly_with_gradients..7srcSrr)new_gsrrzLBaseOptimizer._overwrite_variables_directly_with_gradients..:srcjSr)r)rsrrzLBaseOptimizer._overwrite_variables_directly_with_gradients..:s qwwr) anyrrrrWr6rBr]rhrrrr*)rHrvarsfiltered_grads filtered_varsrbrr new_g_accrrrrrs` @@@@rrz:BaseOptimizer._overwrite_variables_directly_with_gradients sZKdKK$; eT s>*Q.B7 %A!!$mA&6DAq55a833 <},,rct|}t|}g}tt|dz ddD]F}|| |j||j|} |j |j H|s td|r+tjdtt|d||fS#t $r|j |jYwxYw)Nrrz'No gradients provided for any variable.z%Gradients do not exist for variables zc when minimizing the loss. If using `model.compile()`, did you forget to provide a `loss` argument?) rrrr*r`rrAttributeErrorr$r-r+r,reversed)rHrrrrmissing_grad_varsrbrs rrz%BaseOptimizer._filter_empty_gradientsDseT s>*Q.B7 5Aa (""1%!%%a(5%,,QVV4  5FG G  MM7!23456##  },,&5%,,QVV45sB99$C C c|jr2|jdkDr#|Dcgc]}||j|n|c}S|jr%|jdkDrt||jS|jrE|jdkDr6|j}|Dcgc]}|t j || |n| c}S|Scc}wcc}w)Nr)r1 _clip_by_normr2clip_by_global_normr3rclip)rHrrrs rrzBaseOptimizer._clip_gradients_s ==T]]Q.GLBC""1%A=  ! !d&:&:Q&>&ud.B.BC C ^^ 2AHMN1!-CHHQA&Q>N NLOs C#CcTtdrjr td|rtfd|D_nt_|rAt |dkDr3t jdjt|_ nd_ t_ y)aExclude variables from weight decay. This method must be called before the optimizer's `build` method is called. You can set specific variables to exclude out, or set a list of strings as the anchor words, if any of which appear in a variable's name, then the variable is excluded. Args: var_list: A list of `Variable`s to exclude from weight decay. var_names: A list of strings. If any string in `var_names` appear in the model variable's name, then this model variable is excluded from weight decay. For example, `var_names=['bias']` excludes all bias variables from weight decay. _builtzS`exclude_from_weight_decay()` can only be configured before the optimizer is built.c3@K|]}j|ywr)r_)rrSrHs rrz:BaseOptimizer.exclude_from_weight_decay..s2,4 h'2rr|N) rwrr-set_exclude_from_weight_decayrrecompilejoin"_exclude_from_weight_decay_patterndict _exclude_from_weight_decay_cache)rHvar_list var_namess` rexclude_from_weight_decayz'BaseOptimizer.exclude_from_weight_decayls 4 "t{{*  .128@2/D +/2eD + Y!+68jjY(7D 37;D 315-rc|j|}t|dst|_||jvr|j|St |dt }t |dd}||vrd|j|<y|1t j||j d|j|<yd|j|<y)Nr rr FT) r_rwr r rVrrsearchr$)rHrS variable_idr!exclude_from_weight_decay_patterns r_use_weight_decayzBaseOptimizer._use_weight_decaysmmH- t?@48FD 1 $?? ?88E E%, .% !-4 6- ) 3 3AFD 1 1+ > , 8 ;X]]KFK55kB=A--k:rc,|jy|D]}|j|stj|j|j }tj|j|j }|j |||z|zz yr)r0rrcastr(r%r)rHrrSlrwds rrz!BaseOptimizer._apply_weight_decays|    $ ! ?H%%h/XXd00(..AXXd//@8b=2+= =>  ?rcbt|ds#td|jjdy)Nr)zIn optimizer 'zh', you forgot to call `super().__init__()` as the first statement in the `__init__()` method. Go add it!)rw RuntimeErrorr.r/rOs rrmz!BaseOptimizer._check_super_calleds:tW% !8!8 9: &rc4|jrt||jD]r\}}| tj|j d}tj ||j|jz}|j||zd|z |zztyy)z8Update the stored moving average using the latest value.Nrr) r4rr\r not_equalrKrr%r7r)rHrrqrYnot_first_stepmomentums rrz4BaseOptimizer._update_model_variables_moving_averages << ##T%I%I! N W&%(]]4??A%FN;d>O>OONN8g#5X8L#LM N rct|t|jk7r.tdt|dt|jdt||jD]\}}| |j |y)z2Overwrite model variables with its moving average.zThe length of model variables (zT) to override does not match the length of model variables stored in the optimizer (z:). Please check if the optimizer was called on your model.N)rr\r-rr)rHrrq average_vars rrz;BaseOptimizer._overwrite_model_variables_with_average_values " #s  0 0(  1#6I2J1KL<<=>?CC !$ !E!E!  ( C& ;'  (rc@|jr|j|yy)aSet the final value of model's trainable variables. Sometimes there are some extra steps before ending the variable updates, such as overriding the model variables with its average value. Args: var_list: list of model variables. N)r4r)rHr s rfinalize_variable_valuesz&BaseOptimizer.finalize_variable_valuess  <<  > >x H rcy)N OptimizerrrOs r _obj_typezBaseOptimizer._obj_typesrc t|jtjr tj|j}nt|jt j r$t|jj}nltj|jrt|j}n7t|jr tj|j}nd}|j||j|j |j"|j$|j&|j(|j*|j,|j.d }|S)aReturns the config of the optimizer. An optimizer config is a Python dictionary (serializable) containing the configuration of an optimizer. The same optimizer can be reinstantiated later (without any saved state) from this configuration. Subclass optimizer should override this method to include other hyperparameters. Returns: Python dictionary. g?) r$r(r0r1r2r3r4r7r8r5r6)rrDrrC serializerrrFrr is_tensorrErserialize_keras_objectr$r0r1r2r3r4r7r8r5r6)rHr(configs r get_configzBaseOptimizer.get_configs    !7!L!L 3<<##M++W-=-= >!$"5"5";";"=>M ]]4.. /!$"5"56M d)) *-DD##M MII* -- #33|| --'+'C'C!%!7!7+/+K+K   rczd|vr0t|dtrtj|d||d<|di|S)aCreates an optimizer from its config. This method is the reverse of `get_config`, capable of instantiating the same optimizer from the config dictionary. Args: config: A Python dictionary, typically the output of get_config. custom_objects: A Python dictionary mapping names to additional user-defined Python objects needed to recreate this optimizer. Returns: An optimizer instance. r()custom_objectsr)rr rdeserialize_keras_object)clsr)r,s r from_configzBaseOptimizer.from_config%sK f $&148%>>/' }V}rc|dk7r|jt|dr|jj|}t|||S)Nr)r>)rmrwr>tracksuper __setattr__)rHr$rr.s rr3zBaseOptimizer.__setattr__=sH 7?  $ $ & 4 $MM''.Ew"4//rctjtj||d}|dkD}tj||tj|}tj|tj ||}tj ||j}tj|tj||jz }|S)NT)keepdimsr) rr9squarewhere ones_likesqrtmultiplyr1convert_to_tensorr) rHvaluesaxesl2sumpred l2sum_safel2norm intermediate values_clips rrzBaseOptimizer._clip_by_normHs 6*D4@qyYYtUCMM%,@A 4*!5u=||FDMM: ++L9CKK DMM=  rc|jj}|jj|jj||dur|jj yy)NT)r>lockedunlockuntracklock)rHrSprevious_lock_states rrzBaseOptimizer._untrack_variableUsQ"mm22  h' $ & MM    'r) NNNNFgGz?NNNN)rNnoneNN)Nr)rr)NN)7r/ __module__ __qualname____doc__rLpropertyrKrArWr no_automatic_dependency_trackingrcr_rrhrorar[rrrrrrrrrrrrrr(setterrrrrrrrrrrrmrrr!r$r* classmethodr/r3rr __classcell__)r.s@rr r s3p $$( }0~  :C../&""J  +\:A( V6=L#\  # #" C'JM^7H " 28h 1100:$$- +$#5-n-6 &7P:? N(( I-^. 0 !rr a2 name: String. The name to use for momentum accumulator weights created by the optimizer. weight_decay: Float. If set, weight decay is applied. clipnorm: Float. If set, the gradient of each weight is individually clipped so that its norm is no higher than this value. clipvalue: Float. If set, the gradient of each weight is clipped to be no higher than this value. global_clipnorm: Float. If set, the gradient of all weights is clipped so that their global norm is no higher than this value. use_ema: Boolean, defaults to `False`. If `True`, exponential moving average (EMA) is applied. EMA consists of computing an exponential moving average of the weights of the model (as the weight values change after each training batch), and periodically overwriting the weights with their moving average. ema_momentum: Float, defaults to 0.99. Only used if `use_ema=True`. This is the momentum to use when computing the EMA of the model's weights: `new_average = ema_momentum * old_average + (1 - ema_momentum) * current_variable_value`. ema_overwrite_frequency: Int or None, defaults to None. Only used if `use_ema=True`. Every `ema_overwrite_frequency` steps of iterations, we overwrite the model variable by its moving average. If None, the optimizer does not overwrite model variables in the middle of training, and you need to explicitly overwrite the variables at the end of training by calling `optimizer.finalize_variable_values()` (which updates the model variables in-place). When using the built-in `fit()` training loop, this happens automatically after the last epoch, and you don't need to do anything. loss_scale_factor: Float or `None`. If a float, the scale factor will be multiplied the loss before computing gradients, and the inverse of the scale factor will be multiplied by the gradients before updating variables. Useful for preventing underflow during mixed precision training. Alternately, `keras.optimizers.LossScaleOptimizer` will automatically set a loss scale factor. gradient_accumulation_steps: Int or `None`. If an int, model & optimizer variables will not be updated at every step; instead they will be updated every `gradient_accumulation_steps` steps, using the average value of the gradients since the last update. This is known as "gradient accumulation". This can be useful when your batch size is very small, in order to reduce gradient noise at each update step. EMA frequency will look at "accumulated" iterations value (optimizer steps // gradient_accumulation_steps). Learning rate schedules will look at "real" iterations value (optimizer steps). c|Dcgc]-}|tjtj|/}}tjtj|}tj|Scc}w)z-Computes the global norm of multiple tensors.)rr9r6stackr9) value_listr squared_norms squared_norms r global_normrXs`)3#$am 1 M77399]34L 88L !! s A6*A6ct|}|tjd|z d|z z}|||z z}|Dcgc] }|||zn| c}Scc}w)Ng?)rXrminimum)rU clip_normuse_normscale_for_finiterrs rrrs\:&H 3;;sX~sY#OO 8 3 4E7A B!AIA - BB BsA )rr+ keras.srcrrrkeras.src.optimizers.schedulesrkeras.src.savingrkeras.src.saving.keras_saveablerkeras.src.utilsr keras.src.utils.namingr r base_optimizer_keyword_argsrXrrrrresF "A.9$,L!ML!^"1h"Cr