BVh)dZddlmZddlmZddlmZddlmZddlmZddl m Z ddl m Z dd lmZdd lmZej$d ed d g Gdde j&Zy)zContains LossScale classes.)distribute_lib)indexed_slices) smart_cond)control_flow_ops)math_ops) optimizer) loss_scale) deprecation) tf_exportz3train.experimental.MixedPrecisionLossScaleOptimizerz0mixed_precision.MixedPrecisionLossScaleOptimizer)v1ceZdZdZfdZdZdejjdddfdZ dZ dZ d Z dd Z dd Zd Zd ZdZdZdZdZxZS) MixedPrecisionLossScaleOptimizeraAn optimizer that applies loss scaling. Loss scaling is a process that multiplies the loss by a multiplier called the loss scale, and divides each gradient by the same multiplier. The pseudocode for this process is: ``` loss = ... loss *= loss_scale grads = gradients(loss, vars) grads /= loss_scale ``` Mathematically, loss scaling has no effect, but can help avoid numerical underflow in intermediate gradients when float16 tensors are used for mixed precision training. By multiplying the loss, each intermediate gradient will have the same multiplier applied. The loss scale can either be a fixed constant, chosen by the user, or be dynamically determined. Dynamically determining the loss scale is convenient as a loss scale does not have to be explicitly chosen. However it reduces performance. This optimizer wraps another optimizer and applies loss scaling to it via a `LossScale`. Loss scaling is applied whenever gradients are computed, such as through `minimize()`. ct|tjstdt |z||_|j }|j}tt|+||tj||_ |j td|j|j d|j|jdy)Nz3"opt" must be an instance of Optimizer, but got: %szloss_scale cannot be Nonebase_optimizerr ) isinstancer Optimizer ValueErrortype _optimizer _use_lockingget_namesuperr__init__loss_scale_moduleget _loss_scale_track_trackable)selfoptr use_lockingname __class__s l/home/dcms/DCMS/lib/python3.12/site-packages/tensorflow/python/training/experimental/loss_scale_optimizer.pyrz)MixedPrecisionLossScaleOptimizer.__init__<s c9.. / LCy! ""DO""K <<>D *D:;M(,,Z8D  2 33$//+;<$**L9cJt|jtjS)z:Check if `_loss_scale` dynamically manages the loss scale.)rrrDynamicLossScalers r#_doing_dynamic_loss_scalingzMixedPrecisionLossScaleOptimizer._scale_loss..new_losss$6(-- HNNCCCr$)rcallablerr?r@)rr*rBr s ` @r#r0z,MixedPrecisionLossScaleOptimizer._scale_losss?!!#J~Do HMM*djj9 99r$c~|j}d|z }|Dcgc]}|dn|j||c}Scc}w)N)r _scale_grad)rr8r loss_scale_reciprocalr6s r#r2z/MixedPrecisionLossScaleOptimizer._unscale_gradssS!!#J N   t//3HII  s:ct|tjr:|j|z}tj||j|j S||zSr>)rr IndexedSlicesvaluesindices dense_shape)rgradrG grad_valss r#rFz,MixedPrecisionLossScaleOptimizer._scale_gradsQ$445++ 55i  ) ))T\\*.*:*:<< ' ''r$ctjr td|js|jj |||Stj }t|}|j|j|||fS)aRApply gradients to variables. This is the second part of `minimize()`. It returns an `Operation` that conditionally applies gradients if all gradient values are finite. Otherwise no update is performed (nor is `global_step` incremented). Args: grads_and_vars: List of (gradient, variable) pairs as returned by `compute_gradients()`. global_step: Optional `Variable` to increment by one after the variables have been updated. name: Optional name for the returned operation. Default to the name passed to the `Optimizer` constructor. Returns: An `Operation` that conditionally applies the specified gradients. If `global_step` was not None, that operation also increments `global_step`. Raises: RuntimeError: If you should use `_distributed_apply()` instead. z6apply_gradients() must be called in a replica context.args) rin_cross_replica_contextrr(rapply_gradientsget_replica_contexttuple merge_call_distributed_apply)rr5 global_stepr!replica_contexts r#rSz0MixedPrecisionLossScaleOptimizer.apply_gradientss,..0 O PP  + + - __ , ,^[$ OO$88:O>*N  % % ~{D&I & KKr$c0njDcgc]\}}| }}}jj|\}} fd} tj| | tj } t j | |Scc}}w)aA version of `apply_gradients` for cross replica context. When users are in a cross replica strategy, they must call this rather than `apply_gradients()`. Args: distribution: a `DistributionStrategy` object. grads_and_vars: List of (gradient, variable) pairs as returned by `compute_gradients()` and then aggregated across replicas. global_step: Optional (mirrored) `Variable` to increment by one after the variables have been updated. name: Optional name for the returned operation. Default to the name passed to the `Optimizer` constructor. Returns: An `Operation` that applies the specified gradients across all replicas. If `global_step` was not None, that operation also increments `global_step` c2jdzS)Nz-wrapped)_apply_gradients) distributionrXr5r!rsr#apply_fnzEMixedPrecisionLossScaleOptimizer._distributed_apply..apply_fns$  " "<#'*#466r$)r!)rrupdaterrno_opgroup) rr]r5rXr!r6r7r8loss_scale_update_opshould_apply_gradsr^maybe_apply_ops ````` r#rWz3MixedPrecisionLossScaleOptimizer._distributed_applys0#4D) *41aQ *E *040@0@0G0G0N,,66 **+=x+;+A+ACN  ! !,4 99 +s Bc|jj|jj|||f}|j |S)z9Unconditionally apply gradients in cross replica context.rP)extendedcall_for_each_replicarrSra)rr]r5rXr! update_opss r#r\z1MixedPrecisionLossScaleOptimizer._apply_gradientssG&&<< ''k4 0=2J   j ))r$ctdz%This function should never be called.z$This function should never be called RuntimeErrorrrMvars r# _apply_sparsez.MixedPrecisionLossScaleOptimizer._apply_sparse = >>r$ctdrjrkrms r# _apply_densez-MixedPrecisionLossScaleOptimizer._apply_denserpr$ctdrjrk)rrMhandlerKs r#_resource_apply_sparsez7MixedPrecisionLossScaleOptimizer._resource_apply_sparserpr$ctdrjrk)rrMrts r#_resource_apply_densez6MixedPrecisionLossScaleOptimizer._resource_apply_denserpr$c|jjt|jjj zS)z'Returns the variables of the Optimizer.)rr:r3r_weightsrJr's r#r:z*MixedPrecisionLossScaleOptimizer.variabless9 OO % % ' !!**113 4 56r$)NN)__name__ __module__ __qualname____doc__rr(rrGATE_OPr1r0r2rFrSrWr\rorrrurwr: __classcell__)r"s@r#rrsx 8: L "&'0':':'B'B+/49"& .0`:(!KL&*" #9J*????6r$rN)r}tensorflow.python.distributertensorflow.python.frameworkrrtensorflow.python.opsrrtensorflow.python.trainingr'tensorflow.python.training.experimentalr rtensorflow.python.utilr tensorflow.python.util.tf_exportr deprecated_endpointsrrr$r#rst"7622*0S.6"!!9; ADFG\6y':':\6G;\6r$