BVhB"dZddlZddlmZddlmZddlmZddlmZddlm Z ddlm Z dd l m Z dd l m Z dd l mZdd l mZdd l mZddlmZddlmZddlmZddlmZej2ddegdGddej4ej6Zej2ddegdGddeZdZdZd Z ej2d!d"egd#Gd$d%eZ!d&Z"y)'zContains LossScale classes.N)distribute_lib) reduce_util)context)dtypes)indexed_slices)ops)cond)control_flow_ops)math_ops) variable_v1) variables)base) deprecation)nest) tf_export&mixed_precision.experimental.LossScaletrain.experimental.LossScale)zmixed_precision.LossScalerr)v1ceZdZdZdZej dZej dZd dZ e jjffd Z d fd Zej dZed ZxZS) LossScalea Base class for all TF1 loss scales. This is an abstract base class, so you cannot instantiate it directly. Instead, use one of its concrete subclasses: * `tf.compat.v1.mixed_precision.DynamicLossScale` * `tf.compat.v1.mixed_precision.FixedLossScale` Loss scaling is a process that multiplies the loss by a multiplier called the loss scale, and divides each gradient by the same multiplier. The pseudocode for this process is: ``` loss = ... loss *= loss_scale grads = gradients(loss, vars) grads /= loss_scale ``` Mathematically, loss scaling has no effect, but can help avoid numerical underflow in intermediate gradients when float16 tensors are used for mixed precision training. By multiplying the loss, each intermediate gradient will have the same multiplier applied. Instances of this class represent a loss scale. Calling instances of this class returns the loss scale as a scalar float32 tensor, while method `update()` updates the loss scale depending on the values of the gradients. Optimizers use instances of this class to scale loss and gradients. In most functions that accept a LossScale, you can also pass an int (such as 8) to create a `FixedLossScale` or the string `"dynamic"` to create a dynamic loss scale. ci|_y)z!Initializes the loss scale class.N)_weightsselfs b/home/dcms/DCMS/lib/python3.12/site-packages/tensorflow/python/training/experimental/loss_scale.py__init__zLossScale.__init__Ms DMcy)z>1166 8H  "i##%e""i  C }}d#/ ;BB3G HH!DMM#&&DH&E Orc 8tjrd}n tj}|j}i}t |j jdD]\\}}}||k(s|||<|jtt|.|fi||S)z6From Trackable. Gather graph-specific weights to save.Nc|ddSNrr)is rz/LossScale._trackable_children..sAaDGr)r>) rr4rr5r6sortedritemsr%superr_trackable_children) r save_typekwargsr<r=weightsr(gv __class__s rrHzLossScale._trackable_childrens  "i##%e""iGt}}224:KL q1 i  NN i29GGI Nrctt| ||}||Stjrd}n t j }|j}|jj||fdS)z3From Trackable. Find a weight in the current graph.N) rGr_lookup_dependencyrr4rr5r6rr7)rr(cached_dependencies unconditionalr<r=rNs rrPzLossScale._lookup_dependencysl)T= !#M   "i##%e""i ==  dI. 55rcy)z&Returns the config of this loss scale.Nrrs r get_configzLossScale.get_configr!rc|di|S)z&Creates the LossScale from its config.rr)clsconfigs r from_configzLossScale.from_configs ==rN)__name__ __module__ __qualname____doc__rabcabstractmethodr r%r?r.SaveType CHECKPOINTrHrPrT classmethodrX __classcell__rNs@rrr#sB  " " H#L%.$6$6$A$A" 6  rr) metaclass+mixed_precision.experimental.FixedLossScale!train.experimental.FixedLossScale)zmixed_precision.FixedLossScalerfrgcfeZdZdZej ddfdZdZdZdZ dZ xZ S) FixedLossScalezLoss scale with a fixed value. The loss scale is not updated for the lifetime of instances of this class. A given instance of this class always returns the same number when called. Nz{Use tf.keras.mixed_precision.LossScaleOptimizer instead. LossScaleOptimizer now has all the functionality of FixedLossScalectt| t|tt fs t d|dkr t dt ||_y)aCreates the fixed loss scale. Args: loss_scale_value: A Python float. Its ideal value varies depending on models to run. Choosing a too small loss_scale might affect model quality; a too big loss_scale might cause inf or nan. There is no single right loss_scale to apply. There is no harm choosing a relatively big number as long as no nan or inf is encountered in training. Raises: ValueError: If loss_scale_value is less than 1. z/loss_scale_value must be a Python int or float.z$loss_scale_value must be at least 1.N)rGrir isinstanceintfloat ValueError_loss_scale_value)rloss_scale_valuerNs rrzFixedLossScale.__init__sQ" .$(* &e 5 H II! = >>##34Drc@tj|jSrY)rconvert_to_tensorrprs rr zFixedLossScale.__call__s  !7!7 88rc0~tjdfS)NT)r no_opr#s rr%zFixedLossScale.updates  ! ! #T ))rc d|jzS)NzFixedLossScale(%s)rprs r__repr__zFixedLossScale.__repr__s $"8"8 88rcd|jiS)Nrqrwrs rrTzFixedLossScale.get_configs  6 6 77r) rZr[r\r]r deprecatedrr r%rxrTrcrds@rririsE ;  5 549*98rric d}|Dcgc]2}|.tjtj||4}}tj|Scc}w)zGReturns a scalar boolean tensor indicating if all gradients are finite.cRt|tjr |jS|SrY)rlr IndexedSlicesvalues)rLs r raw_valuesz"_is_all_finite..raw_valuess !!^%A%AB188IIr)r reduce_all is_finite)r$rrLis_finite_per_grads r_is_all_finiters`J  (,,Z];<   / 00 s7AcFtjr|S|jS)a(Returns the tensor's op in graph mode, or the tensor in eager mode. This is useful because sometimes an op is needed in graph mode instead of a tensor. In eager mode, there are no ops. Args: tensor: A tensor. Returns: The tensor's op in graph mode. The tensor in eager mode. )rr4op)tensors r_op_in_graph_moders  M rc~tjtjfdtjS)z5Assigns a value to a variable if the value is finite.c8tjSrY)rassign)valuevarsrrDz#_assign_if_finite..%s):3::e;L)Mr)r r rr ru)rrs``r_assign_if_finiter"s. !M r-mixed_precision.experimental.DynamicLossScale#train.experimental.DynamicLossScale)z mixed_precision.DynamicLossScalerrceZdZdZej dd d fd ZedZedZ edZ dZ d Z d Z d ZxZS) DynamicLossScaleaLoss scale that dynamically adjusts itself. Dynamic loss scaling works by adjusting the loss scale as training progresses. The goal is to keep the loss scale as high as possible without overflowing the gradients. As long as the gradients do not overflow, raising the loss scale never hurts. The algorithm starts by setting the loss scale to an initial value. Every N steps that the gradients are finite, the loss scale is increased by some factor. However, if a NaN or Inf gradient is found, the gradients for that step are not applied, and the loss scale is decreased by the factor. This process tends to keep the loss scale as high as possible without gradients overflowing. Nz}Use tf.keras.mixed_precision.LossScaleOptimizer instead. LossScaleOptimizer now has all the functionality of DynamicLossScalec:tt| t||_t ||_t||_|jdtj|j|_ |jdtjd|_ y)aCreates the dynamic loss scale. Args: initial_loss_scale: A Python float. The loss scale to use at the beginning. It's better to start this at a very high number, because a loss scale that is too high gets lowered far more quickly than a loss scale that is too low gets raised. The default is 2 ** 15, which is approximately half the maximum float16 value. increment_period: Increases loss scale every `increment_period` consecutive steps that finite gradients are encountered. If a nonfinite gradient is encountered, the count is reset back to zero. multiplier: The multiplier to use when increasing or decreasing the loss scale. current_loss_scale)r(r)r' good_stepsrN)rGrrrn_initial_loss_scalerm_increment_period _multiplierr?rfloat32_current_loss_scaleint64_num_good_steps)rinitial_loss_scaleincrement_period multiplierrNs rrzDynamicLossScale.__init__Bs, D*,$%78D !12DZ(D#// !nn.. 0 0D  ++ Q,@Drc|jSrY)rrs rrz#DynamicLossScale.initial_loss_scalefs  # ##rc|jSrY)rrs rrz!DynamicLossScale.increment_periodjs  ! !!rc|jSrY)rrs rrzDynamicLossScale.multiplierns   rc@tj|jSrY)rrsrrs rr zDynamicLossScale.__call__rs  !9!9 ::rctj|}tjrtj}d}|j j ||f}|jtjj|d}tj||j}n t|}fd}fd}tj|||} |} | | fS)zDUpdates loss scale based on if gradients are finite in current step.c`t|}tj|tjSrY)rr castrr)r$rs r get_is_finitez.DynamicLossScale.update..get_is_finite{s#"5) }}Y77r)argsN)axiscvfd}tjjdzjk\|fdS)z)Update assuming the gradients are finite.cjjz}tjt j|j j dSrB)rrr grouprrrnew_loss_scalers rincr_loss_scalezPDynamicLossScale.update..update_if_finite_grads..incr_loss_scalesP11D4D4DD%% d66 G  ' ' *, ,rrkcLtjjdS)Nrk)rr assign_addrsrrDzIDynamicLossScale.update..update_if_finite_grads..s!#4""--a0$2r)r rr)rrs rupdate_if_finite_gradsz7DynamicLossScale.update..update_if_finite_gradss<, YY    "d&<&< < 233rctjjjz d}t j j jdjj|S)z,Update assuming the gradients are nonfinite.rkr)r maximumrrr rrrrs rupdate_if_not_finite_gradsz;DynamicLossScale.update..update_if_not_finite_gradssf ''  " "T%5%5 5q:n  # #    % %a (  " " ) ). 9;;r)rflattenr has_strategyget_cross_replica_contextextendedcall_for_each_replicareducerReduceOpSUMr equalnum_replicas_in_syncrr ) rr$ distributionris_finite_floatreduced_is_finite_floatrrr update_opshould_apply_gradientss ` rr%zDynamicLossScale.updateus LL E""$#==?l8 %--CC uhD(o , 3 3K4H4H4L4L4C$!4!P..!8!-!B!BDi!'i 3; )%;46I& , ,,rc >tjr`d|jjd|jjd|j d|j d|jd Sd|j d|j d|jdS)Nz$DynamicLossScale(current_loss_scale=z, num_good_steps=z, initial_loss_scale=z, increment_period=z , multiplier=)z$DynamicLossScale(initial_loss_scale=)rr4rnumpyrrrrrs rrxzDynamicLossScale.__repr__s  "''--/1E1E1K1K1M&&(=(=tPQ#&&(=(=tPQrcJ|j|j|jdS)Nrrrrrs rrTzDynamicLossScale.get_configs%"55 11oo r)iig@)rZr[r\r]rrzrpropertyrrrr r%rxrTrcrds@rrr)s ;   #* $@  @@ $ $ " "  ;/-b Qrrct|ttfr t|S|dk(r t St|t r|S|yt d|z)zGet a loss scale object.dynamicNz-Could not interpret loss scale identifier: %s)rlrmrnrirrro) identifiers rr7r7s] S%L) * %%9   I&   D  !!r)#r]r^tensorflow.python.distributerrtensorflow.python.eagerrtensorflow.python.frameworkrrrtensorflow.python.opsr r r r r tensorflow.python.trackablerr.tensorflow.python.utilrr tensorflow.python.util.tf_exportrdeprecated_endpoints TrackableABCMetarrirrrrr7rrrrs)" 74+.6+&2*-+9.'6"!!"J"@B  [ ##s{{[ B[|"!!"O"EG  08Y08 G08f 1""!!3)+  DyD +DN !r