BVhdZddlZddlZddlZddlmZmZmZmZmZm Z m Z m Z m Z m Z mZddlmZddlmZddlmZddlmZddlmZdd lmZdd lmZdd lmZdd lmZdd lmZddlmZddlm Z ddlm!Z!ddlm"Z#ddl$m%Z%ddl&m'Z'ddl(m)Z)e dejTe#jVZ,ee,ee ee jZgee e,ffZ.ee e/e/fe/fZ0GddejbZ2e)dGdde2Z3e)dGdde2Z4e)dGd d!e2Z5e)d"Gd#d$e2Z6e)d%Gd&d'e2Z7e)d(Gd)d*e2Z8e)d+Gd,d-Z9e)d.Gd/d0Z:e)d1Gd2d3Z;e)d4Gd5d6Zd:ee?d8ee?fd;Z@dz>Companion classes for mid level API for TPU Embeddings in TF2.N) AnyCallableDictIterableListOptionalSequenceTextTupleTypeVarUnion)logging)optimization_parameters_pb2)tpu_embedding_configuration_pb2) device_util)sharded_variable) tpu_strategy) device_spec)dtypes)ops) TensorShape) array_ops) init_ops_v2)math_ops) variables)tpu_ops)core) tf_export TableVariablec(eZdZdZ ddeeegeffdedeedeedeeded ee d ee d efd Z e jd eefdZe jd eej$fdZdej*fdZe jd edej0ffdZe jd edej6ffdZdddeeej$gej<fd eeej<ffdZ de!d ee!effdZ"d e#fdZ$y) _Optimizerz6Base class for all optimizers, with common parameters.N learning_rateuse_gradient_accumulationclip_weight_minclip_weight_maxweight_decay_factor-multiply_weight_decay_factor_by_learning_rate clipvalueslot_variable_creation_fnlow_dimensional_packing_statusc &||_||_||_||_|s|t d|d|d}nt |t sd|z|f}|\|_|_||_ ||_ |t|st d|||_ | |_ y)NzWhen `use_gradient_accumulation` is False, gradient clipping cannot be used and `clipvalue` should be left as None. Received value z for argument `clipvalue`.)NNgzRArgument `slot_variable_creation_fn` must be either None or a callable. Received: )r"r#r$r% ValueError isinstancetupleclip_gradient_minclip_gradient_maxr&r'callabler)r*) selfr"r#r$r%r&r'r(r)r*s \/home/dcms/DCMS/lib/python3.12/site-packages/tensorflow/python/tpu/tpu_embedding_v2_utils.py__init__z_Optimizer.__init__5s'D%>D"*D*D $)>  %;&@ B CCi  5 )?I.i5>2DD22D5 6 "- . /  !!: ; = >>&?D"*HD'returnct)zReturns the name of all the slot variables. This does not include the 'parameters' variable and these names must match the names of the slots variables as used in the corresponding `tpu_ops.load_tpu_embedding_*` ops. NotImplementedErrorr2s r3 _slot_namesz_Optimizer._slot_names\s  r5ct)zfReturns initializers for slot variables. This returns a parallel list to self._slot_names(). r8r:s r3_slot_initializersz_Optimizer._slot_initializersfs r5 parametersc|jr tjj|_ntjj |_|j %|j |jj_ |j%|j|jj_ |j%|j|jj_ |j%|j|jj_ |jr$|j|_|j rd|_|j"|_y)z8Sets the optimizer fields in the OptimizationParameters.NT)r#rGradientAccumulationStatusENABLEDgradient_accumulation_statusDISABLEDr$clipping_limitslowervaluer%upperr/gradient_clipping_limitsr0r&r'r*r2r>s r3_set_optimization_parametersz'_Optimizer._set_optimization_parametersns %% % @ @ H H- & @ @ I I- '/3/C/Cj  &&, '/3/C/Cj  &&, )8<8N8Nj))//5 )8<8N8Nj))//5 '+'?'?j$ ; ;CG @ ++-r5.ct)z,Returns the load function for the optimizer.r8r:s r3_loadz_Optimizer._load  r5ct)z0Returns the retrieve function for the optimizer.r8r:s r3 _retrievez_Optimizer._retrieverMr5table TableConfigvariable_creatorc|j/|j||j|jSi}t|j|jD]\}}|||||<|S)aCreates slot variables for table. Args: table: The table variable to create slots for. variable_creator: A function which creates variables. Takes parameters 'name', 'initializer'. Returns: A dict of variables, keyed by self._slot_names(). )r)r;r=zip)r2rPrRslotsslot initializers r3 _create_slotsz_Optimizer._create_slotss %%1  + +E43C3C3E,0,C,C,EGGe"4#3#3#5#'#:#:#< >: $ &t[9d :lr5otherct||jr[tt|jj |jj Dcgc] \}}||k( c}}Sycc}}w)NF)r- __class__allrT__dict__items)r2rYattr1attr2s r3__eq__z_Optimizer.__eq__sf%( !$--"5"5"79M9M9OPeU 5.   sA3 cZtt|jjSN)hashr.r]r^r:s r3__hash__z_Optimizer.__hash__s dmm))+, --r5)NNF)%__name__ __module__ __qualname____doc__r floatrboolr ClipValueTypeSlotVarCreationFnTyper4abcabstractmethodrr r;r Initializerr=rOptimizationParametersrJr OperationrLrTensorrO tf_variablesVariablerrXrraintrer5r3r!r!2s>,0CG-2%I5(2u9"556%I"&%I  %I   %I $E? %I6:%I-(%I"**?!@%I'+%IN4:${'>'>"?3JJ>Xc3==01#t{{"23  $ (?(?!@!-!6!6"78 D,'' '( 2#%T "2..r5r!) metaclassz*tpu.experimental.embedding.CustomOptimizerceZdZdZ ddej deeegeffde e e de e e jde e eeegeffdef fd Zde efd Zde e jfd Zded ej(ffd Zded ej,ffd Zedeeeegeffd ffdZedej6fdZxZS)CustomOptimizeraP Optimization parameters for custom optimizer for TPU embeddings. This optimizer gives the user the ability to define a custom optimizer for running embedding lookups on TPU v5p. The custom computation should be a function which takes gradient, embedding table, a list of slot variables, learning_rate and a list of hyperparameters. The function should perform the gradient update on the embedding_table + slot_variables and return the updated embedding_table and slot_variables. e.g. ```python @tf.function def sgd_optimizer_computation( gradient, embedding_table, slot_variables, learning_rate, hyperparameters, ): del slot_variables, hyperparameters return embedding_table - gradient * learning_rate ``` Above is a simple example of a sgd optimizer. You can also define a more complex optimizer which updates multiple tables and slot variables. ```python @def_function.function def adagrad_optimizer_computation( gradient, embedding_table, slot_variables, learning_rate, hyperparameters, ): del hyperparameters accumulator = slot_variables[0] new_accumulator = accumulator + gradient * gradient updated_embedding_table = ( embedding_table - learning_rate * gradient / math_ops.sqrt(new_accumulator) ) return (updated_embedding_table, new_accumulator) ``` The custom computation is defined as a per-row update function and it will be auto scaled for the entire table (slot variables). NOTE: This optimizer can only be used with the `TPUEmbeddingV2` class. Pass this to `tf.tpu.experimental.embedding.TPUEmbeddingV2` via the `optimizer` argument to set the global optimizer and its parameters: ```python optimizer = tf.tpu.experimental.embedding.CustomOptimizer( custom_computation=adagrad_optimizer_computation, learning_rate=1.0, slot_names=['accumulators'], slot_initializers=[ tf.constant_initializer(0.1, support_partition=True) ], ) embedding = tf.tpu.experimental.embedding.TPUEmbeddingV2( ... optimizer=optimizer) ``` This can also be used in a `tf.tpu.experimental.embedding.TableConfig` as the optimizer parameter to set a table specific optimizer. This will override the optimizer and parameters for global embedding optimizer defined above: ```python table_one = tf.tpu.experimental.embedding.TableConfig( vocabulary_size=..., dim=..., optimizer=optimizer) table_two = tf.tpu.experimental.embedding.TableConfig( vocabulary_size=..., dim=...) feature_config = ( tf.tpu.experimental.embedding.FeatureConfig( table=table_one), tf.tpu.experimental.embedding.FeatureConfig( table=table_two)) embedding = tf.tpu.experimental.embedding.TPUEmbedding( feature_config=feature_config, batch_size=... optimizer=optimizer) ``` In this example, the optimizer of the first table will be the one specified in the table config. The second table will use the optimizer specified in the TPUEmbedding argument. custom_computationr" slot_namesslot_initializershyperparametersr6c >t||dddddddd t|r|nd|_t|r|nd|_t |j}t |j}||k7rt d|d|dt|r|nd|_||_y)NF)r#r$r%r&r'r(r)r*rwzThe number of slot_names (z.) must match the number of slot_initializers (z).) superr4r._slot_names_attr_slot_initializers_attrlenr,_hyperparameters_custom_computation) r2r{r"r|r}r~num_slot_namesnum_slot_initializersr[s r3r4zCustomOptimizer.__init__s G"' 6:"&', " *CD#(.B$D ../N < <=..  &~&67$%R )  "_/"MD1Dr5c,t|jSrc)listrr:s r3r;zCustomOptimizer._slot_namesCs %% &&r5c,t|jSrc)rrr:s r3r=z"CustomOptimizer._slot_initializersFs ,, --r5.ctd)NzSCustom optimizer does not support load op since it is only used for TPUEmbeddingV2.r8r:s r3rLzCustomOptimizer._loadIs   r5ctd)NzWCustom optimizer does not support retrieve op since it is only used for TPUEmbeddingV2.r8r:s r3rOzCustomOptimizer._retrieveOs   r5c|jSrc)rr:s r3r~zCustomOptimizer.hyperparametersUs   r5c|jSrc)rr:s r3r{z"CustomOptimizer.custom_computationYs  # ##r5){Gz?NNN)rfrgrhrirPolymorphicFunctionr rjrrrstrrrprr4r r;r=rrrrLrsrOpropertyr r~ConcreteFunctionr{ __classcell__r[s@r3rzrzs[^F:>(,CGKO "222"25(2u9"556"249% "2 "${'>'>"?@ "2  U5(2u92E+E%F GH "2 "2H'4:'.${'>'>"?.Xc3==01 #t{{"23  !uU5(2u92E+E%F%KL! ! $$"7"7$ $r5rzztpu.experimental.embedding.SGDc8eZdZdZ ddeeegeffdedeedeedeedeedee d effd Z d e e fd Z d e ejfd Zdej$ffd Zd edej*ffdZd edej0ffdZxZS)SGDa!Optimization parameters for stochastic gradient descent for TPU embeddings. Pass this to `tf.tpu.experimental.embedding.TPUEmbedding` via the `optimizer` argument to set the global optimizer and its parameters: ``` embedding = tf.tpu.experimental.embedding.TPUEmbedding( ... optimizer=tf.tpu.experimental.embedding.SGD(0.1)) ``` This can also be used in a `tf.tpu.experimental.embedding.TableConfig` as the optimizer parameter to set a table specific optimizer. This will override the optimizer and parameters for global embedding optimizer defined above: ``` table_one = tf.tpu.experimental.embedding.TableConfig( vocabulary_size=..., dim=..., optimizer=tf.tpu.experimental.embedding.SGD(0.2)) table_two = tf.tpu.experimental.embedding.TableConfig( vocabulary_size=..., dim=...) feature_config = ( tf.tpu.experimental.embedding.FeatureConfig( table=table_one), tf.tpu.experimental.embedding.FeatureConfig( table=table_two)) embedding = tf.tpu.experimental.embedding.TPUEmbedding( feature_config=feature_config, batch_size=... optimizer=tf.tpu.experimental.embedding.SGD(0.1)) ``` In the above example, the first feature will be looked up in a table that has a learning rate of 0.2 while the second feature will be looked up in a table that has a learning rate of 0.1. See 'tensorflow/core/protobuf/tpu/optimization_parameters.proto' for a complete description of these parameters and their impacts on the optimizer algorithm. r"r#r$r%r&r'r(r*c 4t ||||||||d| y)aOptimization parameters for stochastic gradient descent. Args: learning_rate: The learning rate. It should be a floating point value or a callable taking no arguments for a dynamic learning rate. use_gradient_accumulation: setting this to `False` makes embedding gradients calculation less accurate but faster. clip_weight_min: the minimum value to clip by; None means -infinity. clip_weight_max: the maximum value to clip by; None means +infinity. weight_decay_factor: amount of weight decay to apply; None means that the weights are not decayed. Weights are decayed by multiplying the weight by this factor each step. multiply_weight_decay_factor_by_learning_rate: if true, `weight_decay_factor` is multiplied by the current learning rate. clipvalue: Controls clipping of the gradient. Set to either a single positive scalar value to get clipping or a tiple of scalar values (min, max) to set a separate maximum or minimum. If one of the two entries is None, then there will be no clipping that direction. Note if this is set, you may see a decrease in performance as gradient accumulation will be enabled (it is normally off for SGD as it has no affect on accuracy). See 'tensorflow/core/protobuf/tpu/optimization_parameters.proto' for more information on gradient accumulation and its impact on tpu embeddings. low_dimensional_packing_status: Status of the low-dimensional embedding packing optimization controls whether to optimize the packing of 1-dimensional, 2-dimensional, and 4-dimensional embedding tables in memory. N)rr4) r2r"r#r$r%r&r'r(r*r[s r3r4z SGD.__init__s.N G!5 & r5r6cgSrcrwr:s r3r;zSGD._slot_names Ir5cgSrcrwr:s r3r=zSGD._slot_initializersrr5r>cXt|||jjyrc)rrJstochastic_gradient_descent SetInParentr2r>r[s r3rJz SGD._set_optimization_parameterss" G(4**668r5.c"tjSrc)r9load_tpu_embedding_stochastic_gradient_descent_parametersr:s r3rLz SGD._loads  L LLr5c"tjSrc)r=retrieve_tpu_embedding_stochastic_gradient_descent_parametersr:s r3rOz SGD._retrieves  P PPr5)rTNNNNNF)rfrgrhrir rjrrkrrlr4rr r;rrpr=rrqrJrrrrLrrsrOrrs@r3rr^s+^:>(,)-)--1FJ+/-215(2u9"5561"&1  1   1 $E? 16>d^1-(1'+1f4:${'>'>"?93JJ9 MXc3==01MQ#t{{"23Qr5rz"tpu.experimental.embedding.AdagradcJeZdZdZ ddeeegeffdededeedeedeedeed ee d ee d effd Z d e e fdZd e ej fdZdej&ffd Zd edej,ffdZd edej2ffdZxZS)Adagrada&Optimization parameters for Adagrad with TPU embeddings. Pass this to `tf.tpu.experimental.embedding.TPUEmbedding` via the `optimizer` argument to set the global optimizer and its parameters: ```python embedding = tf.tpu.experimental.embedding.TPUEmbedding( ... optimizer=tf.tpu.experimental.embedding.Adagrad(0.1)) ``` This can also be used in a `tf.tpu.experimental.embedding.TableConfig` as the optimizer parameter to set a table specific optimizer. This will override the optimizer and parameters for global embedding optimizer defined above: ```python table_one = tf.tpu.experimental.embedding.TableConfig( vocabulary_size=..., dim=..., optimizer=tf.tpu.experimental.embedding.Adagrad(0.2)) table_two = tf.tpu.experimental.embedding.TableConfig( vocabulary_size=..., dim=...) feature_config = ( tf.tpu.experimental.embedding.FeatureConfig( table=table_one), tf.tpu.experimental.embedding.FeatureConfig( table=table_two)) embedding = tf.tpu.experimental.embedding.TPUEmbedding( feature_config=feature_config, batch_size=... optimizer=tf.tpu.experimental.embedding.Adagrad(0.1)) ``` In the above example, the first feature will be looked up in a table that has a learning rate of 0.2 while the second feature will be looked up in a table that has a learning rate of 0.1. See 'tensorflow/core/protobuf/tpu/optimization_parameters.proto' for a complete description of these parameters and their impacts on the optimizer algorithm. r"initial_accumulator_valuer#r$r%r&r'r)r(r*c ht |||||||| || |dkrtd|||_y)aOptimization parameters for Adagrad. Args: learning_rate: The learning rate. It should be a floating point value or a callable taking no arguments for a dynamic learning rate. initial_accumulator_value: initial accumulator for Adagrad. use_gradient_accumulation: setting this to `False` makes embedding gradients calculation less accurate but faster. clip_weight_min: the minimum value to clip by; None means -infinity. clip_weight_max: the maximum value to clip by; None means +infinity. weight_decay_factor: amount of weight decay to apply; None means that the weights are not decayed. multiply_weight_decay_factor_by_learning_rate: if true, `weight_decay_factor` is multiplied by the current learning rate. slot_variable_creation_fn: If you wish do directly control the creation of the slot variables, set this to a callable taking three parameters: a table variable, a list of slot names to create for it, and a list of initializers. This function should return a dict with the slot names as keys and the created variables as values with types matching the table variable. When set to None (the default), uses the built-in variable creation. clipvalue: Controls clipping of the gradient. Set to either a single positive scalar value to get clipping or a tuple of scalar values (min, max) to set a separate maximum or minimum. If one of the two entries is None, then there will be no clipping that direction. low_dimensional_packing_status: Status of the low-dimensional embedding packing optimization controls whether to optimize the packing of 1-dimensional, 2-dimensional, and 4-dimensional embedding tables in memory. rIArgument `initial_accumulator_value` must be a positive float. Received: N)rr4r,r) r2r"rr#r$r%r&r'r)r(r*r[s r3r4zAdagrad.__init__s_V G!5!& !A%  01 3 44&?D"r5r6cdgS)N accumulatorsrwr:s r3r;zAdagrad._slot_names=s  r5cFtj|jdgSNT)support_partitionrConstantrr:s r3r=zAdagrad._slot_initializers@s%  * *d  r5r>cXt|||jjyrc)rrJadagradrrs r3rJz$Adagrad._set_optimization_parametersGs" G(4""$r5.c"tjSrc)r%load_tpu_embedding_adagrad_parametersr:s r3rLz Adagrad._loadLs  8 88r5c"tjSrc)r)retrieve_tpu_embedding_adagrad_parametersr:s r3rOzAdagrad._retrieveOs  < <d^:?"**?!@:?-(:?'+:?x4:${'>'>"?%3JJ% 9Xc3==019=#t{{"23=r5rz*tpu.experimental.embedding.AdagradMomentumcbeZdZdZ ddeeegeffdedededededed eed eed eed eed ee dee deffd Z de e fdZde ej fdZdej&ffd Zdedej,ffdZdedej2ffdZxZS)AdagradMomentumaIOptimization parameters for Adagrad + Momentum with TPU embeddings. Pass this to `tf.tpu.experimental.embedding.TPUEmbedding` via the `optimizer` argument to set the global optimizer and its parameters: ```python embedding = tf.tpu.experimental.embedding.TPUEmbedding( ... optimizer=tf.tpu.experimental.embedding.AdagradMomentum(0.1)) ``` This can also be used in a `tf.tpu.experimental.embedding.TableConfig` as the optimizer parameter to set a table specific optimizer. This will override the optimizer and parameters for global embedding optimizer defined above: ```python table_one = tf.tpu.experimental.embedding.TableConfig( vocabulary_size=..., dim=..., optimizer=tf.tpu.experimental.embedding.AdagradMomentum(0.2)) table_two = tf.tpu.experimental.embedding.TableConfig( vocabulary_size=..., dim=...) feature_config = ( tf.tpu.experimental.embedding.FeatureConfig( table=table_one), tf.tpu.experimental.embedding.FeatureConfig( table=table_two)) embedding = tf.tpu.experimental.embedding.TPUEmbedding( feature_config=feature_config, batch_size=... optimizer=tf.tpu.experimental.embedding.AdagradMomentum(0.1)) ``` In the above example, the first feature will be looked up in a table that has a learning rate of 0.2 while the second feature will be looked up in a table that has a learning rate of 0.1. See 'tensorflow/core/protobuf/tpu/optimization_parameters.proto' for a complete description of these parameters and their impacts on the optimizer algorithm. r"momentum use_nesterovexponentbeta2epsilonr#r$r%r&r'r)r(r*c t||||| | | | | | |dkr td|dkr td||_||_||_||_||_y)aLOptimization parameters for Adagrad + Momentum. Args: learning_rate: The learning rate. It should be a floating point value or a callable taking no arguments for a dynamic learning rate. momentum: Moving average parameter for the momentum accumulator. use_nesterov: Whether to use the Nesterov variant of momentum. See Sutskever et al., 2013. exponent: Exponent for the Adagrad accumulator. beta2: Moving average parameter for the Adagrad accumulator. epsilon: initial accumulator for Adagrad accumulator. use_gradient_accumulation: setting this to `False` makes embedding gradients calculation less accurate but faster. clip_weight_min: the minimum value to clip by; None means -infinity. clip_weight_max: the maximum value to clip by; None means +infinity. weight_decay_factor: amount of weight decay to apply; None means that the weights are not decayed. multiply_weight_decay_factor_by_learning_rate: if true, `weight_decay_factor` is multiplied by the current learning rate. slot_variable_creation_fn: If you wish do directly control the creation of the slot variables, set this to a callable taking three parameters: a table variable, a list of slot names to create for it, and a list of initializers. This function should return a dict with the slot names as keys and the created variables as values with types matching the table variable. When set to None (the default), uses the built-in variable creation. clipvalue: Controls clipping of the gradient. Set to either a single positive scalar value to get clipping or a tuple of scalar values (min, max) to set a separate maximum or minimum. If one of the two entries is None, then there will be no clipping that direction. low_dimensional_packing_status: Status of the low-dimensional embedding packing optimization controls whether to optimize the packing of 1-dimensional, 2-dimensional, and 4-dimensional embedding tables in memory. rz*Adagrad momentum: epsilon must be positivez/Adagrad momentum: Precondition exponent must >0N)rr4r,rrrrr)r2r"rrrrrr#r$r%r&r'r)r(r*r[s r3r4zAdagradMomentum.__init__szh G!5!& !| C DD1} H IIDM$DDMDJDLr5r6c ddgS)Nrmomentarwr:s r3r;zAdagradMomentum._slot_names I &&r5cZtjdtjdgSrrrr:s r3r=z"AdagradMomentum._slot_initializers(t4t4 r5r>cft|||jj|j|j_|j |j_|j |j_|j|j_|j|j_yrc) rrJadagrad_momentumrrrrrrrs r3rJz,AdagradMomentum._set_optimization_parameterss G(4++-+/==J(/3/@/@J,+/==J((, J%*.,,J'r5.c"tjSrc)r.load_tpu_embedding_adagrad_momentum_parametersr:s r3rLzAdagradMomentum._loads  A AAr5c"tjSrc)r2retrieve_tpu_embedding_adagrad_momentum_parametersr:s r3rOzAdagradMomentum._retrieves  E EEr5)rFg|=TNNNNNNFrrs@r3rrSst+^:? (,)-)--1FJCG+/-2G5(2u9"556GG G  G  GG"&G G G$E?G6>d^G"**?!@G-(G'+GR'4:'${'>'>"? 73JJ 7BXc3==01BF#t{{"23Fr5rztpu.experimental.embedding.FTRLc!neZdZdZ ddeeegeffdedededededed eed eed eed eed ee dee dededef fd Z de e fdZde ej fdZdej&ffd Zdedej,ffdZdedej2ffdZxZS)FTRLatOptimization parameters for FTRL with TPU embeddings. See Algorithm 1 of this [paper](https://research.google.com/pubs/archive/41159.pdf). Pass this to `tf.tpu.experimental.embedding.TPUEmbedding` via the `optimizer` argument to set the global optimizer and its parameters: ```python embedding = tf.tpu.experimental.embedding.TPUEmbedding( ... optimizer=tf.tpu.experimental.embedding.FTRL(0.1)) ``` This can also be used in a `tf.tpu.experimental.embedding.TableConfig` as the optimizer parameter to set a table specific optimizer. This will override the optimizer and parameters for global embedding optimizer defined above: ```python table_one = tf.tpu.experimental.embedding.TableConfig( vocabulary_size=..., dim=..., optimizer=tf.tpu.experimental.embedding.FTRL(0.2)) table_two = tf.tpu.experimental.embedding.TableConfig( vocabulary_size=..., dim=...) feature_config = ( tf.tpu.experimental.embedding.FeatureConfig( table=table_one), tf.tpu.experimental.embedding.FeatureConfig( table=table_two)) embedding = tf.tpu.experimental.embedding.TPUEmbedding( feature_config=feature_config, batch_size=... optimizer=tf.tpu.experimental.embedding.FTRL(0.1)) ``` In the above example, the first feature will be looked up in a table that has a learning rate of 0.2 while the second feature will be looked up in a table that has a learning rate of 0.1. See 'tensorflow/core/protobuf/tpu/optimization_parameters.proto' for a complete description of these parameters and their impacts on the optimizer algorithm. r"learning_rate_powerl1_regularization_strengthl2_regularization_strengthbetarr#r$r%r&r'r)r( multiply_linear_by_learning_rateallow_zero_accumulatorr*c t||||| | | | | | |dkrtd|||_||_||_||_||_||_||_ y)a Optimization parameters for Adagrad. Args: learning_rate: The learning rate. It should be a floating point value or a callable taking no arguments for a dynamic learning rate. learning_rate_power: A float value, must be less or equal to zero. Controls how the learning rate decreases during training. Use zero for a fixed learning rate. l1_regularization_strength: A float value, must be greater than or equal to zero. l2_regularization_strength: A float value, must be greater than or equal to zero. beta: A float value, representing the beta value from the paper. initial_accumulator_value: The starting value for accumulators. Only zero or positive values are allowed. use_gradient_accumulation: setting this to `False` makes embedding gradients calculation less accurate but faster. clip_weight_min: the minimum value to clip by; None means -infinity. clip_weight_max: the maximum value to clip by; None means +infinity. weight_decay_factor: amount of weight decay to apply; None means that the weights are not decayed. multiply_weight_decay_factor_by_learning_rate: if true, `weight_decay_factor` is multiplied by the current learning rate. slot_variable_creation_fn: If you wish do directly control the creation of the slot variables, set this to a callable taking three parameters: a table variable, a list of slot names to create for it, and a list of initializers. This function should return a dict with the slot names as keys and the created variables as values with types matching the table variable. When set to None (the default), uses the built-in variable creation. clipvalue: Controls clipping of the gradient. Set to either a single positive scalar value to get clipping or a tuple of scalar values (min, max) to set a separate maximum or minimum. If one of the two entries is None, then there will be no clipping that direction. multiply_linear_by_learning_rate: If set to True, a modified formula is used for FTRL that treats the "linear" accumulator as being pre-multiplied by the learning rate (i.e., the accumulator named "linear" actually stores "linear * learning_rate"). Other than checkpoint compatibility, this is mathematically equivalent for a static learning rate; for a dynamic learning rate, it is nearly the same as long as the learning rate does not change quickly. The benefit of this is that the modified formula handles zero and near-zero learning rates without producing NaNs, improving flexibility for learning rate ramp-up. allow_zero_accumulator: If set to True, changes some internal formulas to allow zero and near-zero accumulator values at the cost of some performance; this only needs to be set if you are using an initial accumulator value of zero, which is uncommon. low_dimensional_packing_status: Status of the low-dimensional embedding packing optimization controls whether to optimize the packing of 1-dimensional, 2-dimensional, and 4-dimensional embedding tables in memory. rrN) rr4r,rrrrrrr)r2r"rrrrrr#r$r%r&r'r)r(rrr*r[s r3r4z FTRL.__init__sN G!5!& !A%  01 3 44&?D"2D&@D#&@D#DI,LD)"8Dr5r6c ddgS)Nrlinearsrwr:s r3r;zFTRL._slot_namesvrr5cptj|jdtjdgSrrr:s r3r=zFTRL._slot_initializersys5  * *d  t4  r5r>ct|||j}|j|_|j |_|j|_|j|_ |j|_ |j|_ yrc) rrJftrlrl1rl2rlr_powerrrmultiply_linear_by_lrr)r2r>rr[s r3rJz!FTRL._set_optimization_parameterssi G(4 ??D--DG--DG,,DM DI!%!F!FD"&"="=Dr5.c"tjSrc)r"load_tpu_embedding_ftrl_parametersr:s r3rLz FTRL._load  5 55r5c"tjSrc)r&retrieve_tpu_embedding_ftrl_parametersr:s r3rOzFTRL._retrieve  9 99r5)rgrrrrTNNNNNNFFFrrs@r3rrs.d:?#'*-*-),(,)-)--1FJCG+//4%*-2#\95(2u9"556\9!\9#( \9 #( \9  \9"'\9"&\9 \9 \9$E?\96>d^\9"**?!@\9-(\9)-\9 #!\9"'+#\9|'4:'${'>'>"? >3JJ >6Xc3==016:#t{{"23:r5rztpu.experimental.embedding.AdamcbeZdZdZ ddeeegeffdedededededed eed eed eed eed ee dee deffd Z de e fdZde ej fdZdej&ffd Zdedej,ffdZdedej2ffdZxZS)AdamaOptimization parameters for Adam with TPU embeddings. Pass this to `tf.tpu.experimental.embedding.TPUEmbedding` via the `optimizer` argument to set the global optimizer and its parameters: NOTE: By default this optimizer is lazy, i.e. it will not apply the gradient update of zero to rows that were not looked up. You can change this behavior by setting `lazy_adam` to `False`. ```python embedding = tf.tpu.experimental.embedding.TPUEmbedding( ... optimizer=tf.tpu.experimental.embedding.Adam(0.1)) ``` This can also be used in a `tf.tpu.experimental.embedding.TableConfig` as the optimizer parameter to set a table specific optimizer. This will override the optimizer and parameters for global embedding optimizer defined above: ```python table_one = tf.tpu.experimental.embedding.TableConfig( vocabulary_size=..., dim=..., optimizer=tf.tpu.experimental.embedding.Adam(0.2)) table_two = tf.tpu.experimental.embedding.TableConfig( vocabulary_size=..., dim=...) feature_config = ( tf.tpu.experimental.embedding.FeatureConfig( table=table_one), tf.tpu.experimental.embedding.FeatureConfig( table=table_two)) embedding = tf.tpu.experimental.embedding.TPUEmbedding( feature_config=feature_config, batch_size=... optimizer=tf.tpu.experimental.embedding.Adam(0.1)) ``` In the above example, the first feature will be looked up in a table that has a learning rate of 0.2 while the second feature will be looked up in a table that has a learning rate of 0.1. See 'tensorflow/core/protobuf/tpu/optimization_parameters.proto' for a complete description of these parameters and their impacts on the optimizer algorithm. r"beta_1beta_2r lazy_adamsum_inside_sqrtr#r$r%r&r'r)r(r*c Btt| |||| | | | | | |dks|dk\rtd|d|dks|dk\rtd|d|dkrtdj ||s |s td||_||_||_||_||_ y) a Optimization parameters for Adam. See 'tensorflow/core/protobuf/tpu/optimization_parameters.proto' for a complete description of these parameters and their impacts on the optimizer algorithm. Args: learning_rate: The learning rate. It should be a floating point value or a callable taking no arguments for a dynamic learning rate. beta_1: A float value. The exponential decay rate for the 1st moment estimates. beta_2: A float value. The exponential decay rate for the 2nd moment estimates. epsilon: A small constant for numerical stability. lazy_adam: Use lazy Adam instead of Adam. Lazy Adam trains faster. sum_inside_sqrt: When this is true, the Adam update formula is changed from `m / (sqrt(v) + epsilon)` to `m / sqrt(v + epsilon**2)`. This option improves the performance of TPU training and is not expected to harm model quality. use_gradient_accumulation: Setting this to `False` makes embedding gradients calculation less accurate but faster. clip_weight_min: the minimum value to clip by; None means -infinity. clip_weight_max: the maximum value to clip by; None means +infinity. weight_decay_factor: amount of weight decay to apply; None means that the weights are not decayed. multiply_weight_decay_factor_by_learning_rate: if true, `weight_decay_factor` is multiplied by the current learning rate. slot_variable_creation_fn: If you wish do directly control the creation of the slot variables, set this to a callable taking three parameters: a table variable, a list of slot names to create for it, and a list of initializers. This function should return a dict with the slot names as keys and the created variables as values with types matching the table variable. When set to None (the default), uses the built-in variable creation. clipvalue: Controls clipping of the gradient. Set to either a single positive scalar value to get clipping or a tiple of scalar values (min, max) to set a separate maximum or minimum. If one of the two entries is None, then there will be no clipping that direction. low_dimensional_packing_status: Status of the low-dimensional embedding packing optimization controls whether to optimize the packing of 1-dimensional, 2-dimensional, and 4-dimensional embedding tables in memory. rg?z2Argument `beta_1` must be >= 0 and < 1. Received: .z2Argument `beta_2` must be >= 0 and < 1. Received: z!epsilon must be positive; got {}.z{When disabling lazy Adam (`lazy_adam=False`), gradient accumulation must be used. Set `use_gradient_accumulation` to False.N) rrr4r,formatrrrrr)r2r"rrrrrr#r$r%r&r'r)r(r*r[s r3r4z Adam.__init__sx $!5!& {fl  >vha H JJ {fl  >vha H JJ"} :AA'J KK $Y  6 77 DKDKDLDN*Dr5r6c ddgS)Nr velocitiesrwr:s r3r;zAdam._slot_names"s | $$r5cZtjdtjdgSrrr:s r3r=zAdam._slot_initializers%rr5r>c<tt| ||j|j_|j |j_|j|j_|j |j_ |j|j_ yrc) rrrJradambeta1rrrruse_non_lazy_adamruse_sum_inside_sqrtrs r3rJz!Adam._set_optimization_parameters+si $2:> KKJOO KKJOO"llJOO,0NN(:JOO%*.*>*>JOO'r5.c"tjSrc)r"load_tpu_embedding_adam_parametersr:s r3rLz Adam._load5rr5c"tjSrc)r&retrieve_tpu_embedding_adam_parametersr:s r3rOzAdam._retrieve8rr5)rg?g+?gHz>TTTNNNNNNFrrs@r3rrsp/f:?"(,)-)--1FJCG+/-2Y+5(2u9"556Y+Y+ Y+  Y+  Y+Y+"&Y+ Y+ Y+$E?Y+6>d^Y+"**?!@Y+-(Y+'+Y+v%4:%${'>'>"? ?3JJ?6Xc3==016:#t{{"23:r5rz-tpu.experimental.embedding.QuantizationConfigcJeZdZdZdededefdZdejfdZ dZ y ) QuantizationConfigacSettings for simulated quantization of the tpu embedding table. When simulated quantization is enabled, the results of the embedding lookup are clipped and quantized according to the settings here before the combiner is applied. For example, to quantize `input` the following is done: ```python if input < lower input = lower if input > upper input = upper quantum = (upper - lower) / (num_buckets - 1) input = math.floor((input - lower) / quantum + 0.5) * quantum + lower ``` See tensorflow/core/protobuf/tpu/optimization_parameters.proto for more details. NOTE: This does not change the storage type of the embedding table, that will continue to be float32 as will the saved variable in the checkpoint. You will have to manually quantize the variable (typically with the same algorithm and settings as above) manually. num_bucketsrErGcV|dkrtd|d||_||_||_y)a[Simulated quantizaiton configuration. Args: num_buckets: The number of quantization buckets, must be atleast 2. lower: The lower bound for the quantization range. upper: The upper bound for the quantization range. Returns: `QuantizationConfig`. Raises: ValueError: if `num_buckets` is less than 2. rznum_buckets is z0, must be at least 2 for simulated quantization.N)r,rrErG)r2rrErGs r3r4zQuantizationConfig.__init__Ws@Q  612 33#DDJDJr5r>cd|j_|j|j_|j|jjj_|j |jjj _y)NT)simulated_quantizationenabledrrErDrFrGrIs r3rJz/QuantizationConfig._set_optimization_parametersms`04J%%-484D4DJ%%1DHJJJ%%55;;ADHJJJ%%55;;Ar5cfdj|j|j|jS)NzQQuantizationConfig(num_buckets={num_buckets!r}, lower={lower!r}, upper={upper!r}))rrErG)rrrErGr:s r3__repr__zQuantizationConfig.__repr__ts2 %v ,,jjjj & "#r5N) rfrgrhrirvrjr4rrqrJrrwr5r3rr<s;2#eE,O3JJO#r5rz&tpu.experimental.embedding.TableConfigceZdZdZ ddededeeegdfdeede dee d e d eefd Z d Z d e jjdedeegefeffdZy)rQa9Configuration data for one embedding table. This class holds the configuration data for a single embedding table. It is used as the `table` parameter of a `tf.tpu.experimental.embedding.FeatureConfig`. Multiple `tf.tpu.experimental.embedding.FeatureConfig` objects can use the same `tf.tpu.experimental.embedding.TableConfig` object. In this case a shared table will be created for those feature lookups. ```python table_config_one = tf.tpu.experimental.embedding.TableConfig( vocabulary_size=..., dim=...) table_config_two = tf.tpu.experimental.embedding.TableConfig( vocabulary_size=..., dim=...) feature_config = { 'feature_one': tf.tpu.experimental.embedding.FeatureConfig( table=table_config_one), 'feature_two': tf.tpu.experimental.embedding.FeatureConfig( table=table_config_one), 'feature_three': tf.tpu.experimental.embedding.FeatureConfig( table=table_config_two)} embedding = tf.tpu.experimental.embedding.TPUEmbedding( feature_config=feature_config, batch_size=... optimizer=tf.tpu.experimental.embedding.Adam(0.1)) ``` The above configuration has 2 tables, and three features. The first two features will be looked up in the first table and the third feature will be looked up in the second table. Nvocabulary_sizedimrW optimizercombinernamequantization_configlayoutc t|tr|dkrtd|t|tr|dkrtd||t|std||-t j ddt j|z }d} || vrtd | d ||tjd ||_ ||_ ||_ ||_ ||_||_||_||_y) aEmbedding table configuration. Args: vocabulary_size: Size of the table's vocabulary (number of rows). dim: The embedding dimension (width) of the table. initializer: A callable initializer taking one parameter, the shape of the variable that will be initialized. Will be called once per task, to initialize that task's shard of the embedding table. If not specified, defaults to `truncated_normal_initializer` with mean `0.0` and standard deviation `1/sqrt(dim)`. optimizer: An optional instance of an optimizer parameters class, instance of one of `tf.tpu.experimental.embedding.SGD`, `tf.tpu.experimental.embedding.Adagrad` or `tf.tpu.experimental.embedding.Adam`. If set will override the global optimizer passed to `tf.tpu.experimental.embedding.TPUEmbedding`. combiner: A string specifying how to reduce if there are multiple entries in a single row. Currently 'mean', 'sqrtn', 'sum' are supported, with 'mean' the default. 'sqrtn' often achieves good accuracy, in particular with bag-of-words columns. For more information, see `tf.nn.embedding_lookup_sparse`. name: An optional string used to name the table. Must be defined if running on SparseCore. quantization_config: The simulated quantization config. An instance of `tf.tpu.experimental.embedding.QuantizationConfig`. See the class for more documentation. layout: If the table already has its layout computed, you can pass it in here. Otherwise, we will compute it for you. Most users should leave this as None. Returns: `TableConfig`. Raises: ValueError: if `vocabulary_size` is not a positive integer. ValueError: if `dim` is not a positive integer. ValueError: if `initializer` is specified and is not callable. ValueError: if `combiner` is not supported. rzFArgument `vocabulary_size` must be an int and must be >= 1. Received: zPArgument `dim` (embedding dimension) must be an int and must be >= 1. Received: Nz?Argument `initializer` must be a callable (or None). Received: r)meanstddev)rsumsqrtnz#Argument `combiner` must be one of z . Received: zuName of the table config must be specified for running on SparseCore. Different table configs must have unique names.)r-rvr,r1rTruncatedNormalmathsqrtrwarningrrrWr r r r r ) r2rrrWr r r r r accepted_combinerss r3r4zTableConfig.__init__sBf os +/B  &' ) ** c3 37  88;u > ?? (;*?  "m % &&//S7837GIk1))  /0B/CDj " ## | oo I +DDH"DDNDMDI2DDKr5c |j}t|tjrut j tj|}|j dk(rBtj|jdtj|jz rd}dj|j|j||j|j|j |j"S)NrrzTableConfig(vocabulary_size={vocabulary_size!r}, dim={dim!r}, initializer={initializer!r}, optimizer={optimizer!r}, combiner={combiner!r}, name={name!r}, quantization_config={quantization!r}))rrrWr r r  quantization)rWr-rrtypingcastrriscloserrrrrr r r r )r2rWs r3rzTableConfig.__repr__s""K+{::;KK ; ;[Ik   c !ll;--q4881D/DE  45;F $ 4 4HH'..YY!555;5 r5table_descriptor num_hostslearning_rate_indexc|j|_t|j||_|j|_|j }|j rt|j jr3||j j|jj_ n%|j j|j_ |j jr)tjjj |_ |j j#||j$r|j$j#|yy)z-Set the table descriptor from the table data.N)r maxrr dimensionoptimization_parametersr r1r"dynamictagconstantr*rLowDimensionalPackingStatusStatusrArJr )r2rrrr>s r3_set_table_descriptorz!TableConfig._set_table_descriptors!II(+4+?+?'K$!%!99J  ~~ $.... /  < < =   ((,-1NN,H,H   )  6 6 ' C C J J R R 1 nn11*=  ;;JG r5)NNrNNN)rfrgrhrirvrrrr!r rr4rrTPUEmbeddingConfigurationTableDescriptorrr)rwr5r3rQrQ|s!N6:(,!04#WW WHcUD[12 W *% W  W TNW.WsmWr0!H7  !H !H  S 13 67 !Hr5rQz+tpu.experimental.embedding.RowIdInitializercjeZdZdZd defdZdeeeefde jde jfdZ y) RowIdInitializerz>An initializer that initializes the table with vocabulary ids.offsetc||_yrc)r.)r2r.s r3r4zRowIdInitializer.__init__:s DKr5shapedtyper6ctj|j|j|dzd|dddftj||zS)Nrr)startlimitdeltar1)r1)rranger.rones)r2r0r1s r3__call__zRowIdInitializer.__call__=sN >>kkuQx!7q g U3 44r5N)r)rfrgrhrirvr4r r rrDTyperrsr8rwr5r3r-r-6sDFS4# 344=C\\4 {{4r5r-z(tpu.experimental.embedding.FeatureConfigc XeZdZdZ d dedededeee ee fdee f dZ d Z y) FeatureConfigaMConfiguration data for one embedding feature. This class holds the configuration data for a single embedding feature. The main use is to assign features to `tf.tpu.experimental.embedding.TableConfig`s via the table parameter: ```python table_config_one = tf.tpu.experimental.embedding.TableConfig( vocabulary_size=..., dim=...) table_config_two = tf.tpu.experimental.embedding.TableConfig( vocabulary_size=..., dim=...) feature_config = { 'feature_one': tf.tpu.experimental.embedding.FeatureConfig( table=table_config_one), 'feature_two': tf.tpu.experimental.embedding.FeatureConfig( table=table_config_one), 'feature_three': tf.tpu.experimental.embedding.FeatureConfig( table=table_config_two)} embedding = tf.tpu.experimental.embedding.TPUEmbedding( feature_config=feature_config, batch_size=... optimizer=tf.tpu.experimental.embedding.Adam(0.1)) ``` The above configuration has 2 tables, and three features. The first two features will be looked up in the first table and the third feature will be looked up in the second table. You can also specify the output shape for each feature. The output shape should be the expected activation shape excluding the table dimension. For dense and sparse tensor, the output shape should be the same as the input shape excluding the last dimension. For ragged tensor, the output shape can mismatch the input shape. NOTE: The `max_sequence_length` will be only used when the input tensor has rank 2 and the `output_shape` is not set in the feature config. When feeding features into `embedding.enqueue` they can be `tf.Tensor`s, `tf.SparseTensor`s or `tf.RaggedTensor`s. When the argument `max_sequence_length` is 0, the default, you should expect a output of `embedding.dequeue` for this feature of shape `(batch_size, dim)`. If `max_sequence_length` is greater than 0, the feature is embedded as a sequence and padded up to the given length. The shape of the output for this feature will be `(batch_size, max_sequence_length, dim)`. NrPmax_sequence_lengthvalidate_weights_and_indices output_shaper c.t|tstdt|dt|tr|dkrtd|||_||_||_t||_ t|tstd|||_ y)aUFeature configuration. Args: table: An instance of `tf.tpu.experimental.embedding.TableConfig`, describing the table in which this feature should be looked up. max_sequence_length: If positive, the feature is a sequence feature with the corresponding maximum sequence length. If the sequence is longer than this, it will be truncated. If 0, the feature is not a sequence feature. validate_weights_and_indices: If true, uses safe_embedding_lookup during serving which ensures there are no empty rows and all weights and ids are positive at the expense of extra compute cost. output_shape: Optional argument to config the output shape of the feature activation. If provided, the feature feeding to the `embedding.enqueue` has to match the shape (for ragged tensor, the input shape and output shape can mismatch). If not provided, the shape can be either provided to the `embedding.build` or auto detected at the runtime. name: An optional string used to name the table. Must be defined if running on SparseCore. Returns: `FeatureConfig`. Raises: ValueError: if `table` is not an instance of `tf.tpu.experimental.embedding.TableConfig`. ValueError: if `max_sequence_length` not an integer or is negative. z"Argument `table` has invalid type z7. Expected `tf.tpu.experimental.embedding.TableConfig`.rzJArgument `max_sequence_length` must be an int and must be >= 0. Received: zEArgument `validate_weights_and_indices` must be a boolean. Received: N) r-rQr,typervrPr<r rr>rkr=)r2rPr<r=r>r s r3r4zFeatureConfig.__init__wsD e[ ) ;DK=IOO PP )3 /3F3J  *+ - ..DJ2DDI#L1D $d ,  34 6 77)ED%r5cdj|j|j|j|j|j S)NzFeatureConfig(table={table!r}, max_sequence_length={max_sequence_length!r}, validate_weights_and_indices={validate_weights_and_indices!r}, output_shape={output_shape!r}, name={name!r}))rPr<r=r>r )rrPr<r=r>r r:s r3rzFeatureConfig.__repr__sH <=CFjj$($<$<-1-N-N!..YY =C= !r5)rTNN)rfrgrhrirQrvrkrr rrr r4rrwr5r3r;r;Esl.d+,48GK&* 6E!6E$'6E.26E&eDI{,B&CD 6E d^ 6Ep !r5r;configr6ctjdt|jD]}tj|tjdy)aLogs a TPUEmbeddingConfiguration proto across multiple statements. Args: config: TPUEmbeddingConfiguration proto to log. Necessary because logging.info has a maximum length to each log statement, which particularly large configs can exceed. z+Beginning log of TPUEmbeddingConfiguration.z+Done with log of TPUEmbeddingConfiguration.N)rinfor splitlines)rBlines r3log_tpu_embedding_configurationrGsF ,,<=&k$$&d LL ,,<=r5device_stringscrtd|Dd}|Dcgc]}|jc}Scc}w)Nc3ZK|]#}tjj|%ywrc)r DeviceSpecV2 from_string).0specs r3 z,_sort_device_spec_strings..s!Md{++D1Ms)+cH|j|j|jfSrc)replicatask device_index)ss r3z+_sort_device_spec_strings..sQYY7r5)key)sorted to_string)rH sorted_specsrNs r3_sort_device_spec_stringsrZs4MnM 7,(4 4t$..  44 4s4strategycg}t|jjD]-}tj|}||vs|j |/t ||jjk(sJ|S)zReturns a sorted list of CPU devices for the remote jobs. Args: strategy: A TPUStrategy object. Returns: A sorted list of device host strings. )rZextendedworker_devicesrget_host_for_deviceappendrr)r[ list_of_hosts tpu_devicehosts r3get_list_of_hostsrdsv--h.?.?.N.NO!j  * *: 6D = 4 ! ] x00:: :: : r5)Crirnrrrrrrrrr r r r r abslrtensorflow.core.protobuf.tpurrtensorflow.python.distributerrrtensorflow.python.frameworkrrr(tensorflow.python.framework.tensor_shapertensorflow.python.opsrrrrrttensorflow.python.tpu.opsrtensorflow.python.typesr tensorflow.python.util.tf_exportrShardedVariablerurrprmrjrlABCMetar!rzrrrrrrrQr-r;r*rGrrZ TPUStrategyrdrwr5r3rqsE ggggDH4953.+@+-*;-(6)9)I)I$--/  DJ[%<%< =>} eE5L)501 H.3;;H.V 78]$j]$9]$@ +,pQ*pQ-pQf /0}=j}=1}=@ 78OFjOF9OFd ,-j::j:.j:Z ,-d::d:.d:N :;<#<#<<#~ 34vHvH5vHr 89 4 4: 4 56r!r!7r!j > + E E >JN >5hsm5S 5 8 8T$Zr5