BVh.1 dZddlZddlmZmZmZmZmZmZm Z m Z m Z ddl m Z ddlmZddlmZddlmZddlmZdd lmZdd lmZdd lmZdd lmZdd lmZddlmZddlmZddlmZddlm Z ddlm!Z!ddlm"Z#ddl$m%Z%ddl&m'Z'ddl&m(Z(ddl&m)Z)ddl&m*Z*ddl&m+Z,ddl-m.Z.ddl/m0Z0ddl/m1Z1ddl2m3Z3ddl2m4Z4ddl2m5Z5dd l6m7Z7dd!l8m9Z9dd"l8m:Z:dd#l;mm?Z?dd%l>m@Z@dd&l>mAZAdd'lBmCZCd(ZDd)ZEGd*d+ejZGd,ZHeCd-Gd.d/e9jZJejd0e d1ee eLe fd2ee ee e,jffd3e4jfd4ZOejd0e d1ee eLe fd2ee ee e,jffd3e4jfd5ZPd6ZQd7ZRe0jd8d9eQeRd:;dZUd1ee d>!  + ++N)__name__ __module__ __qualname____doc__propertyr6r:r9r3r3?s( , ,r:r3c~|jttjt j |y)N)s) _set_attr _NAME_KEYr AttrValuer,as_bytes)opnames r9 _add_key_attrrIGs$,,y.22V__T5JKLr:z'tpu.experimental.embedding.TPUEmbeddingc eZdZdZ d6deej efdeejde fdZ d7dZ d8deee eeffd Z d7d ee ed eefd Zd e ede efdZedeej*ej.ffdZdej4fdZd8deefdZd8deefdZdeeeeej.fffdZdZ dZ!dZ"dZ#dZ$de e%jLde ee%jLde ej dedede'jPf d Z)d!Z*d"Z+d#Z, d9d$e deed%eefd&Z-d'e d(e de efd)Z.defd*Z/defd+Z0defd,Z1d-e efd.Z2d-e efd/Z3d0Z4d1ed2ede fd3Z5d4Z6d5Z7y): TPUEmbeddinga_The TPUEmbedding mid level API. NOTE: When instantiated under a TPUStrategy, this class can only be created once per call to `tf.tpu.experimental.initialize_tpu_system`. If you wish to re-initialize the embedding engine you must re-initialize the tpu as well. Doing this will clear any variables from TPU, so ensure you have checkpointed before you do this. If a further instances of the class are needed, set the `initialize_tpu_embedding` argument to `False`. This class can be used to support training large embeddings on TPU. When creating an instance of this class, you must specify the complete set of tables and features you expect to lookup in those tables. See the documentation of `tf.tpu.experimental.embedding.TableConfig` and `tf.tpu.experimental.embedding.FeatureConfig` for more details on the complete set of options. We will cover the basic usage here. NOTE: multiple `FeatureConfig` objects can use the same `TableConfig` object, allowing different features to share the same table: ```python table_config_one = tf.tpu.experimental.embedding.TableConfig( vocabulary_size=..., dim=...) table_config_two = tf.tpu.experimental.embedding.TableConfig( vocabulary_size=..., dim=...) feature_config = { 'feature_one': tf.tpu.experimental.embedding.FeatureConfig( table=table_config_one), 'feature_two': tf.tpu.experimental.embedding.FeatureConfig( table=table_config_one), 'feature_three': tf.tpu.experimental.embedding.FeatureConfig( table=table_config_two)} ``` There are two modes under which the `TPUEmbedding` class can used. This depends on if the class was created under a `TPUStrategy` scope or not. Under `TPUStrategy`, we allow access to the method `enqueue`, `dequeue` and `apply_gradients`. We will show examples below of how to use these to train and evaluate your model. Under CPU, we only access to the `embedding_tables` property which allow access to the embedding tables so that you can use them to run model evaluation/prediction on CPU. First lets look at the `TPUStrategy` mode. Initial setup looks like: ```python strategy = tf.distribute.TPUStrategy(...) with strategy.scope(): embedding = tf.tpu.experimental.embedding.TPUEmbedding( feature_config=feature_config, optimizer=tf.tpu.experimental.embedding.SGD(0.1)) ``` When creating a distributed dataset that is to be passed to the enqueue operation a special input option must be specified: ```python distributed_dataset = ( strategy.distribute_datasets_from_function( dataset_fn=..., options=tf.distribute.InputOptions( experimental_fetch_to_device=False)) dataset_iterator = iter(distributed_dataset) ``` Different feature inputs can have different shapes. For dense and sparse tensor, rank 2 and above is supported. For ragged tensor, although only rank 2 is supported, you can specify the output shape to be rank 2 and above. The output shape specified in the FeatureConfig has the first priority. The input shape passed in build method has second priority and the input shapes auto detected from input feature has the lowest priority. The latter two will be converted to output shapes by omitting the last dimension. If the lower priority one has output shapes which don't match the former one. A ValueError will be raised. Only when the former one has undefined output shapes, the latter one can override. NOTE: All batches passed to the layer can have different input shapes. But these input shapes need to match with the output shapes set by either `FeatureConfig` or build method except for ragged tensor. Only 2D ragged tensor with output shape set to higher dimensions is allowed as long as the total number of elements matches. All subsequent calls must have the same input shapes. In the event that the input shapes cannot be automatically determined by the enqueue method, you must call the build method with the input shapes or provide output shapes in the `FeatureConfig` to initialize the layer. To use this API on TPU you should use a custom training loop. Below is an example of a training and evaluation step: ```python @tf.function def training_step(dataset_iterator, num_steps): def tpu_step(tpu_features): with tf.GradientTape() as tape: activations = embedding.dequeue() tape.watch(activations) model_output = model(activations) loss = ... # some function of labels and model_output embedding_gradients = tape.gradient(loss, activations) embedding.apply_gradients(embedding_gradients) # Insert your model gradient and optimizer application here for _ in tf.range(num_steps): embedding_features, tpu_features = next(dataset_iterator) embedding.enqueue(embedding_features, training=True) strategy.run(tpu_step, args=(tpu_features, )) @tf.function def evaluation_step(dataset_iterator, num_steps): def tpu_step(tpu_features): activations = embedding.dequeue() model_output = model(activations) # Insert your evaluation code here. for _ in tf.range(num_steps): embedding_features, tpu_features = next(dataset_iterator) embedding.enqueue(embedding_features, training=False) strategy.run(tpu_step, args=(tpu_features, )) ``` NOTE: The calls to `enqueue` have `training` set to `True` when `embedding.apply_gradients` is used and set to `False` when `embedding.apply_gradients` is not present in the function. If you don't follow this pattern you may cause an error to be raised or the tpu may deadlock. In the above examples, we assume that the user has a dataset which returns a tuple where the first element of the tuple matches the structure of what was passed as the `feature_config` argument to the object initializer. Also we utilize `tf.range` to get a `tf.while_loop` in order to increase performance. When checkpointing your model, you should include your `tf.tpu.experimental.embedding.TPUEmbedding` object in the checkpoint. It is a trackable object and saving it will save the embedding tables and their optimizer slot variables: ```python checkpoint = tf.train.Checkpoint(model=model, embedding=embedding) checkpoint.save(...) ``` On CPU, only the `embedding_table` property is usable. This will allow you to restore a checkpoint to the object and have access to the table variables: ```python model = model_fn(...) embedding = tf.tpu.experimental.embedding.TPUEmbedding( feature_config=feature_config, optimizer=tf.tpu.experimental.embedding.SGD(0.1)) checkpoint = tf.train.Checkpoint(model=model, embedding=embedding) checkpoint.restore(...) tables = embedding.embedding_tables ``` You can now use table in functions like `tf.nn.embedding_lookup` to perform your embedding lookup and pass to your model. feature_config optimizer#pipeline_execution_with_tensor_corectj|_t|jtj tj f|_||_||_ g|_ tj|D]'}|jj|j)t|jj dd}|r |j"nd|_g|_tj|D]@}|j(|j&vs|j&j|j(Bg}t+|j&D]\}}|j,||_|j, |jrQt|j,t.j0s-t3dj5t7|j,|j8dj5||_|j8|vrt3d|j8d|j|j8|jrg|_|j&D]t}t=|j,j>s#|j,j>|j:vsF|j:j|j,j>vt/j@|j|_!d|_"d|_#y) a,Creates the TPUEmbedding mid level API object. ```python strategy = tf.distribute.TPUStrategy(...) with strategy.scope(): embedding = tf.tpu.experimental.embedding.TPUEmbedding( feature_config=tf.tpu.experimental.embedding.FeatureConfig( table=tf.tpu.experimental.embedding.TableConfig( dim=..., vocabulary_size=...))) ``` Args: feature_config: A nested structure of `tf.tpu.experimental.embedding.FeatureConfig` configs. optimizer: An instance of one of `tf.tpu.experimental.embedding.SGD`, `tf.tpu.experimental.embedding.Adagrad` or `tf.tpu.experimental.embedding.Adam`. When not created under TPUStrategy may be set to None to avoid the creation of the optimizer slot variables, useful for optimizing memory consumption when exporting the model for serving where slot variables aren't needed. pipeline_execution_with_tensor_core: If True, the TPU embedding computations will overlap with the TensorCore computations (and hence will be one step old). Set to True for improved performance. Raises: ValueError: If optimizer is not one of tf.tpu.experimental.embedding.(SGD, Adam or Adagrad) or None when created under a TPUStrategy. _device_assignmentNz{} is an unsupported optimizer class. Please pass an instance of one of the optimizer classes under tf.tpu.experimental.embedding.ztable_{}z:Tables must have a unique name. Multiple tables with name z found.FT)$r get_strategy _strategy isinstancer TPUStrategy TPUStrategyV2 _using_tpu$_pipeline_execution_with_tensor_core_feature_config_output_shapesr-flattenappend output_shapegetattrextendednum_cores_per_replica_num_cores_per_replica _table_configtable enumeraterMr& _Optimizer ValueErrorformattyperH_dynamic_learning_ratescallable learning_rateget_list_of_hosts_hosts_built _verify_output_shapes_on_enqueue) r8rLrMrNfeaturedevice_assignment table_namesirbs r9__init__zTPUEmbedding.__init__sD$002DN ,2J2J2>2L2L2NODO , -*DD<</7   !5!567  !5t4E//$ ,D<</1 d00 0 !!'--01Kd001%5  # ?? &$//*@*K*KL::@&!%//2;45 5  &&q) { "66;jj\JK K$%"  &(d"%%M% U__22 3 OO ) )1M1M M  & & - -eoo.K.K LM +<.load_configs //0B0BCr:z'Done initializing TPU Embedding engine.T)rmrVr(is_tpu_embedding_initialized RuntimeError(_get_and_update_output_shapes_from_input_create_config_protorxr infor&log_tpu_embedding_configurationrfunction_create_variables_and_slots _variables_load_variables)r8per_replica_input_shapesper_replica_batch_sizerys` r9buildzTPUEmbedding.buildas@ {{    - - /   334L4JL 446d ll78<> " =!""  ""s =Arrcd}|r(|&tjd|j|}|t|tr'tjd|j|}nlt j t j|t j|jt j|}|j|}|"|j||j||jy)z>@ "),,$/ $>!$$++L,@,@,BC D (AAJJ..55K 77<1!^^44>>L$/2G$GL! (AAMM" 114 ""+/l  (  % %  6 _'Ks I / IrHc |js td|js td|jxsd}t j |j |g}tt j|t j|j |jD]\\}}}}|D cgc]} | |z c} |jjgz} |5t|tjst!dt#|d|d|5|j$| k7rat!dj'|j$|| t)j*d |t-j.| t0j2 }|j5t-j6||j$ t9j:||j<D cgc],} t?j@| t0j2 .c} |jBjE } | tG| |yycc} wcc} w) awApplies the gradient update to the embedding tables. If a gradient of `None` is passed in any position of the nested structure, then an gradient update with a zero gradient is applied for that feature. For optimizers like SGD or Adagrad, this is the same as applying no update at all. For lazy Adam and other sparsely applied optimizers with decay, ensure you understand the effect of applying a zero gradient. ```python strategy = tf.distribute.TPUStrategy(...) with strategy.scope(): embedding = tf.tpu.experimental.embedding.TPUEmbedding(...) distributed_dataset = ( strategy.distribute_datasets_from_function( dataset_fn=..., options=tf.distribute.InputOptions( experimental_fetch_to_device=False)) dataset_iterator = iter(distributed_dataset) @tf.function def training_step(): def tpu_step(tpu_features): with tf.GradientTape() as tape: activations = embedding.dequeue() tape.watch(activations) loss = ... # some computation involving activations embedding_gradients = tape.gradient(loss, activations) embedding.apply_gradients(embedding_gradients) embedding_features, tpu_features = next(dataset_iterator) embedding.enqueue(embedding_features, training=True) strategy.run(tpu_step, args=(tpu_features, )) training_step() ``` Args: gradients: A nested structure of gradients, with structure matching the `feature_config` passed to this object. name: A name for the underlying op. Raises: RuntimeError: If called when object wasn't created under a `TPUStrategy` or if not built (either by manually calling build or calling enqueue). ValueError: If a non-`tf.Tensor` non-`None` gradient is passed in, or a `tf.Tensor` of the incorrect shape is passed in. Also if the size of any sequence in `gradients` does not match corresponding sequence in `feature_config`. TypeError: If the type of any sequence in `gradients` does not match corresponding sequence in `feature_config`. zYapply_gradients is not valid when TPUEmbedding object is not created under a TPUStrategy.zzapply_gradients called on unbuilt TPUEmbedding object. Please either call enqueue first or manually call the build method.rNzfound non-tensor type: z at path .z9Found gradient of shape {} at path {}. Expected shape {}.z|No gradient passed for feature %s, sending zero gradient. This may not be correct behavior for certain optimizers like Adam.dtype)shape)inputslearning_ratesconfig)$rVr{rmr`r-rrXr flatten_with_joined_string_pathsrZrYrbdimrS tensor_libTensorrergrrfr rrzerosrfloat32r[reshaper(send_tpu_embedding_gradientsrhrcastrxSerializeToStringrI) r8 gradientsrHr_updated_gradientspathgradientror\xfull_output_shapefnrGs r9apply_gradientszTPUEmbedding.apply_gradientsis,n ?? F GG ;; 2 33!77<1t33Y?36 --i8 T))*D,?,?4A=/x'<?KK144K --  O  j:;L;L&M%d8n%5YtfA FH H   >>. .''-vhnnd.?(AB B  $%) +??#4FNNK   HHNN ;=3=6  - - 22  MM"$fnn 5 !!335  7B BCL4 s 7 H:1H? cl|js td|js tdtjt |j j|j j}|t|dj|tj|j|S)aGet the embedding results. Returns a nested structure of `tf.Tensor` objects, matching the structure of the `feature_config` argument to the `TPUEmbedding` class. The output shape of the tensors is `(*output_shape, dim)`, `dim` is the dimension of the corresponding `TableConfig`. For output_shape, there are three places where it can be set. 1. FeatureConfig provided in the __init__ function. 2. Per_replica_output_shapes by directly calling the build method after initializing the tpu embedding class. 3. Auto detected from the shapes of the input feature. The priority of these places is the exact same order. ```python strategy = tf.distribute.TPUStrategy(...) with strategy.scope(): embedding = tf.tpu.experimental.embedding.TPUEmbedding(...) distributed_dataset = ( strategy.distribute_datasets_from_function( dataset_fn=..., options=tf.distribute.InputOptions( experimental_fetch_to_device=False)) dataset_iterator = iter(distributed_dataset) @tf.function def training_step(): def tpu_step(tpu_features): with tf.GradientTape() as tape: activations = embedding.dequeue() tape.watch(activations) loss = ... # some computation involving activations embedding_gradients = tape.gradient(loss, activations) embedding.apply_gradients(embedding_gradients) embedding_features, tpu_features = next(dataset_iterator) embedding.enqueue(embedding_features, training=True) strategy.run(tpu_step, args=(tpu_features, )) training_step() ``` Args: name: A name for the underlying op. Returns: A nested structure of tensors, with the same structure as `feature_config` passed to this instance of the `TPUEmbedding` object. Raises: RuntimeError: If called when object wasn't created under a `TPUStrategy` or if not built (either by manually calling build or calling enqueue). zQdequeue is not valid when TPUEmbedding object is not created under a TPUStrategy.zrdequeue called on unbuilt TPUEmbedding object. Please either call enqueue first or manually call the build method.) num_outputsrr)rVr{rmr(recv_tpu_embedding_activationsrrxrrrIrGr-pack_sequence_asrX)r8rH activationss r9dequeuezTPUEmbedding.dequeuesp ?? 8 99 ;; - .. 88**==>!!3357K  KN%%t,  !5!5{ CCr:c fd}i}jD]j}js||||j<%tjt j 5||||j<dddl|S#1swYxxYw)aWCreate variables for TPU embeddings. Note under TPUStrategy this will ensure that all creations happen within a variable creation scope of the sharded variable creator. Returns: A dict of dicts. The outer dict is keyed by the table names and the inner dicts are keyed by 'parameters' and the slot variable names. cjjffddfd jjj }fd}j j j ||}ni}||d<|S)zCreate all variables.cj~tj||}tj||||S)Nr)rH initial_valuerr trainable) functoolspartial tf_variablesVariable)rHrr initializerrrvariable_shapes r9getterzRTPUEmbedding._create_variables_and_slots..create_variables..getter2sA "))+~057 $$'  ! !r:cNj||tj|S)N)rHrrrrr) _add_variable_with_custom_getterrr)rHrrrr8rs r9variable_creatorz\TPUEmbedding._create_variables_and_slots..create_variables..variable_creator?s344# .. 5! !r:)rc8jdz|z|dS)N/FrH)rHrrbrs r9 slot_creatorzXTPUEmbedding._create_variables_and_slots..create_variables..slot_creatorNs& S 04 7 + %' 'r:r)T)vocabulary_sizerrHrrVrM _create_slots)rbrr slot_varsrrrr8s` @@@r9create_variableszBTPUEmbedding._create_variables_and_slots..create_variables.s--uyy9n ! !$EJJ0A0A26//.ACj'  $OO11*lK  *i  r:N)rarVrHr variable_creator_scopemake_sharded_variable_creatorrl)r8rr!rbs` r9rz(TPUEmbedding._create_variables_and_slots!s*\I##: __ 0 7 %**  2 2 )$++ 68 :"25"9)EJJ  : : :   : :s "BB c|jr{|jrntjst j sEt |jj|j|j|jyyyyrv) rVrmrexecuting_eagerlyr$r_load_variables_implrxrrlrrar7s r9rzTPUEmbedding._load_variablesgse  4;;  % % 'L,H,H,J4--??A;;??--/-K'r:c|jr{|jrntjst j sEt |jj|j|j|jyyyyrv) rVrmrrr$r_retrieve_variables_implrxrrlrrar7s r9_retrieve_variablesz TPUEmbedding._retrieve_variablessse  4;;  % % 'L,H,H,Jt11CCE#{{##113-K'r:c |tdj||j||jtjt j |dgtj|j|y)Nz`Weight specified for dense input {}, which is not allowed. Weight will always be 1 in this case.r) rerfr[rrrrrint64) r8rweightindicesvaluesweights int_zeros float_zerosrs r9_add_data_for_tensorz!TPUEmbedding._add_data_for_tensorsi   228&, @@ NN9 MM(-- 1 1&2$ ?NO NN;r:c 0tj|jtj} |j j dk(r9| jsf| jdkDrWtj| ddgddgg} n9| jdkDr*tjd|j j |j| |jtj|jtj||t!|t"j$s$t'dj)|t+||jtj|jtj,y|j|y)Nrrr)paddingszZInput tensor is rank %d which is above 2, the max_sequence_length setting will be ignored.zOWeight for {} is type {} which does not match type input which is SparseTensor.)rrrrint32rrr\rrpadr rr[rrrSr SparseTensorrerfrgr) r8rrrrrrrrrosample_indicess r9_add_data_for_sparse_tensorz(TPUEmbedding._add_data_for_sparse_tensors*]]6>>6<<@N ||A  ! !g&A&AA&E" q!fq!f%57  $ $q (@ LL     NN>" MM(-- v||<=   : : ;==CV!4<>12 2nnX]]6==&..AB nn[!r:c |jtj|jtj |jtj|j tj||t|tjs$tdj|t||jtj|j tjy|j|y)NzOWeight for {} is type {} which does not match type input which is RaggedTensor.)r[rr row_splitsrrrrrSr" RaggedTensorrerfrgr) r8rrr rrrrrros r9_add_data_for_ragged_tensorz(TPUEmbedding._add_data_for_ragged_tensorshmmF$5$5v||DE MM(-- v||<=   : : ;==CV!4<>12 2nnX]]6==&..AB nn[!r: flat_inputs flat_weights flat_featuresdevice_ordinal mode_overridec |jDcgc]}|j}}g}g} g} tjdtj } tjdtj } t|||D]\} }\}}t| tjr|j| ||| | | | |=t| tjr|j| ||| | | | || qt| tj r|j#| ||| | | | || t%dj'|t)| t+j,|| | |||Scc}w)aOutputs a the enqueue op given the inputs and weights. Args: flat_inputs: A list of input tensors. flat_weights: A list of input weights (or None) of the same length as flat_inputs. flat_features: A list of FeatureConfigs of the same length as flat_inputs. device_ordinal: The device to create the enqueue op for. mode_override: A tensor containing the string "train" or "inference". Returns: The enqueue op. )rrzjInput {} is of unknown type {}. Please only pass Tensor, SparseTensor or RaggedTensor as input to enqueue.)sample_indices_or_row_splitsembedding_indicesaggregation_weightsrr combiners)racombinerrrrrrrrSrrrrr r r"rrrerfrgr(,enqueue_tpu_embedding_arbitrary_tensor_batch)r8rrrrrrbrindices_or_row_splitsrrrrinprrros r9_generate_enqueue_opz!TPUEmbedding._generate_enqueue_opsr,.2-?-?@E@I@ FG FLL9I//$fnn=K),\=)2=$V_dG C** + !!#v/Df")9k4 I c=55 6 ((f6K)/))4dG E c=55 6 ((f6K)/))4dG E$$*F4c$;= ==$  ? ?%: ##%  MAsE&cdtj}d}|Q|j}|,t|tj rd}n|j }|,|rnt|dd}|Q|tjk7r/|r-tdjtj||S)z9Raises an error if we are not in the TPUReplicateContext.FNT outer_graphaFCurrent graph {} does not match graph which contains TPUReplicateContext {}. This is most likely due to the fact that enqueueing embedding data is called inside control flow or a tf.function inside `strategy.run`. This is not supported because outside compilation fails to extract the enqueue ops as the head of a computation.) rget_default_graph_get_control_flow_contextrSr'TPUReplicateContext outer_contextr]r{rf)r8graph in_tpu_ctxctxs r9/_raise_error_for_incorrect_control_flow_contextz> ?*  O  e]D1e   %%''J   "6#"7"7"95A  CC r:c\tj|dD]\}}|jjdk(r |jj d}|r?t dj ||jj|jjy#t $rd}YVwxYw)zAChecks all tensors in features to see if they are a direct input.Texpand_composites Placeholder_tpu_input_identityFaMReceived input tensor {} which is the output of op {} (type {}) which does not have the `_tpu_input_identity` attr. Please ensure that the inputs to this layer are taken directly from the arguments of the function called by strategy.run. Two possible causes are: dynamic batch size support or you are using a keras layer and are not passing tensors which match the dtype of the `tf.keras.Input`s.If you are triggering dynamic batch size support, you can disable it by passing tf.distribute.RunOptions(experimental_enable_dynamic_batch_size=False) to the options argument of strategy.run().N)r-rrGrgget_attrrerfrH)r8featuresr input_tensoris_inputs r9"_raise_error_for_non_direct_inputsz/TPUEmbedding._raise_error_for_non_direct_inputss#CCD*Hl    .??++,AB *+1&1=1E1E1=1E1E+G H HH sB B+*B+cd}t||D]\}}tj|rtj|d}n|g}|D]}|jj dk(rv|jj djj dk(rF|jj djj D]}|||j|||jy)z.check_device:sM  ! ! - -m j@jC|jD}tGjHtKjL|5jO|||||||| } tQ|| jS|dddFyrdnd} t>j@jC|}|jTd k7rtd jW|tGjHtKjL|5jO|jD| } tQ|dddy#1swYcxYw#1swYyxYw) aEnqueues id tensors for embedding lookup. This function enqueues a structure of features to be looked up in the embedding tables. We expect that the input shapes of each of the tensors in features matches the output shapes set via FeatureConfig or build method (if any). the output shapes will be auto detected based on the input shapes with the max_sequence_length or output shape setting in the FeatureConfig. Note that the output shapes is based on per replica batch size. If your input dataset is batched to the global batch size and you use `tf.distribute.TPUStrategy`'s `experimental_distribute_dataset` or if you use `distribute_datasets_from_function` and batch to the per core batch size computed by the context passed to your input function, the output shapes should match automatically. The auto detected the output shapes: 1. For dense tensor, if rank 2 or above, make sure the tensor has last dimension as 1. The output shape will be the input shape excluding the last dimension. 2. For sparse tensor, make sure the tensor has rank 2 and above. a. If feature config has max_sequence_length equals 0 or output shape set (the max_sequence_length setting will be ignored), the output shape will be the input shape excluding the last dimension. b. Otherwise, if the tensor is rank 2, the output shape will be input shape with last dimension set as max_sequence_length. If the tensor is above rank 2, the output shape will be the input shape excluding the last dimension and the last dimension of the output shape will be set to max_sequence_length. 3. For ragged tensor, make sure the tensor has rank 2. a. If feature config has max_sequence_length equals 0 or output shape set (the max_sequence_length setting will be ignored), the output shape will be the input shape excluding the last dimension. b. Otherwise, the output shape will be the input shape excluding the last dimension and the last dimension of the output shape will be set to max_sequence_length. ```python strategy = tf.distribute.TPUStrategy(...) with strategy.scope(): embedding = tf.tpu.experimental.embedding.TPUEmbedding(...) distributed_dataset = ( strategy.distribute_datasets_from_function( dataset_fn=..., options=tf.distribute.InputOptions( experimental_fetch_to_device=False)) dataset_iterator = iter(distributed_dataset) @tf.function def training_step(): def tpu_step(tpu_features): with tf.GradientTape() as tape: activations = embedding.dequeue() tape.watch(activations) loss = ... # some computation involving activations embedding_gradients = tape.gradient(loss, activations) embedding.apply_gradients(embedding_gradients) embedding_features, tpu_features = next(dataset_iterator) embedding.enqueue(embedding_features, training=True) strategy.run(tpu_step, args=(tpu_features,)) training_step() ``` NOTE: You should specify `training=True` when using `embedding.apply_gradients` as above and `training=False` when not using `embedding.apply_gradients` (e.g. for frozen embeddings or when doing evaluation). For finer grained control, in the above example the line ``` embedding.enqueue(embedding_features, training=True) ``` may be replaced with ``` per_core_embedding_features = self.strategy.experimental_local_results( embedding_features) def per_core_enqueue(ctx): core_id = ctx.replica_id_in_sync_group device = strategy.extended.worker_devices[core_id] embedding.enqueue(per_core_embedding_features[core_id], device=device) strategy.experimental_distribute_values_from_function( per_core_queue_inputs) ``` Args: features: A nested structure of `tf.Tensor`s, `tf.SparseTensor`s or `tf.RaggedTensor`s, with the same structure as `feature_config`. Inputs will be downcast to `tf.int32`. Only one type out of `tf.SparseTensor` or `tf.RaggedTensor` is supported per call. weights: If not `None`, a nested structure of `tf.Tensor`s, `tf.SparseTensor`s or `tf.RaggedTensor`s, matching the above, except that the tensors should be of float type (and they will be downcast to `tf.float32`). For `tf.SparseTensor`s we assume the `indices` are the same for the parallel entries from `features` and similarly for `tf.RaggedTensor`s we assume the row_splits are the same. training: Defaults to `True`. If `False`, enqueue the batch as inference batch (forward pass only). Do not call `apply_gradients` when this is `False` as this may lead to a deadlock. name: A name for the underlying op. device: The device name (e.g. '/task:0/device:TPU:2') where this batch should be enqueued. This should be set if and only if features is not a `tf.distribute.DistributedValues` and enqueue is not being called inside a TPU context (e.g. inside `TPUStrategy.run`). Raises: ValueError: When called inside a strategy.run call and input is not directly taken from the args of the `strategy.run` call. Also if the size of any sequence in `features` does not match corresponding sequence in `feature_config`. Similarly for `weights`, if not `None`. If input shapes of features is unequal or different from a previous call. RuntimeError: When called inside a strategy.run call and inside XLA control flow. If batch_size is not able to be determined and build was not called. TypeError: If the type of any sequence in `features` does not match corresponding sequence in `feature_config`. Similarly for `weights`, if not `None`. zQenqueue is not valid when TPUEmbedding object is not created under a TPUStrategy.zConfigured not to check output shapes on each enqueue() call; please ensure build() was called with output shapes to initialize the TPU for embeddings.Nctjtjdtjd}j d|} t |yy)z-Generate enqueue ops for outside compilation.train inferencerrrN)rwhere_v2rconstantrrI)r enqueue_oprrrrHr8rEs r9generate_enqueue_opsz2TPUEmbedding.enqueue..generate_enqueue_opssn"**8+6+?+?+H+6+?+? +LN .. }R'/)    D ) r:rHrIc|yt|tjr%tj|j d|St|t jr%tj|j d|Std)Nr)num_or_size_splitsaxis)sp_input num_splitrQz'SPMD does not support raggedTensor yet.) rSrrrsplitr`rr rsparse_split_v2re)tsidxr8s r9 _split_fnz'TPUEmbedding.enqueue.._split_fns : J-- .!%!._maybe_split/s:  & & . &&yg>(##Hi8 8r:rrJr5z$Non-TPU device {} passed to enqueue.),rVr{r(r-rrXrnrYrmre_get_input_shapesrrrrZrrrrDr2r'outside_compilationrangerRrrselect_replicar`r^ _tpu_devicesworker_devicesr6r7r8 device_indexrrrget_host_for_devicerrIr[r9rf)r8r/rrErHrin_tpu_context per_replicarr@_rNr enqueue_opsr^ replica_idreplica_inputsreplica_weights tpu_devicesr\ tpu_devicerrM device_specrXrrrs` `` @@@@r9enqueuezTPUEmbedding.enqueueTsL ?? 8 99IIKNt33X>  0 0    &' ' dNk++ Kl  %   3 3L AC,,x(K6C ,,L   !5!5w?\\'*l99$:N:NOM'MJ++KD --h7**())*>? !)g{m kF 9dnnAAB+*)889DF*99*:FH  & &//< +G"7+*""..z:GG zz+99*EF +22^W5_g6-] 3LJJ-   z * + +  ++<"*g{m((44V*.. }&33'/)    D )**% + +$**s;AN;/O;O Orhrgc Bg}ttj|tj|jD]\\}}}|st j d|}n|}t|tjr$|j|j||||bt|tjr$|j|j||||t|tj s|j|j#|||||S)z+Get the input shapes from the input tensor.r)rr-rrZrXrrbrSrrr[_get_input_shape_for_tensorrr "_get_input_shape_for_sparse_tensorr"r"_get_input_shape_for_ragged_tensor) r8tensorsrhrgrr maybe_tensorrors r9r_zTPUEmbedding._get_input_shapesds L), --g6 T))**, %|g!00LA FJ-- .  , ,VWk4 P  fm88 9  3 3d  fm88 9  3 3d  ' 0 r:cV|jj}t|dkr$tdj t||t|dkDr"|ddk7rtdj ||j r|r|d|j z|d<t |S)z)Get the input shape for the dense tensor.rzXOnly rank 1 and above dense tensor is supported, find rank {} sparse tensor for input {}rzRank 2 or above dense tensor should have last dimension as 1 as the last dimension will always be reduced. Instead got dense tensor as shape {}r)rrrrerfr`rr8rrorhrrs r9rsz(TPUEmbedding._get_input_shape_for_tensors LL "E 5zA~ BBH&u:tC- .. 5zA~%)q.  117 @@  ""{qT888eAh u r:c|jj}t|dkr$tdj t|||j sE|j dkDr6t|dk(r(|jt|dz |j |jr|r|dr|d|jz|d<t|S)z*Get the input shape for the sparse tensor.rzYOnly rank 2 and above sparse tensor is supported, find rank {} sparse tensor for input {}rr) rrrrerfr\rrr`rrys r9rtz/TPUEmbedding._get_input_shape_for_sparse_tensors LL "E 5zA~ BBH&u:tC- ..   G$?$?!$C Uq  SZ!^W%@%@A ""{uQxqT888eAh u r:c8~|jj}t|dk7r$tdj t|||j s7|j dkDr(|jt|dz |j t|S)z*Get the input shape for the ragged tensor.rzOOnly rank 2 ragged tensor is supported, find rank {} ragged tensor for input {}rr) rrrrerfr\rrrrys r9ruz/TPUEmbedding._get_input_shape_for_ragged_tensors  LL "E 5zQ BBH&u:tC- ..   G$?$?!$C ll3u:>7#>#>? u r:incoming_output_shapesctj|j|g}t|j|D]*\}}|r|j ||j |,||_y)a"Update the existing output shapes based on the new output shapes. The existing output shapes always have higher piority than the new incoming output shapes. Args: incoming_output_shapes: nested structure of TensorShape to override the existing output shapes. N)r-rrYrr[)r8r|updated_output_shapesold_output_shapeincoming_output_shapes r9rz"TPUEmbedding._update_output_shapesss t224JK36t7J7J7M4O<// $$%56$$%:; < 0Dr:c tj|j|ttj|j |j|D]\\}}}}|s |st |dk(st |dk(rt |t |kDrDt |t |k7s|j||rntd|d|d|y)zBCheck the incoming output shapes against the output shapes stored.rrz-Inconsistent shape founded for input feature z, Output shape is set to be z , But got incoming output shape N) r-rrYrrrXr_is_tensor_shape_matchre)r8r|rrirrs r9rz!TPUEmbedding._check_output_shapess t224JK>A --d.B.BC 3?5H: q#%: 3 % &! +s3H/I0*+c2G.HH   C !%# #*.*E*E "7+9=dVD++;*<=//D.EGH HHr:cttj|j|jD]*\\}}}|j rt d|d|dy)z+Check if the output shape is fully defined.zInput Feature z has output shape set as zt which is not fully defined. Please specify the fully defined shape in either FeatureConfig or for the build method.N)rr-rrXrYis_fully_definedre)r8rrir\s r9rz/TPUEmbedding._check_output_shapes_fully_definedsp#& --d.B.BC $( q< * * ,TF";n' '( ( (r:shape_ashape_bc~t|j|jD]\}}|s |s ||k7syy)z&Check if shape b matches with shape a.FT)rr)r8rrs_as_bs r9rz#TPUEmbedding._is_tensor_shape_matchs?)7??+<=S  r:c g}tj|jD]^}|js6|jdkDr'|j t ||jgE|j t |`|S)z*Get the output shapes from the batch size.r)r-rZrXr\rr[r)r8rrros r9rz/TPUEmbedding._get_output_shapes_from_batch_sizes~M<< 4 45B  ! !g&A&AA&E /1L1LM N P [)?@A B r:ct|||S)zECreate a TPUEmbedding copy for checkpoint/async_checkpoint_helper.py.)rLrMrN)rK)r8rLrMrNs r9!_create_copy_for_async_checkpointz.TPUEmbedding._create_copy_for_async_checkpoint s %,O QQr:)F)NNrv)NTNN)8r;r<r=r>r r& FeatureConfigrrrdboolrsrrrrrr|rr?r TableConfigrrrrrr}r rrrrrrr rinternal_types NativeObjectr Operationrr(r2rDrqr_rsrtrurrrrrrr@r:r9rKrKKs`L38 p12@@(JKp10;;<p1,0 p1dDNJN ""*5cH1D+E"F "?C.24. (k): ;4.'sm4.l{+04[0A4  - " . . 0E0E EF -  -DB&@@BHiXd^iVKD(4.KDZD D$t\2223 34DL / 3  "< "B334B."="=>?B0>>? B  B  B }}BH6H@-@!# N* N* TN N* tn N*`"&8< K>(0&0${:K0&Hk9JH2 (K&16: Qr:rKrhostsr! table_configc Zd}t|D]\}}tj|5tj|||}|D]I}|j j d|jt|||d||jd}K dddy#1swYxYw)aLoad embedding tables to onto TPU for each table and host. Args: config: A serialized TPUEmbeddingConfiguration proto. hosts: A list of CPU devices, on per host. variables: A dictionary of dictionaries of TPUEmbeddingVariables. First key is the table name, second key is 'parameters' or the optimizer slot name. table_config: A list of tf.tpu.experimental.embedding.TableConfig objects. cfd}|S)Nct|jk\r"tj|jdS|jSr5)rr!r zeros_like)rhost_ids r9select_or_zerosz@_load_variables_impl..select_fn..select_or_zeros's> C $ $ ##AKKN33 [[ !!r:r@)rrs` r9 select_fnz'_load_variables_impl..select_fn%s" r: table_name num_shardsshard_idrNr@) rcrrr-rZrM_loadrHr) rrr!rrrhosthost_variablesrbs r9rrs !'mgt D  )))G*"?@ @GAt"%**-d3+ s;001 1  ! !' * 1 1)A, ? @3sB'D'DD c P|jD]}|jgSrv)rr trackables unused_kwargs trackables r9_save_callbackrps+$$&$i !!#$ )r:c N|jD]}|jyrv)rrrs r9_restore_callbackrvs&$$& i  r:TPUEmbeddingCallbackc"t|tSrv)rSrK)rs r9r}s 1l3r:F) predicatesave_fn restore_fnstrict_predicate_restorer.ct|dtjrd|djvs|djrld|djvr|djd}n|djd}|d||djj d|d|dj fSd|vs|dt|dstdj|d|d|d|d|dfS)zExtracts the variable creation attributes from the kwargs. Args: kwargs: a dict of keyword arguments that were passed to a variable creator scope. Returns: A tuple of variable name, shape, dtype, initialization function. rrrrHra2Unable to extract initializer function and shape from {}. Please either pass a function that expects a shape and dtype as the initial value for your variable or functools.partial object with the shape and dtype kwargs set. This is needed so that we can initialize the shards of the ShardedVariable locally.) rSrrkeywordsargsgetfuncrirerf)kwargsrs r9extract_variable_infors+()*;*;< (111 _""&)222_%..w7e_%**1-e 6NE ? # , , 0 0&/ J ? # ( ( **fw 7x _@  @AG ? #A%  && 6NF7OVG_ ? # %%r:cFdtdtjfffd }|S)zMakes a sharded variable creator given a list of hosts. Args: hosts: a list of tensorflow devices on which to shard the tensors. Returns: A variable creator function. next_creator.c d|d<t}t|\}}}}|d}|d} |d} | |z} | |z} | dzg| z| g|| z zz} g}dtj|jv}d}||d<t | D]\}}|dk(r t j|5dj|||d <|| f|d <|r;tj|d |df}tj|| |d<||z }ntj||d | |d<|j||i|d d d t||S#1swYxYw)zThe sharded variable creator.Tskip_mirrored_creatorrrr shard_inforz{}_{}rHr)rrNr)rrr. getargspecrrcrrrfr* ShardInforrr[r3)rrrrrHrrunwrapped_initial_valuerrowscolspartial_partitionfull_rows_per_host partitionsr!sharding_awareoffsetrrprrs r9sharded_variable_creatorz?make_sharded_variable_creator..sharded_variable_creators'+F "#E I2G2O/D%/?+M 8D 8Dy(* a  #44  ).?"? @ AI!Z%:%:=%I%N%NNNFF7O*%81 a  ::eAh  8 a0vd)w ~~fWo{C*$-$5$5 %4& ! A+&$-$5$5%vge%E& !t6v67 8 8 8$  55 8 8s 'BE  E )rrr)rrs` r9rrs)+6S,"7"778+6X "!r:)Wr>rtypingrrrrrrr r r abslr tensorflow.core.frameworkr tensorflow.core.protobuf.tpurtensorflow.python.distributerrrrrtensorflow.python.eagerrrtensorflow.python.frameworkrrr6rrrrr(tensorflow.python.framework.tensor_shapertensorflow.python.opsrrrr r!rtensorflow.python.ops.raggedr"tensorflow.python.saved_modelr#r$tensorflow.python.tpur%r&r'tensorflow.python.tpu.opsr(tensorflow.python.trackabler)r*tensorflow.python.typesr+rtensorflow.python.utilr,r-r. tensorflow.python.util.tf_exportr/ _HOOK_KEYrDShardedVariableMixinr3rI AutoTrackablerKrrrrrrrrregister_tf_checkpoint_saverDTyperrr@r:r9rs1(TTT4H47995+03;.+5<@+*,0;666%81-5,>)'-6 $ " ,+@@,M 45GQ=..GQ6GQT.) ) c4i !)D$t\%:%::;;<))44 ))X) ) c4i !)D$t\%:%::;;<))44 ))X  * ))3  # !%T5c?FLL(2s7:KKL!%H7" :7""3(<#<=7"r: