BVhH4dZddlmZmZmZmZmZmZddlm Z ddlm Z ddl m Z ddl m Z ddl mZddl mZdd lmZdd lmZdd lmZdd lmZdd lmZddlmZddlmZddlmZddlmZddlmZddlm Z e dGddejBZ"y)z?Mid level API for TPU Embeddings without Embedding Accelerator.)AnyDictIterableOptionalTextUnion)distribute_lib) tpu_strategy)dtypes)ops) sparse_tensor)tensor) array_ops) embedding_ops)math_ops) sparse_ops) variables) ragged_tensor)tpu_embedding_base)tpu_embedding_v2_utils)tpu_replication)nest) tf_exportz)tpu.experimental.embedding.TPUEmbeddingV0c eZdZdZdeej efdeejffd Z e de ejejffdZde ee eejfffdZdZ dd ej*d ej*d eedej*fd Zd ej*d edej*fdZ ddedeedefdZdej8d eej8dejdej dej*f dZdej>d eej>dejdej dej*f dZ xZ!S)TPUEmbeddingV0a The TPUEmbedding mid level API running on TPU without Embedding accelerator. NOTE: This mid level API is not intended for large embedding table lookup. Embedding tables will be replicated across devices rather than sharding across them. To do large embedding table lookup, please use the `tpu.experimental.embedding.TPUEmbedding` class. This class is an alternative way to do embedding lookups when the TPU doesn't support any version of embedding feature. See `tpu.experimental.tpu_hardware_feature.embedding_feature` for a detailed explanation. This class has to be created under the `TPUStrategy`, Otherwise a RuntimeError will be raised. ```python strategy = tf.distribute.TPUStrategy(...) with strategy.scope(): embedding = tf.tpu.experimental.embedding.TPUEmbeddingV0( feature_config=feature_config, optimizer=tf.tpu.experimental.embedding.SGD(0.1)) ``` When creating a distributed dataset that is to be passed to the lookup operation a special input option must be specified: ```python distributed_dataset = ( strategy.distribute_datasets_from_function( dataset_fn=..., options=tf.distribute.InputOptions( experimental_fetch_to_device=False)) dataset_iterator = iter(distributed_dataset) ``` Below is an example of a training and evaluation step: ```python optimizer = tf.keras.optimizers.SGD(0.1) @tf.function def training_step(dataset_iterator, num_steps): def tpu_step(embedding_features): with tf.GradientTape() as tape: tape.watch(embedding.embedding_table.values()) activation = embedding(embedding_features) model_output = model(activations) loss = ... # some function of labels and model_output embedding_gradients = tape.gradient(loss, embedding.embedding_table.values()) optimizer.apply_gradients(list(zip(gradients, mid_level_api.embedding_tables.values()))) # Insert your model gradient and optimizer application here for _ in tf.range(num_steps): strategy.run(tpu_step, args=(next(dataset_iterator), )) @tf.function def evalution_step(dataset_iterator, num_steps): def tpu_step(embedding_features): activations = embedding(embedding_features) model_output = model(activations) # Insert your evaluation code here. for _ in tf.range(num_steps): strategy.run(tpu_step, args=(next(dataset_iterator), )) ``` NOTE: The optimizer used here is a Keras optimizer. In order to make the slot variable creation stay consistent between Keras optimizers and embedding optimizers, the `slot_variable_creation_fn` argument of the embedding optimizers has to be passed with the Keras `add_slot` function. Also note that the slot names might be slightly different between them. ```python optimizer = tf.keras.optimizers.Adagrad(learning_rate=0.1) def slot_variable_creation_fn(table, slot_names, slot_initializers): slots = {} for slot, initializer in zip(slot_names, slot_initializers): slots[slot] = optimizer.add_slot(table, slot, initializer) return slots embedding_optimizer = tf.experimental.embedding.Adagrad( learning_rate=0.1, slot_variable_creation_fn=slot_variable_creation_fn) # Use the embedding optimizer to create mid level api and keras optimizer to # apply gradients. ``` feature_config optimizerctt| ||tj|_t |j tjtjfs$tdj|j d|_ y)Nz@TPUEmbeddingV0 should be created under TPUStrategy but found {}.F) superr__init__r get_strategy _strategy isinstancer TPUStrategy TPUStrategyV2 RuntimeErrorformat_built)selfrr __class__s V/home/dcms/DCMS/lib/python3.12/site-packages/tensorflow/python/tpu/tpu_embedding_v1.pyr zTPUEmbeddingV0.__init__sp .$(C#002DN dnn#//1K1KL N  L 6$.. ! ##DKreturnc|j|jDcic]}||j|jd!c}Scc}w)z;Returns a dict of embedding tables, keyed by `TableConfig`. parameters) _maybe_build _table_config _variablesname)r)tables r+embedding_tableszTPUEmbeddingV0.embedding_tablessO ''   tuzz*<88  s$Acli}|jD]"}|j|d||j<$|S)a)Create variables for TPU embeddings. Note that this will always ensure that the variable is created under the TPUStrategy. Returns: A dict of dicts. The outer dict is keyed by the table names and the inner dicts are keyed by 'parameters' and the slot variable names. T) trainable)r1_create_variablesr3)r)rr4s r+_create_variables_and_slotsz*TPUEmbeddingV0._create_variables_and_slotssFI##L"44Ud4Ki L r,c|js.tj5|jdddyy#1swYyxYwN)r(r init_scopebuild)r)s r+r0zTPUEmbeddingV0._maybe_builds= ;; >>    s <A embeddingsweightcombinerc|d}|dk(rtj|d}|S|dk(rFtj|d}tj|d}tj||}|S|dk(rqtj|d}tj|d}tj|d}tj|}tj||}|St d|)aApply the combiner to the embedding look up result on second to last axis. Args: embeddings: A Tensor of the embedding lookup result. weight: A Tensor of weight which has the same shape of the embeddings. combiner: One of "mean", "sum", "sqrtn". Defaults to "mean". Raises: ValueError: If the combiner is not one of 'mean', 'sqrtn' or 'sum'. Returns: A Tensor. meansum)axissqrtnz6combiner must be one of 'mean', 'sqrtn' or 'sum', got )r reduce_sum div_no_nanpowsqrt ValueError)r)r>r?r@ weight_sumweight_squaredweight_sum_sqrts r+_apply_combiner_to_embeddingsz,TPUEmbeddingV0._apply_combiner_to_embeddingss"h5&&z;j  V &&z;j&&vB7j&&z:>j  W &&z;j||FA.n&&~B?j j1o&&z?Cj   B8* M OOr,sequence_lengthc|jd}||kDrtj|gdd|dg}|Stj|ddgd||z gddgg}|S)a?Pad or truncate the embedding lookup result based on the sequence length. Args: embeddings: A rank 3 Tensor of the embedding lookup result. sequence_length: number of the max sequence length set in the feature config. Returns: A Tensor with second last axis padded or truncated. )rrr)beginsizer)paddings)shaperslicepad)r)r>rQoriginal_sequence_lengths r+%_pad_or_truncate_with_sequence_lengthz4TPUEmbeddingV0._pad_or_truncate_with_sequence_lengths} *//2/1?? IR",EGj  == FQ2J JKFj r,featuresweightsc `|js|jtj||jtj |}dgt |z}|+tj||tj |}tj|j}g}t|||D]\}}\} } |j| j} |t|tjrtdj| t!|t!|ur.tdj| t!|t!|| j"dkDrtdj| t|tjrP| j"dkDrtdj| |j%t'j(| |1t|t*j,r%|j%|j/||| | pt|t0j2r%|j%|j5||| | tdj| t!|tj6|j|S)aApply embedding lookup on TPUs using Tensorcore. Note that all the sparse and ragged tensors will be converted to dense tensors on CPU and then passed to the TPU to do embedding look up. Large embedding lookup is not supported by this API, use the TPUEmbedding mid level api instead. Args: features: a nested structure of Tensors, SparseTensors or RaggedTensors. weights: a nested structure of Tensors, SparseTensors or RaggedTensors or None for no weights. If not None, structure must match that of inputs, but entries are allowed to be None. Returns: A nested structure of Tensors with the same structure as inputs. Nz,Weight specified for {}, but input is dense.zPWeight for {} is of type {} but it does not match type of the input which is {}.rz8Weight specified for {}, but this is a sequence feature.z?Feature {} is a sequence feature but a dense tensor was passed.zCInput {} is type {}. Tensor, SparseTensor or RaggedTensor expected.)r(r=rassert_same_structure_feature_configflattenlen flatten_with_joined_string_pathszipr5r4r#rTensorrLr'typemax_sequence_lengthappendrembedding_lookup_v2r SparseTensor#_embedding_lookup_for_sparse_tensorr RaggedTensor#_embedding_lookup_for_ragged_tensorpack_sequence_as) r)r]r^ flat_inputs flat_weights flat_featuresoutputsinpr?pathfeaturer4s r+embedding_lookupzTPUEmbeddingV0.embedding_lookupsY& ;; jjlx)=)=>,,x(K6C ,,L   73\\'*l99$:N:NOMG(+K,9);!K$V_dG##GMM2e   c6== )<CCDIK K &\c *##)6$V d3i#HJ J ( (1 ,&&,fTl4 4 C '  & & *"F4L* * }88DE c=55 6  4 4S&%5< > ? c=55 6  4 4S&%5< > ?228&tCy2IK KA!KD  !5!5w ??r,rtr4rvcdd}tj|||\}}tj||}t j |d}||z}|j s-|jdkDr|j||j}|S|j|||jj}|S)a.Embedding lookup for sparse tensor based on its feature config. Args: inp: a single SparseTensor input. weight: None or SparseTensor which has the same shape of the input. table: a table variable. feature: a feature config. Returns: Embedding lookup result. c|Ytj|jtj|j t j|j}tj|}tj|}||fS)Ndtype) dense_shape) r rkindicesr ones_likevaluesr float32r|rsparse_tensor_to_densertr?s r+sparse_to_dense_computationzWTPUEmbeddingV0._embedding_lookup_for_sparse_tensor..sparse_to_dense_computationHsi ++ KK    &.. A)  - -c 2c008f &[r,rrTr) routside_compilationrrjr expand_dims output_shaperhr\rPr4r@)r)rtr?r4rvrr>s r+rlz2TPUEmbeddingV0._embedding_lookup_for_sparse_tensor4s( "55#V=KC225#>J  " "62 .F&J   G$?$?!$C== g113j 55j&6=mm6L6LNj r,cR|jjdk7r.tdj|jj|jd}d}t j |||||\}}t j||}tj|d}||z}|jrtj5tj|jj}ddd|k(r'|j!|||j"j$}tj&||j|j"j(gz}|S|j*dk(r'|j!|||j"j$}|S#1swYxYw) aEmbedding lookup for ragged tensor based on its feature config. Args: inp: a single rank 2 RaggedTensor input. weight: None or RaggedTensor which has the same shape of the input. table: a table variable. feature: a feature config. Returns: Embedding lookup result. Raises: ValueError: if input ragged tensor is not rank 2 or output shape set in the feature config doesn't match with the first dim size of the input. rGz7Only rank 2 ragged tensor is supported, but got rank {}rc|Wtjjtj|j t j|j}|jsV|jdkDrG|j||jf}tj|t j}||fS|jrtj5tj|jj!}ddd|k(r$|j|j}}||fS||kDrJ||zdk(rB||z}|j||f}tj|t j}||fSt#dj%|j||j|j}}||fS#1swYxYw)NrzrrXzOutput shape set in the FeatureConfig should be the factor of the input data batch size. But instead got output shape {}, input data batch size {})rrmfrom_row_splitsrr~rr r row_splitsrrh to_tensorr r<r reduce_prodnumpyrLr')rtr? batch_sizervoutput_batch_size seq_lengths r+#ragged_to_dense_outside_compilationz_TPUEmbeddingV0._embedding_lookup_for_ragged_tensor..ragged_to_dense_outside_compilation~s ++;;    &.. A NN ! !g&A&AA&Emm:w/J/J"KmL$$S?<&[;   ^^  Q&2273G3GHNNP  Q  * )9)9);v#*&[!  *!J.!3)J6* Z$< =#&&s&..A&&[))/0D0D0:*<= = mmov'7'7'9V &[5 Q Qs .GG)rtr?rrvrTNr)rXrankrLr'rrrrjrrrr r<rrrrPr4r@reshapedimrh) r)rtr?r4rvrrr>rs r+rnz2TPUEmbeddingV0._embedding_lookup_for_ragged_tensoras( yy~~  C J Jiinn  1J &P"55+  KC225#>J  " "62 .F&J >> O$001E1EFLLNO j (77 F8? 8N8NP $$ G00GMM4E4E3FFHj   $ $ )77 F8? 8N8NP OOs .FF&r;)"__name__ __module__ __qualname____doc__rr FeatureConfigrr _Optimizerr propertyr TableConfig tf_variablesVariabler5rr9r0rrfrPintr\rrwr rkrlrrmrn __classcell__)r*s@r+rr&sXt 2@@(JK 0;;<  *66 8M8MMN D$t\%:%::;;<""& "--"mm" "+1-- "H 8; }}415A@!$A@ ( A@9<A@F++++}112+ " "+&33 +9? +Z\++\}112\ " "\&33 \9? \r,rN)#rtypingrrrrrrtensorflow.python.distributer r tensorflow.python.frameworkr r r rtensorflow.python.opsrrrrrrtensorflow.python.ops.raggedrtensorflow.python.tpurrrtensorflow.python.utilr tensorflow.python.util.tf_exportrTPUEmbeddingBaserr,r+rsoF==75.+5.+/*,;6481'6 67V'88V8Vr,