BVhkM dZddlZddlZddlZddlZddlZddlZddlmZm Z m Z m Z m Z m Z mZmZmZddlmZddlmZddlmZddlmZddlmZdd lmZdd lmZdd lmZdd lmZdd lmZddlm Z ddl!m"Z"ddl!m#Z$ddl!m%Z%ddl!m&Z&ddl!m'Z'ddl!m(Z(ddl!m)Z)ddl*m+Z+ddl*m,Z,ddl*m-Z-ddl*m.Z.ddl*m/Z/ddl*m0Z0ddl*m1Z2ddl3m4Z4ddl5m6Z6ddl5m7Z7dd l5m8Z8dd!l5m9Z9dd"l5m:Z:dd#l5m;Z;dd$ldd%l?m@Z@dd&lAmBZBdd'lCmDZDdd(lCmEZEdd)lCmFZFdd*lGmHZHd+ZId,ZJd-ZKe8jZLe8jZMe8jZNeHd.ejd/0Gd1d2ZPGd3d4e,jZRGd5d6eBjZTd7ZUejGd8d9ZVejd:Gd;dZ\d?Z] dQd@eeLdAeeeeMfdBe^dCe^dDe ePdEeVf dFZ_eHdGGdHdIe7jZadJedEeebee^dKfe%je gefe e^ffdLZddMedEeefdNZfdOejdEe dKeffdPZhy)Rz?Mid level API for TPU Embeddings With V2 Embedding Accelerator.N) AnyCallableDictIterableListOptionalSequenceTupleUnion)logging)attr_value_pb2)sparse_core_layout_pb2)saveable_compat) device_util)distribute_lib) tpu_strategy)tpu_util) tpu_values)values) values_util) constant_op)device)dtypes)ops) sparse_tensor)tensor) tensor_shape) array_ops)control_flow_ops)gen_resource_variable_ops)math_ops)summary_ops_v2)variable_scope) variables) ragged_tensor)_pywrap_sparse_core_layout)tpu_embedding_base)tpu_embedding_v2_utils)#tpu_embedding_v3_checkpoint_adapter)tpu_embedding_v3_utils)tpu_replication) gen_xla_ops)base)saveable_object)compat)nest) tf_inspect) tf_export_embedding_pipeliningforwardbackwardz4tpu.experimental.embedding.SparseCoreEmbeddingConfigT)frozenceZdZUdZdZeed<dZeed<dZ e e e efed<dZ e e e efed<dZeed <d Zeed <dZeed <y) SparseCoreEmbeddingConfigz Config for sparsecore embedding.Fdisable_table_stacking@max_ids_per_chip_per_sampleNmax_ids_per_tablemax_unique_ids_per_tableallow_id_droppingTinitialize_tables_on_host enable_fast_table_initialization)__name__ __module__ __qualname____doc__r9bool__annotations__r;intr<rrstrr=r>r?r@V/home/dcms/DCMS/lib/python3.12/site-packages/tensorflow/python/tpu/tpu_embedding_v3.pyr8r8Msn)!&$&%'s'04Xd38n-47;HT#s(^4;!T!$(T(+0"D0rJr8cdeZdZdZdedeffd Z d dedeffd Zde jfd Z xZ S) EmbeddingPipeliningContextzISets the _embedding_pipelining attribute on all ops created in the scope.modeenablect|d|_tjt j ||_tj}|r|rtjdd|_ y||_ y)NEmbeddingPipelinigContext)sz`Embedding pipelining requested but summaries are being recorded: Disabling embedding pipelining.F) super__init___namer AttrValuer/as_bytes_moder"is_recording_summariesr info_enable)selfrNrOrecording_summaries __class__s rKrTz#EmbeddingPipeliningContext.__init__^sf G,DJ))FOOD,ABDJ(??A % ll -dldlrJ context_def export_scopec&t|||yN)rSto_control_flow_context_def)r\r_r`r^s rKrcz6EmbeddingPipeliningContext.to_control_flow_context_defls  G' \BrJopc|jr |jt|j|jr|jj |yyrb)r[ _set_attr_PIPELINE_ATTRIBUTErX_outer_contextAddOp)r\rds rKriz EmbeddingPipeliningContext.AddOpss> ||ll& 3  #rJrb) rArBrCrDrHrErTrrcr Operationri __classcell__r^s@rKrMrM[sFQ 3  37CC,/C$cmm$rJrMc eZdZdZdej dedededef fd Zde e jd e e jd efd ZxZS) TPUEmbeddingShardedSaveablezIDefines how to save and restore a shard of TPUEmbedding sharded variable.variableshard_id num_shards shard_dimnamec||_||_dgt|jz}||j|z||<|jj }|||z||<t j j||||jj }tj|j|j||j|j} t |=|j| g|y)z!Init TPUEmbeddingShardedSaveable.r) full_name full_shape var_offset var_shape)r slice_specrsdtyperN) _shard_id _variablelenshapeas_list tf_variablesVariable SaveSliceInfor.SaveSpec read_valuespecrzrrSrT) r\rorprqrrrsrw fullshapesave_slice_inforr^s rKrTz$TPUEmbeddingShardedSaveable.__init__~sDNDNs8>>**J$x~~i'@@Jy&&(I% )(<._saveable_factorysdit{{#jJ'  ( ' H%     rJ) _common_namer-VARIABLE_VALUE_KEY)r\rs` rK _gather_saveables_for_checkpointz}|j"rJ||_2tAjBtD}| D]`}|j"}|j>|jk(sJ||jF|j<||jHjK|b|jMD]^\}}|dj*|jN|<||jP|<tjRd d ||D]}|j"}tjRd d |j|jT|jV||jT|z|z|jVf|jX|j<|jZd|_|jZd |_|jZd|j\dz |j^|j<|jZd |j\d z |j`|j<atjdt|jP|jPDcic]}|dc}|_1|D]\}} |jX| jjd}|jb||jd|<|jb|xxtjtj| jz cc<|cSycc}wcc}wcc}w)z3Stack tables with the same table dim and optimizer.z&Number of tables before stacking is %dFzTable stacking is disabled.)rsparse_cores_per_partitionr9rc|jSrb)rs)ts rKzA_stack_tables_with_same_table_dim_and_optimizer..qs qvvrJ)key)usedforsecurity_xxtpuv3internal_) table_name table_height table_widthgroupoutput_sampleszStacked table name: %sz" Table %s: offset %d, rotation %dz&Number of tables after stacking is %d.N)3r rZr}r9warnr&SparseCoreLayoutStackerrrsrr functoolsreduceoperatormul output_shapesortedlayouthash optimizerreprquantization_confighashlibsha1encode hexdigestAddTablevocabulary_sizedim GetLayoutstablesr collections defaultdictlistrstacked_table_nameritemsrrvlogsparse_core_shard_row_offsetsparse_core_shard_rotationrunsharded_padded_shapeunsharded_shaperrrr)rrrrrr9stackerrRrtable_to_num_samples_r sorted_tables key_tuplekey_strrrtables_by_stack stack_namerr feature_pathrs rK/_stack_tables_with_same_table_dim_and_optimizerr(Rs ,,7\9JK !9PP LL./ & > >#!53 ' o!:FG5::u,G!5ABE%**a-BB!m ja++, 0@0@ g**1, <-=>M  \\  ! ** +  ,, O " " $!  )+ "G+zz.. / ;  / H$$&--##F$5$56e el"--d3O?||f   %** ,, ,&,a #f//077> ? .335  F+1!9+H+HaZ(.4a + ll1. ; % 0 JJ  / /  - -     / / " #  - - 7 '' 3!' = =a @11!4  ) )! ,v/E/Ea/H H  +  ) )! ,v/E/Ea/H H ""5::.-  > LL0 A % %& )*(A(A$ A A"/  g::7==;M;MN 231H1H 2a  .01Y5E5E ,,,,61  H[m HB@s-S" S'; S,z)tpu.experimental.embedding.TPUEmbeddingV2ceZdZdZdZdZ dNdeeje fde ejde de e ffd Zded eeefd eed ed ed ej(f dZdedej(d eeeffdZdeed ed eeefdej2d eeeeeej8fff dZdZed eeje jBffdZ"ed eejee jBffdZ#ed eee$jJffdZ&ed eejeee jBfffdZ'deejded eee jBffdZ(d eeeee jBfffdZ)dOdZ*dZ+dZ,d Z-d!e.d"eee/ffd#Z0 dPd$e.d%e e.d e1e.eee/fffd&Z2d'Z3e4 dQd$e.deee fd(ed)ede ejde e d e1e.e.ffd*Z5 dRd$e.d%e e.d+e ed e.fd,Z6d-eee.fd e.fd.Z7d-e1eee/feefd e1e.eee/fffd/Z8 dPd$e.d%e e.d e1e.eee/fffd0Z9 dPd1ed2ed3ed)ed4ed5eee.fd6eee1eeeffd7eeefd8eeefd9e.d:e.d;e e.d e.fd<Z:e4d=eej8e;jxe=j|fd>e ej8dejd?ed@edAedBed)ed4edCed e.fdDZ?e4d:e.d;e.d9e.d5eee.fd6eee1eeeffd8eeefd)edEeeefd4ed eee.ffdFZ@d1edGeee.fd5eee.fd)ed e1eee.feej8ff dHZAd1ed2ed3edIeee.fd5eee.fd7eeefd)ed e.fdJZBdKeeej8ffdLZCdMZDxZES)STPUEmbeddingV2zKThe TPUEmbedding mid level API running on TPU with sparse core accelerator.rr #pipeline_execution_with_tensor_corercdt|t||tj|_t |j tjtjfs$tdj|j |jD]H}|jj|jj 1t!d|j"d|j j$j&j(|_|j*dk(rt-j.dd|_|j j0|j*z|_t5j6|j8|_|0t=|_t-j.d|j>n||_tA|j|j:|j j0|j*|j>|_!|jBjD|_#|jBjH|_%|jBjL|_'|jBjP|_)|jBjT|_+|jBjX|_-|jBj\|_/|jBj`|_1|j>jd|_2d |_3i|_4i|_5|jm||_7y) aCreates the TPUEmbeddingV2 mid level API object. Args: feature_config: A nested structure of `tf.tpu.experimental.embedding.FeatureConfig` configs. optimizer: An instance of one of `tf.tpu.experimental.embedding.SGD`, `tf.tpu.experimental.embedding.Adagrad` or `tf.tpu.experimental.embedding.Adam`. When not created under TPUStrategy may be set to None to avoid the creation of the optimizer slot variables, useful for optimizing memory consumption when exporting the model for serving where slot variables aren't needed. pipeline_execution_with_tensor_core: If True, the TPU embedding computations will overlap with the TensorCore computations (and hence will be one step old). Set to True for improved performance. sparse_core_embedding_config: Configs for sparse core embedding including settings for table stacking, input feature static buffer size etc. Raises: ValueError: If optimizer is not one of tf.tpu.experimental.embedding.(SGD, Adam or Adagrad) or None when created under a TPUStrategy. RuntimeError: If not created under TPUStrategy. z@TPUEmbeddingV2 should be created under TPUStrategy but found {}.Nzpweight_decay_factor and multiply_weight_decay_factor_by_learning_rate are not supported yet. But found in table z setting.rzYNo embedding devices per chip info is found. Using 4 as the default value for SparseCore.zBSparseCoreEmbeddingConfig is not provided. Using default values %sr:)8rSrTrr get_strategy _strategy isinstancer TPUStrategy TPUStrategyV2 RuntimeErrorformat _table_configr weight_decay_factor-multiply_weight_decay_factor_by_learning_raterrsrtpu_hardware_featurenum_embedding_devices_per_chip_num_sc_per_chipr warningnum_replicas_in_sync_num_sc_shardsr0 flatten_with_joined_string_paths_feature_config_flat_featuresr8_sparse_core_embedding_configr(_sr_table_name_to_tabler_stacked_table_to_tablesr_table_to_padding_columnsr_table_to_padding_rowsr_table_to_stacked_table_offsetr_table_to_sample_countr_feature_to_sample_offsetr_quantization_configsr;max_minibatches_per_sc!_table_to_max_ids_per_sparse_core(_table_to_max_unique_ids_per_sparse_core4_update_sparse_core_buffer_size_after_table_stacking _pipelining)r\rr r,rrr^s rKrTzTPUEmbeddingV2.__init__s@ G*>:IF#002DN  11<3M3MN   L 6$.. !  ##   // - - 9 __ J J" ((- |9 >    44SS  ! oo # d ++d.C.CC  ?? D$++D+Fd( oo N  , , ,Hd(=   ++  ** DG!% ; ;D$(GG$C$CD!%)WW%E%ED""&''"?"?D*.''*O*OD'"&''"?"?D%)WW%E%ED"!%!=!=D **FF $#%D-/D*46D1==?:DrJrpartition_shapepartition_offsettotal_vocab_sizesc_idxrc|j|dz|z|jz}|j|dz|z||zz}tj||jg|dgSr)rr;r- ShardInfor)r\rrQrRrSrT sc_shard_sizesc_shard_offsets rK_compute_sc_shard_infoz%TPUEmbeddingV2._compute_sc_shard_infoUs  !       !  1      O >>=%))46J KKrJr shard_infoc*|jjj}|j\}}||z}|jj |d}|j d|jdz}||z |jz}||jz} || fS)Nr) r0r _tpu_devicesr~rCroffsetr>r;) r\rrZ tpu_devices num_replicasnum_cores_per_replica num_devicesshift shard_indexnum_scs rK _compute_sc_shard_idx_and_offsetz/TPUEmbeddingV2._compute_sc_shard_idx_and_offsetms ..))66K*5*;*;'L'!66K GG 1 1* =a @E##A&**:*:1*==K&$*=*==K 400 0F  rJstacked_tablesrzc pi}|jjj}|j\}}||z} |d| z|df}dgt |z} t |D]} t |D]} t jj|| | jddj} | |z| z|dz}g||<t |jD]B}i}||ji|D]#}tj|j}d|j vxsd|j"v}|j$j&sX|r[|| d<|j)||| ||}|j+|j,|\}}t/j0| 5|j,|vr6|j|j2|j4f|||j,<||j,|d|ddf}|||||j,<ddd&E|S#1swY!->-+ejj)'uzz2;3L 3La3OPEM!"23A6uzzBNN5! N ' N3N4Nl NNs "A3H+ +H5c:|jD] }|jj||jjvr0tjd||j |j |<n&|jj||j |<|jj||jjvr0tjd||j|j|<|jj||j|<y)z8Update the sparse core buffer size after table stacking.NzjTable %s is not found in max_ids_per_table provided by SparseCoreEmbeddingConfig. Using default value 256.zqTable %s is not found in max_unique_ids_per_table provided by SparseCoreEmbeddingConfig. Using default value 256.) rErBr<r r<DEFAULT_MAX_IDS_PER_TABLErMr= DEFAULT_MAX_UNIQUE_IDS_PER_TABLErN)r\rs rKrOzCTPUEmbeddingV2._update_sparse_core_buffer_size_after_table_stackings#33&  , , > > F 33EEF  C   * * ..z:  . . @ @ L ..z:  , , E E M 33LLM G     1 1 55jA  . . G G  55jAE& rJc|j|jDcic]}||j|dc}Scc}w);Returns a dict of embedding tables, keyed by `TableConfig`. parameters) _maybe_buildrE _variables)r\rs rKembedding_tableszTPUEmbeddingV2.embedding_tablessL  #'"?"?   DOO,>? MM  s<c J|jg}|jjjD]}|j ||j j Dcic]'\}}||Dcgc]}||j|fc})}}}}|Scc}wcc}}}w)r)rr0rr]extendrrr)r\ordered_devicesdevicesrsvarr table_shardss rKembedding_table_shardsz%TPUEmbeddingV2.embedding_table_shardss  O>>**77&W%&..446  D# AP 7=VS))&1 2  L   s' B4BBBc.|jjS)zReturns how the tables are laid out in the variables. The SparseCoreTableLayout describes how a table is stored in its internal state. You need this only if you need to pull apart the internal state. )rCrrs rKembedding_layoutsz TPUEmbeddingV2.embedding_layoutss 77 " ""rJc:|j|jS)zGReturns a dict of variables, keyed by `TableConfig`, then by slot name.)rrrs rKr$zTPUEmbeddingV2.variabless  ??rJrcj tDcgc]}|jc} dj} |ftjdj }j   d  fd }fd fd tjtj5 |}ddd fd}|Dtjtj5|j|} dddni}  d<| Scc}w#1swYcxYw#1swY!xYw) zBCreate all variables including table variables and slot variables.rNc   jjr2dj|jddjf|Sg}t j D]}D]}tj|j}d|jvxsd|jv}|r j||j|j|}|s rD|j |jd|vr# |jd||j} n j|j|\} } |j|j|jf|| d| ddf} nj|j|j|jf||} n?|j|j|dzz j z|jf|} |j!| t#j$|dS)NrrlrZ)r~rzrZaxis)rBr@rsr~rrr;r1rrrtrurYr^rsrfrrrconcat)r~rzrZconcat_tensorsr{rr}r~rrrdrhost_table_tensorsr\rgrSs rKtable_initialize_fnz=TPUEmbeddingV2._create_variables..table_initialize_fn8sJ  + + L La ,,##A&q(9(=(=>-  n T**+5*!$1 *E..u/@/@A(hmm+5!4!44   77  !!  M"$jj$6z7H7H7K$LQ$OO-j.?.?.BCAFJJ 99 M* \ !,, 00%)).gettersA '' ^5m  " "% rJc2j||dS)NF)rsrsr~rzrr) _add_variable_with_custom_getter)rsrsrr\variable_dtypers rKvariable_creatorz:TPUEmbeddingV2._create_variables..variable_creators/ 2 2! 3rJc"dz|z|S)N/rI)rsrsrrs rK slot_creatorz6TPUEmbeddingV2._create_variables..slot_creators 036={ KKrJrrb) sumrrrfloat32r rr#variable_creator_scopemake_sharded_variable_creatorr0 _create_slots)r\rgrr table_dimr rrr slot_varsrrrSrrrs``` @@@@@@rK_create_variablesz TPUEmbeddingV2._create_variables&sH ~NeE11NOq!%%I& 2N^^Nq!++I55(..E6E6N    . .%dnn5 M$$68KLjM L  0 0 ' 7 F++J E FF i(Il O^MMFFsD1 D3D)D&)D2czi}|jjD]\}}|j||||<|S)zCreate variables for TPU embeddings. Returns: A dict of dicts. The outer dict is keyed by the table names and the inner dicts are keyed by 'parameters' and the slot variable names. )r)rErr)r\r$rrs rK_create_variables_and_slotsz*TPUEmbeddingV2._create_variables_and_slotssVI&*&C&C&I&I&K"F&*&<&< %7'='i"# rJc tj}|jj|jj t jdt|jtjd5|jtjtj|j!t"j$tj&dddy#1swYyxYw)Nz(Saving sparse core layouts for %s tablesz/cpu:0rz)rSparseCoreTableLayoutsrrrrr rZr}rr_track_trackabler*SparseCoreLayoutsTrackablerconstantSerializeToStringrstring!SPARSECORE_LAYOUTS_CHECKPOINT_KEY)r\layoutss rK_track_restore_info_for_cpuz*TPUEmbeddingV2._track_restore_info_for_cpus$;;=G NN$00779: LL2C4G H   ; ;""++-V]] ! B B s A%C44C=c@tjj|Srb)r)TpuEmbeddingV3CheckpointAdaptercreate_from_checkpoint)r\paths rK_checkpoint_adapterz"TPUEmbeddingV2._checkpoint_adapters . N N e e  rJc|js.tj5|jdddyy#1swYyxYwrb)_builtr init_scopebuildrs rKrzTPUEmbeddingV2._maybe_builds= ;; >>    s <Acv|jry|j|_|jd|_y)z8Create variables and slots variables for TPU embeddings.NT)rrrrrs rKrzTPUEmbeddingV2.builds/ {{ 668DO$$&DKrJ gradientspreserved_outputsc |js tdtj|j||j |}t t|j}|jtjfd}t|jdj}|jD]}||}||}|j |d} |j|dj"} t%| t&j(rt+j,|j.|j0|j2|j4||| j6| j9||j:||j<|| } | j?| t%| t&j@r|j |d} t+jB|j.|j0|j2|j4||| j6| j9| j9||j:||j<|| \} } | j?| | j?| t%| t&jDrT|j |d} |j |d}t+jFd&id |j.d |j0d |j2d |j4d |d|| j6d| j9d| j9d|j9d|d| jHd| jJd| jLd| jNd| jPd|j:|d|j<|d|\} } }|j?|| j?| | j?| 9t%| t&jRrG|j |d}|j |d}t+jTd&id |j.d |j0d |j2d |j4d |d|| j6d| j9d|j9d|j9d|d| jVd| jXd| jZd| jPd|j:|d|j<|d|\} }}|j?||j?|| j?| t%| t&j\rT|j |d} |j |d}t+j^d&id |j.d |j0d |j2d |j4d |d|| j6d| j9d| j9d|j9d|d| j`d | jbd!| jdd"| jfd#| jhd|j:|d|j<|d|\}}}|j?|| j?|| j?| tkd$|jmy%)'apApplies the gradient update to the embedding tables. If a gradient of `None` is passed in any position of the nested structure, then a gradient update with a zero gradient is applied for that feature. For optimizers like SGD or Adagrad, this is the same as applying no update at all. For lazy Adam and other sparsely applied optimizers with decay, ensure you understand the effect of applying a zero gradient. Args: gradients: A nested structure of gradients, with structure matching the `feature_config` passed to this object. preserved_outputs: A dicts of PartitionedCsrFormatTensor, coming from the second output of the embedding lookup call. Raises: RuntimeError: if not built. ValueError: If a non-`tf.Tensor` non-`None` gradient is passed in, or a `tf.Tensor` of the incorrect shape is passed in. Also if the size of any sequence in `gradients` does not match corresponding sequence in `feature_config`. TypeError: If the type of any sequence in `gradients` does not match corresponding sequence in `feature_config`. zapply_gradients called on unbuilt TPUEmbeddingV2 object. Please either call the embedding lookup method first or manually call the build method.c~t|rtj||}tj||S)Nr)callabler!castrconvert_to_tensor)paramrzs rK _wrap_paramz3TPUEmbeddingV2.apply_gradients.._wrap_params. % egU3  " "5 66rJrr) rrrractivation_gradients learning_rateembedding_tablermax_ids_per_sparse_coremax_unique_ids_per_sparse_corer accumulators) rrrrrrr accumulatorrrrrmomentarrrrrrrrr use_nesterovexponentbeta1beta2epsilonrrr velocitiesvelocityuse_sum_inside_sqrtlinearslinear multiply_linear_by_learning_ratebetalearning_rate_powerl1_regularization_strengthl2_regularization_strengthz+Unsupported optimizer in minibatching mode.NrI)7rr4r0assert_same_structurer@_stack_gradientsrM_PIPELINE_MODE_BACKWARDrPEnterrrrrrrEr$r r1r(SGDxla_opsHHi I599 :")"f"f+880BB/@@+88!)%i&=&=>!,,.5]$($J$J%,0+X+X,"# "  ,- i!7!?!? @~~j1.A  T T/<<"4"F"F!3!D!D/<<%-))*A*AB % 0 0 2(3359a(,(N(N)04/\/\0&!  5!4( /0 ,- i!7!G!G H~~j1.A ..,Y7  ] ] /<< "4"F"F "4!D!D 0<<   &.   *)*A*AB  !& 0 0 2 )335  **, :b '33 #++  ((  oo "))  )-(N(N)! &04/\/\0' ,&-  F!4o4 '/0 ,- i!7!>*-l;  Q Q /<< "4"F"F "4!D!D 0<<   &.   *)*A*AB  !& 0 0 2  **, ",,. :b %.$=$=  &&  && ")) )-(N(N) $04/\/\0% *&+  C2B2 ()' ,- i!7!TPUEmbeddingV2._raise_error_for_incorrect_control_flow_contexts  ! ! #EJ    + + -c O c?>> ?*  O  e]D1e   %%''J   "6#"7"7"95A   rJ num_tpu_chipsnum_sc_per_chipcXt|}g}tj|D]} |j| j|D]} | j || _tj |} t|| |||} tj|} tj| dgt| z| | j| j| j|| j||z }| jDcic]}|d}}| jDcic]}|d}}| jD]}| j|dj}t!| j|D cgc]} | j"c} }t%|D]}||d|}||d|}||d|}||d}||d}t'j(|||||||||| \}}t+j,|||||<t+j,|||||<||fScc}wcc}wcc} w) zAComputes the max_ids/unique ids settings from the input features.N) flat_inputs flat_weightsrrrrrstacked_table_to_sample_count num_sc_shardsrrr\r.) row_ids_list col_ids_list gains_listsample_count_listcol_offset_list num_replicatable_vocab_size feature_widthrr)rr0rrrr r?r(r*5_preprocess_inputs_and_weights_to_list_of_coo_tensorsr}rrrrrrrrr.get_stats_from_list_of_sparse_core_coo_tensorsr!maximum)clsr rrrr rcopy_feature_configrrrrrRrtable_to_list_of_coosr table_to_max_ids_per_sparse_core'table_to_max_unique_ids_per_sparse_corer)rSr{r"r#r$r%r&rrs rKcompute_sparse_core_statsz(TPUEmbeddingV2.compute_sparse_core_statss0?L<< 34)'--()$  #$99:MNM7$  A,,x(K LL##k"22'$%$=$=*+*I*I%&%?%?+*+*A*A)M9 M )*(A(A($ A ($()*(A(A/$ A /+///% // ;A>BBm00<    _% !,Z8;A> ,Z8;A> *:6q9!< 1*=a@/ ;A>  B B%%!/+%-'+!  # *8@7G7G ,Z 8 #8 (4?G>N>N 3J ? *? / ;7 % N )/ [(/s; H H""H' rcTtj|j|tj|}dgt |z}|5tj|j|tj|}|j }|rt j|j|jj|j|j|j|j|j|j |j"|j$|j&|| S||jj(j*}t-j.t1j2|dd5|j|jj|j|j|j|j|j|j |j"|j$|j&|| cdddSt4j6j9|}|j:dk7rt=dj?|t-j.t1j2|5|j|jj|j|j|j|j|j|j |j"|j$|j&|| cdddS#1swYyxYw#1swYyxYw)z#Preprocessing the features on host.N) r=r;rLrr rrrrrrrrTPUz$Non-TPU device {} passed to enqueue.) r0rr@rr}rr+outside_compilation_preprocess_featuresr0r=r;rLr;r>rErHrIrJrArr]rrrget_host_for_devicermrnrorjrr5) r\r r rrrin_tpu_contextr_ device_specs rKenqueuezTPUEmbeddingV2.enqueue8s t33X>,,x(K6C ,,L   !5!5w?\\'*lIIKN  0 0  # ##~~BB&*&F&F!%!#> 11--$($A$A*.*M*M"&"="=%)%C%C--#%)    ((44V ((!%!D!D(,(H(H#'#>#> 11--$($A$A*.*M*M"&"="=%)%C%C--#%)    )  (  s8B L=B LLL'partitioned_tensorsc ji}|D]}||d}||d}||d}tj|j|j|j|j g||||g\}}} } tj ||| | g\}}} } t||| | |j|j||<|S)zCopy tensors to device.rrr\r) rtpu_copy_with_dynamic_shaperrrr'tpu_annotate_tensors_with_dynamic_shaperrr) r\r;partitioned_device_tensorsrrrow_pointers_unpadded_sizeids_unpadded_sizerrrrs rK_copy_tensors_to_devicez&TPUEmbeddingV2._copy_tensors_to_devices "$)) .z:1=#6z#B1#E -j9!<  - -$11$66$55$11 -###  Fl%'7&  9 9 ;  Fl%'70J#-+#)66 II 0 ,A)T &%rJc0tt|j}|jt j |j |}i}t|jdj}|jD]}||}|j|d}|j|}t|tstdt!|dt#j$|j&|j(|j*|j,|j.||||r |j0nd|r |j2nd|r |j4nd|j6||j8|| } | ||<|j;|j=|}||fS)zPerform embedding lookup.)r;rrz*Expect PartitionedCsrFormatTensor but get .) rrrr input_sizerrquantization_config_lowquantization_config_highquantization_config_num_bucketsrrr)rM_PIPELINE_MODE_FORWARDrPrr+r5rBrrrrEr$rKr1rrtyper/xla_sparse_dense_matmul_with_static_buffer_sizerrrrrIlowerupper num_bucketsrMrNr_unstack_activations) r\r;r activationsrrrrr  activations rKdequeuezTPUEmbeddingV2.dequeues) 0 0G MMO)== $$/ K04""$00 1 1-33$+ .z:nnZ(6e 66zB *,F G '() ,  JJ)66.@@->>)6600<3[+>!''A,?!''A2E!--!"&"H"H# *.)V)V*  /j4!+k*I$+L LLN++K8K , --rJc|js|jtt|j}|j |j ||}|j|j|}|S)aPerform embedding lookup on the input feature. Args: features: A nested structure of `tf.Tensor`s, `tf.SparseTensor`s or `tf.RaggedTensor`s, with the same structure as `feature_config`. Inputs will be downcast to `tf.int32`. Only one type out of `tf.SparseTensor` or `tf.RaggedTensor` is supported per call. weights: If not `None`, a nested structure of `tf.Tensor`s, `tf.SparseTensor`s or `tf.RaggedTensor`s, matching the above, except that the tensors should be of float type (and they will be downcast to `tf.float32`). For `tf.SparseTensor`s we assume the `indices` are the same for the parallel entries from `features` and similarly for `tf.RaggedTensor`s we assume the row_splits are the same. Raises: ValueError: If the input feature is not one of the Tensor, SparseTensor or RaggedTensor type. TypeError: If the type of any sequence in `features` does not match corresponding sequence in `feature_config`. Similarly for `weights`, if not `None`. Returns: packed_activations: Embedding lookup results packed as the same sequence of the input feature. packed_output: A dict of PartitionedCsrFormatTensors. ) rrrMrIrPrr:rrR)r\r r rr;results rKr zTPUEmbeddingV2.embedding_lookupse: ;; ( 0 0G MMO,,x9 LLN \\- .F MrJr=r;rLr rrrrrrrc tj| | | ||| ||| } |j|| ||}|j|||||||}|S)z Function to preprocess features.)r*r*_sort_list_of_coo_tensors+_get_csr_wrapped_coo_from_sorted_coo_tensor)r\r=r;rLrr rrrrrrrr/table_to_sorted_coo_tensortable_to_csr_format_tensors rKr6z#TPUEmbeddingV2._preprocess_features(s$ LL    # ) $  !  "&!?!? " 88 ' " & # !   &%rJ input_featureweight row_offset col_offset col_shift vocab_sizestacked_table_sample_countc &tjtj|j} t |t jr*tj|dg}|&tj|tj}nIt |t jrtj|dg}ntdt|tj tj"dtj$t'j(|tj$t'j(|tj| |j*j,||||| |  \} } }nt |t.j0r&|0tj|j2tj}n>t |t.j0r |j2}ntdt|tj t'j(|j4tj$t'j(|j2tj$t'j(|tj| |j*j,||||| |  \} } }nXt |t6j8r%|s0tj|j2tj}n>t |t6j8r |j2}ntdt|tj t'j(|j:tj$t'j(|j2tj$t'j(|tj| |j*j,||||| |  \} } }ntdt|d | | || fS) z8Convert any of the expected input types to a COO format.rz(Expect weight to be Tensor type but got )r) indices_or_row_splitsrr rcombinerrr\r]r^r r`z.Expect weight to be SparseTensor type but got z.Expect weight to be RaggedTensor type but got zInput of unknown type zU. Please only pass Tensor, SparseTensor or RaggedTensor as input to embedding lookup.)rrrrrr1rrrreshape ones_likerrrrJr*convert_to_list_of_sparse_core_coo_tensorszerosint32r!rrrdr SparseTensorrindicesr% RaggedTensor row_splits)r-rZr[rr\r]r^r_rr r`rr"r#r$s rK-_convert_input_feature_to_list_of_coo_tensorsz $$]%9%9P fm88 9 $$]%9%9P fm88 9,V,lG+H --  +'j*i$A#L +L9j  F Fmm++( ;lL*l_%F!j)!,Q/66|AGj)!,Q/66|AGj)!,Q/66z!}EFJ'*11 / )J'*11*==>> ! S sD5r/c~|Dcic] }|ggggf }}|D]}||dj}t||Dcgc]}|jc}} t|D]} ||d| } ||d| } ||d| } ||d}||d}t j | | | |||| |||j ||j|| \}}}}||dj|||dj|||dj|||dj||Scc}wcc}w)z Sort the coo tensors by replica.rrr\r!r.) r"r#r$r%r&r'r(r)rrrr) rrrrr$sort_list_of_sparse_core_coo_tensorsrMrNr)r\r=r/rrrrXr)rrSr{r"r#r$r%r&sorted_row_idssorted_col_idsr id_countss rKrVz(TPUEmbeddingV2._sort_list_of_coo_tensorss8O")3 RR$$"".)D -j9!<@@m-DZ-P$)%   _%"D!,Z8;A> ,Z8;A> *:6q9!< 1*=a@/ ;A>  8 8%%!/+,-'+$($J$J%,0+X+X,"!     ( #:.q188H":.q188H":.q188F":.q188CE"D)DV &%]"s D5D: rXci}|D]} || \} } } } || dj}t|| Dcgc]}|jc}}tj| | | | t j dtj|| |z|||||| |jj \}}}}}}}t|||||| |||f|| <|Scc}w)z6Get csr wrapped coo tensor from the sorted coo tensor.rr) sorted_row_ids_listsorted_col_ids_listsorted_gains_listid_counts_listsplitssample_count_per_scr'rLr;r(r)rr>r) rrrr-convert_to_sparse_core_csr_wrapped_coo_tensorrrrint64rBr>r)r\r=r;rLrXrrrrYrrxryrzr{r)rrSrrrrr@rArs rKrWz:TPUEmbeddingV2._get_csr_wrapped_coo_from_sorted_coo_tensor5s:"$-3 %Z 0     .j9!<@@m-DZ-P$)%     ? ?11-'%%v|| 4J? *!7&A+% >>PP!      $  2( %' 1/'0<7_   %  0 ,Q3h &%SsC rPc g}|jDcic]}|d}}|jD]=}||j}tj|||jd|dg||<?|j D]=\}}t jtj|j}|j|jjd}|j|jj} tj||d||dg|j||jz|jj | z g} tj| t#|j|jj | z gz} |j%| ||xx||jzz cc<@t'j(|j*|Scc}w)z;Untack the incoming per table activations into per feature.rrb)rEr~rrer;rArrrrrrHrrsrFslicerrrr0rr@) r\rPflattened_activationsrtable_to_current_offsetactivation_shaper!rr extra_colsrQs rKrOz#TPUEmbeddingV2._unstack_activationsws (,(E(E$ A 33 $Z066 ) 1 1 j !  "&6r&: ;!k* )) 7%%hllG4H4HIl66w}}7I7IJ1Mj11'--2D2DEj?? j ! %j11 5##d333mm*, j$$  w## $ (9(9J(F'G Gj"":.j) $// /)'.  !5!57L MMAs Gc P|jDcic] }|ggggg }}tj|}t||jD]\}\}}t j tj|j}|5t|tjstdt|d|d|Qtj d|t#j$||j&j(ft*j,}|j.|j&j0d}|j2|j&j0} t#j4|d|j&j(| z g}| dk7rCt#j6|ddgd| gg}|j9||j&j(g||j:z} t=|j:D]I} ||| j?t#j@|| | zdg| |j&j(gK|D]X}t#jBt=|j:D cgc]} t#jB||| d!c} d||<Z|Scc}wcc} w) z4Stack the incoming gradients to per table gradients.zfound non-tensor type: z at path rDz|No gradient passed for feature %s, sending zero gradient. This may not be correct behavior for certain optimizers like Adam.rrrbr)"rEr0rrprArrrrrr1rrrrJr r<rrhrrrrrHrsrFrepad set_shaper;rrrr) r\rrtable_to_gradient_listflattend_gradientsrrrrrper_sc_sample_countr{s rKrzTPUEmbeddingV2._stack_gradientss77  RR$$i0%(D//&* !/4%%hllG4H4HIl  j6==&I%d8n%5YtfA F     (   ?? 7==,, -V^^ 66w}}7I7IJ1Mj11'--2D2DEj"" R**Z78h q==QFQ O+DEL'--*;*;<=(D,A,AAT**+  !z*1-44 OO((!,'MM%%    A* V- +4+;+;T223 5jA!D1M  ,Z( "!qd s J*$J# )NFN)rNrb)r.NN)NN)FrArBrCrDrrr r( FeatureConfigrr _OptimizerrEr8rTrtuplerGrr-rVrYrHrfrDTyperrrrrOrrrrrrrrr$rrrrrrrrr r rr classmethodr2r:rBrRr r6rrjr%rlrnr*rVrWrOrrkrls@rKr*r*sS!%(" @D27JN q;2@@(JKq;0;;<q;,0 q; %--F$G q;fL LS#XLS L  L  L ~~L0.. S#X "F;'FFS#X F \\ F  Cd3 -./ /0 FP( T   " . . 0E0E EF     " . .\5J5J0K KL ( # C'== =># #  (($sL4I4I/I*JJ E1==>EE C&& &' EN Cc<0001 12   LLc#==>L^5944$,SM4 S$s667 7848 ?CJN^^M834^ ^  ^ 0;;< ^%--F$G^ S#X^^F $" I I }I sm I  I V0&S>0& 0&dD. s.. /c 9D.  S$s667 78 D.N59++$,SM+ S$s667 78+t%)3&3&$'3&" 3&  3&  3& $CH~3&&*#uS#s]/C*C%D3&"#s(^3&!%S#X3&3&3&SM3& 3&jd@ --33]5O5O Od@ v}}% d@ -:: d@d@d@d@d@d@d@#&d@ d@d@L6!6!6! 6! $CH~ 6! &*#uS#s]/C*C%D 6!!%S#X6!6!&*#s(^6!6! CH~6!6!p6&6&"#s(^6& $CH~ 6&  6&  T#s(^T&--0 01 6&p@&@&$'@&" @& #'sCx. @& $CH~ @&"#s(^@&@& @&D&Nd3 3E.F&NP="rJr*kwargs.cTdtdtfdtdzfd}t|dtj rd|dj vs|djrd|dj vr|dj d}n|djd}|d||dj jd |d |dj||djfSd|vs|dt|dstd j|d|d|d|d |d||dfS) aExtracts the variable creation attributes from the kwargs. Args: kwargs: a dict of keyword arguments that were passed to a variable creator scope. Returns: A tuple of variable name, shape, dtype, initialization function, restore_uid. r.rNct|ddS)N restore_uid)rrs rKget_restore_uidz.extract_variable_info..get_restore_uids =- 66rJr~rrsrza2Unable to extract initializer function and shape from {}. Please either pass a function that expects a shape and dtype as the initial value for your variable or functools.partial object with the shape and dtype kwargs set. This is needed so that we can initialize the shards of the ShardedVariable locally.) rrrGr1rrkeywordsrtgetfuncrrr5)rrr~s rKextract_variable_inforsn 7Xc3h%77C$J7'):):; (111  % %&)222_%..w7e_%**1-ev ((,,WfWoF$$/445  V  f_- .  @AG ? #A   vww/0  rJrct|tjxs\t|tjxs@t|tj xr$t|j tjS)z-Whether the initial value is from checkpoint.)r1r-CheckpointInitialValueCheckpointInitialValueCallablerrrrs rKis_checkpoint_initial_valuersc ; ;< M4#F#F G ]I$5$5 6   $"E"E rJstrategycBjjfd}|S)zCreate a variable creator which shards across all the tpu device. Args: strategy: a TPUStrategy object. Returns: The sharded variable creator. c d|d<d}j\}}t|d}tj|d}|r'd|jvrd|j vr t dt|\}} } } } tjj| } ||z} | || zdk7rt dj| | | j}||| z||<tj| }d|jv}g}dgt| z}t|D]}t|D]}tj ||5|d||d <||d <|rjt#j$tj&|t)j*|}t-j.|d| |d<||xx||z cc<nt-j.| || |d<|j1||i|d d d t3|t4j6j8d }| |j;| |_|S#1swY#xYw)z%Create a TPUEmbeddingShardedVariable.Tskip_mirrored_creatorrrrZzoWhen a sharded variable is initialized from a checkpoint, shard_info must be in arguments of the init function.zWOnly evenly sharding across devices is currently supported. Got shape {} and {} devicesrrsr~)rZrlN)r~rr1rrrtrurrrrrr5r getargspecr}rrr-rVas_shaperdeepcopyrrrrrVariableAggregationNONE_maybe_initialize_trackable _update_uid) next_creatorrtrrrr`rais_ckpt_init_valuer}rsr~rzunwrapped_initial_valuerrbrQunwrapped_arg_specr~r$rR replica_id logic_core_idrZrTrr_s rK_create_sharded_variablez?make_sharded_variable_creator.._create_sharded_variable7s'+F "#I*5*;*;'L'4VO5LM(()@AH  -  3 3 3  B  f%=D%/    ( ( /E!66K Y+%*  ((.uk(B  mmoO!0!;{!JOI#../FG!%7%<%< ? :"V1ZL1&.+&/ %%o6 ./J'0&7&7'J'F? # Y '?9+E E '&/&7&7'e'F? #   <88 9# : :::*))\==BBDF ((*&f M3 : :s B.II )rr])rrr_s` @rKrr*s%!!..+HT "!rJrb)irDrrrrr rtypingrrrrrrr r r abslr tensorflow.core.frameworkr tensorflow.core.tpu.kernelsrtensorflow.python.checkpointrtensorflow.python.distributerrrrrrrtensorflow.python.frameworkrrrmrrrrrtensorflow.python.opsrrr r!r"r#r$rtensorflow.python.ops.raggedr%tensorflow.python.tpur&r'r(r)r*r+tensorflow.python.tpu.opsr,rtensorflow.python.trackabler-!tensorflow.python.training.savingr.tensorflow.python.utilr/r0r1 tensorflow.python.util.tf_exportr2rgrIrrrr dataclassr8ControlFlowContextrMSaveableObjectrnrrlegacy_saveable_nameTPUVariableMixinDistributedVariabler namedtuplerrrGr(TPUEmbeddingBaser*rHrrrErStrategyrrIrJrKrsF XXX4>847513/43;.+5.4+2;*00;6<48E81<,=)'-6."$%00 &22 +>> ABd# 1 1$C 1$!1!D!D$@./"@"@.b1 :::,&%%b)k!;!;k*k^4[33   =&IM E ;'E E#}"456E E  E #++D"E E  E R 67@"'88@"8@"J08 8 sCx&,,S(98C=H8v s t W"%%W" c3hW"rJ