AVhޓdZddlZddlZddlmZddlZddlZddlm Z ddlm Z ddlm Z ddlm Z ddlmZdd lmZdd lmZdd lmZdd lmZdd lmZddlmZddlmZddlmZddlmZddlmZddlmZddlmZddlm Z!ddl"m#Z#ddl$m%Z%ddl&m'Z'e'dgGdde(Z)e'dgGdde)Z*e'd gGd!d"e)Z+e'd#gGd$d%e)Z,Gd&d'ejZZ.Gd(d)e!j^Z0e'd*gGd+d,e0e jbZ2d1d-Z3ejhe2e3e2jke%jlejne2 d2d.Z7e%jpejr d3d/e2fd0Z9y)4zShardedVariable class.N)Sequence)composite_tensor) constant_op)dtypes)indexed_slices)ops)tensor)tensor_conversion_registry) tensor_shape) type_spec) array_ops) data_flow_ops) embedding_ops)math_ops)partitioned_variables)resource_variable_ops) variables) save_context)base)saveable_object_util)dispatch) tf_exportz0distribute.experimental.partitioners.Partitioner)v1ceZdZdZddZy) PartitionerakPartitioner base class: all partitiners inherit from this class. Partitioners should implement a `__call__` method with the following signature: ```python def __call__(self, shape, dtype, axis=0): # Partitions the given `shape` and returns the partition results. # See docstring of `__call__` method for the format of partition results. ``` ct)aPartitions the given `shape` and returns the partition results. Examples of a partitioner that allocates a fixed number of shards: ```python partitioner = FixedShardsPartitioner(num_shards=2) partitions = partitioner(tf.TensorShape([10, 3], tf.float32), axis=0) print(partitions) # [2, 0] ``` Args: shape: a `tf.TensorShape`, the shape to partition. dtype: a `tf.dtypes.Dtype` indicating the type of the partition value. axis: The axis to partition along. Default: outermost axis. Returns: A list of integers representing the number of partitions on each axis, where i-th value corresponds to i-th axis. )NotImplementedErrorselfshapedtypeaxiss ]/home/dcms/DCMS/lib/python3.12/site-packages/tensorflow/python/distribute/sharded_variable.py__call__zPartitioner.__call__<s ( Nr)__name__ __module__ __qualname____doc__r$r%r#rr.s  r%rz;distribute.experimental.partitioners.FixedShardsPartitionerceZdZdZdZddZy)FixedShardsPartitioneraPartitioner that allocates a fixed number of shards. Examples: >>> # standalone usage: >>> partitioner = FixedShardsPartitioner(num_shards=2) >>> partitions = partitioner(tf.TensorShape([10, 3]), tf.float32) >>> [2, 1] >>> >>> # use in ParameterServerStrategy >>> # strategy = tf.distribute.experimental.ParameterServerStrategy( >>> # cluster_resolver=cluster_resolver, variable_partitioner=partitioner) c||_y)zoCreates a new `FixedShardsPartitioner`. Args: num_shards: `int`, number of shards to partition. N) _num_shards)r num_shardss r#__init__zFixedShardsPartitioner.__init__cs "Dr%c~dgt|z}t|j|j|j||<|S)N)lenminr/dimsvalue)rr r!r"results r#r$zFixedShardsPartitioner.__call__ks? S3u: Ft''D)9)?)?@F4L Mr%Nr&r'r(r)r*r1r$r+r%r#r-r-Ss "r%r-z7distribute.experimental.partitioners.MinSizePartitionerc"eZdZdZ ddZddZy)MinSizePartitionera1Partitioner that allocates a minimum size per shard. This partitioner ensures each shard has at least `min_shard_bytes`, and tries to allocate as many shards as possible, i.e., keeping shard size as small as possible. The maximum number of such shards (upper bound) is given by `max_shards`. Examples: >>> partitioner = MinSizePartitioner(min_shard_bytes=4, max_shards=2) >>> partitions = partitioner(tf.TensorShape([6, 1]), tf.float32) >>> [2, 1] >>> partitioner = MinSizePartitioner(min_shard_bytes=4, max_shards=10) >>> partitions = partitioner(tf.TensorShape([6, 1]), tf.float32) >>> [6, 1] >>> >>> # use in ParameterServerStrategy >>> # strategy = tf.distribute.experimental.ParameterServerStrategy( >>> # cluster_resolver=cluster_resolver, variable_partitioner=partitioner) c|dkrtd||dkrtd||dkrtd|||_||_||_y)a@Creates a new `MinSizePartitioner`. Args: min_shard_bytes: Minimum bytes of each shard. Defaults to 256K. max_shards: Upper bound on the number of shards. Defaults to 1. bytes_per_string: If the partition value is of type string, this provides an estimate of how large each string is. r3z7Argument `min_shard_bytes` must be positive. Received: z2Argument `max_shards` must be positive. Received: 8Argument `bytes_per_string` must be positive. Received: N) ValueError_min_shard_bytes _max_shards_bytes_per_string)rmin_shard_bytes max_shardsbytes_per_strings r#r1zMinSizePartitioner.__init__s  &' ) A~  >zl K !  '( * ,D!D-Dr%c~tj|j||j|j||S)N)max_partitionsr"min_slice_sizebytes_per_string_element)rmin_max_variable_partitionerr@r?rArs r#r$zMinSizePartitioner.__call__sD  = ='' ,,!%!7!7    U  r%N)ir3r&r9r+r%r#r;r;rs,GI.:r%r;z7distribute.experimental.partitioners.MaxSizePartitionerc eZdZdZddZddZy)MaxSizePartitionera,Partitioner that keeps shards below `max_shard_bytes`. This partitioner ensures each shard has at most `max_shard_bytes`, and tries to allocate as few shards as possible, i.e., keeping shard size as large as possible. If the partitioner hits the `max_shards` limit, then each shard may end up larger than `max_shard_bytes`. By default `max_shards` equals `None` and no limit on the number of shards is enforced. Examples: >>> partitioner = MaxSizePartitioner(max_shard_bytes=4) >>> partitions = partitioner(tf.TensorShape([6, 1]), tf.float32) >>> [6, 1] >>> partitioner = MaxSizePartitioner(max_shard_bytes=4, max_shards=2) >>> partitions = partitioner(tf.TensorShape([6, 1]), tf.float32) >>> [2, 1] >>> partitioner = MaxSizePartitioner(max_shard_bytes=1024) >>> partitions = partitioner(tf.TensorShape([6, 1]), tf.float32) >>> [1, 1] >>> >>> # use in ParameterServerStrategy >>> # strategy = tf.distribute.experimental.ParameterServerStrategy( >>> # cluster_resolver=cluster_resolver, variable_partitioner=partitioner) Nc|dkrtd||r|dkrtd||dkrtd|||_||_||_y)apCreates a new `MaxSizePartitioner`. Args: max_shard_bytes: The maximum size any given shard is allowed to be. max_shards: The maximum number of shards in `int` created taking precedence over `max_shard_bytes`. bytes_per_string: If the partition value is of type string, this provides an estimate of how large each string is. r3z6Argument `max_shard_bytes` must be positive. Received z1Argument `max_shards` must be positive. Received r=N)r>_max_shard_bytesr@rA)rmax_shard_bytesrCrDs r#r1zMaxSizePartitioner.__init__s  %& ( j1n  =j\ J !  '( *  ,D!D-Dr%c~tj|j|j|j|||S)N)rOrCrHr")rvariable_axis_size_partitionerrNr@rArs r#r$zMaxSizePartitioner.__call__sD  ? ?--##!%!7!7     U  r%)NrJr&r9r+r%r#rLrLs6.:r%rLcVeZdZdZdgZedZdZdZedZ dZ dZ d Z y ) ShardedVariableSpecz+Type specification for a `ShardedVariable`._variable_specsctSNShardedVariablers r#zShardedVariableSpec.s_r%c$t||_yrV)tuplerT)rvariable_specss r#r1zShardedVariableSpec.__init__s 0Dr%c|jSrVrTrYs r# _serializezShardedVariableSpec._serializes   r%c|jSrVr_rYs r#_component_specsz$ShardedVariableSpec._component_specss   r%c,t|jSrV)r\r)rr7s r#_to_componentsz"ShardedVariableSpec._to_componentss  !!r%ct|SrVrW)rrs r#_from_componentsz$ShardedVariableSpec._from_componentss 9 %%r%c|SrVr+)rr7_s r#_castzShardedVariableSpec._cast s Lr%N) r'r(r)r* __slots__property value_typer1r`rbrdrfrir+r%r#rSrSsE3 !)45*1     "&r%rScJeZdZdZd!fd ZdZdZdZedZ edZ edZ ed Z ed Z d"d Zd#d Zd#d ZdZdZd$dZd$dZd$dZd$dZd$dZd$dZd$dZd$dZd%dZdZdZdZedZedZ edZ!edZ"d Z#xZ$S)&ShardedVariableMixinzMixin for ShardedVariable.cXtt| ||_||_t |t r|rtd|Drtd||Dchc]}|j}}t|dkDr&td|Dcgc]}|jc}|d}|j|_ |Dchc](}t|jjdd*}}t|dkDr&td|Dcgc]}|jc}t!d|D}t#j$|g|jjddz|_|D]}t)j*||_t/t|Dcgc].}t/t|jDcgc]}dc}0c}|_t/dt|D]S} |j0| dxx|j0| dz d|| dz jjdzz cc<U|Dcgc]}|j3} }td | Drtd | t5j6|j&|j|j|jdj8t:j<j>t:j@j> |_!ycc}wcc}wcc}wcc}wcc}wcc}wcc}w) aTreats `variables` as shards of a larger Variable. Example: ``` variables = [ tf.Variable(..., shape=(10, 100), dtype=tf.float32), tf.Variable(..., shape=(15, 100), dtype=tf.float32), tf.Variable(..., shape=(5, 100), dtype=tf.float32) ] sharded_variable = ShardedVariableMixin(variables) assert sharded_variable.shape.as_list() == [30, 100] ``` Args: variables: A list of `ResourceVariable`s that comprise this sharded variable. Variables should not be shared between different `ShardedVariableMixin` objects. name: String. Name of this container. Defaults to "ShardedVariable". c3RK|]}t|tj !ywrV) isinstance variables_libVariable.0vs r# z0ShardedVariableMixin.__init__../s LQ:a!7!788Ls%'zSArgument `variables` should be a non-empty list of `variables.Variable`s. Received r3zPAll elements in argument `variables` must have the same dtype. Received dtypes: rNzkAll elements in argument `variables` must have the same shapes except for the first axis. Received shapes: c3hK|]*}t|jjd,yw)rN)intr as_listrts r#rwz0ShardedVariableMixin.__init__..Hs%A!C)!,-As02c3$K|]}|du ywrVr+)ru slice_infos r#rwz0ShardedVariableMixin.__init__..Zs Dj:T ! Dsz`SaveSliceInfo` should not be set for all elements in argument `variables`. `ShardedVariable` will infer `SaveSliceInfo` according to the order of the elements `variables`. Received save slice info )r r!name trainablesynchronization aggregation)"superrnr1 _variables_namerqrany TypeErrorr!r4r>_dtyper\r rzsumr TensorShape_shapeweakrefref_sharded_containerrange _var_offsets_get_save_slice_inforUninitializedVariabler~rrVariableSynchronizationNONEVariableAggregation_saving_variable) rrr}rv var_dtypes first_varhigher_dim_shapes first_dimrhisave_slice_info __class__s r#r1zShardedVariableMixin.__init__s* .0DODJ y( + L)L L  --6K 9  $--a!''-J- :  09:1qww:; =  ! I//DK@II!qww045II !  09:1qww:; =  AyAAI**  ioo--/33DK/$[[.a/;@I:O56E#ioo./0q0D1c)n % 1   AE "1 % !a%(8(>(>(F(F(H(K K :CCAq--/COC DO DD  &'6%6 8 2GGkkkk ZZ//!$..%==BB!55:: D_.;J ; 1Ds6L L ;-LL %L" L L" L'L"c,t|jS)zBReturn an iterable for accessing the underlying sharded variables.)iterrrYs r#__iter__zShardedVariableMixin.__iter__ns   r%c Tt|tsdt|tjr|jt jk(s-t|t jr5|jtk(r"t|}tj||St|ttfs|f}|d}t|tr|j|}g}t|j D]+\}}|| ||f|ddz}|j#||-|j$|j$dkr|j'|s2t)j*g|j,d|j.ddzStj0|dS|t2ur2tj0|j Dcgc]}|| c}dS|tj4urFtj0|j Dcgc] }||dd c}dtj4St|tjr t7d|dkr||j.dz }|dks||j.dk\rt9d |d t;t=|j D]y}|t=|j dz k(s/||j>|dk\s4||j>|dzdksM|j |||j>|dz f|ddzcSycc}wcc}w) aExtracts the specified region as a Tensor from the sharded variable. The API contract is identical to `Tensor.__getitem__`. Assignment to the sliced range is not yet supported. Args: slice_spec: The arguments to __getitem__, specifying the global slicing of the sharded variable. Returns: The appropriate slice of tensor based on `slice_spec`. Raises: IndexError: If a slice index is out of bound. TypeError: If `spec_spec` contains Tensor. )r maskrNr3r&)r!r r"z:ShardedVariable: using Tensor for indexing is not allowed.zShardedVariable: slice index z of dimension 0 out of bounds.) rqbool tensor_libTensorr!rnpndarray_var_to_tensorr boolean_masklistr\slice_decompose_slice_spec enumeraterappendstepreverserconstantrrconcatEllipsisnewaxisr IndexErrorrr4r) r slice_specr sfirst_dim_slice_specsvaluesrvarall_dim_slice_specs r# __getitem__z ShardedVariableMixin.__getitem__rs, :t$ z:#4#4 5  FKK/ z2:: .:3C3Ct3Kd#f  # #6 CC j4- 0=j1 A!U"88;fdoo.1&!S  # / 5a 8:Z^K  --./ 01    ## dkk$QR*@    f1 -- h   &*oo 6s3z? 6Q i    *.// :33z!"~  : Az(( ) H   Q T[[^ Q!t{{1~%+A3.L M  S)* ! DOO$q( ( ""1%a( (Q1B1B1q51I!1L-L#4$$Q'**,z!"~=   # 7 ;s 6 L :L%c <t|jtjsHt|jtjs$t|j tjr t dg}|j |j nd}|dk(r td|j}||dkDrdn|jddz }n|dkr||jdz }|j}||dkDr|jdnd}n|dkr||jdz }|}|dkDrtt|jD]}|j|d}|t|jdz kr|j|dzdn|jd} ||kr*||ttj||z |z zz }|| k\s||k\r|jd||z } t!|| |z } |jt#| | ||Stt|jdz ddD]}|j|d}|t|jdz kr|j|dzdn|jd} || k\r-||ttj| |z dz |z zz }||ks||kr|jd||z } ||k\r||z } nd} |jt#| | ||j%|S)aDecompose a global slice_spec into a list of per-variable slice_spec. `ShardedVariable` only supports first dimension partitioning, thus `slice_spec` must be for first dimension. Args: slice_spec: A python `slice` object that specifies the global slicing. Returns: A list of python `slice` objects or None specifying the local slicing for each component variable. None means no slicing. For example, given component variables: v0 = [0, 1, 2] v1 = [3, 4, 5] v2 = [6, 7, 8, 9] If `slice_spec` is slice(start=None, stop=None, step=None), we will have: v0[returned[0]] = [0, 1, 2] v1[returned[1]] = [3, 4, 5] v2[returned[2]] = [6, 7, 8, 9] If `slice_spec` is slice(start=2, stop=8, step=3), we will have: v0[returned[0]] = [2] v1[returned[1]] = [5] returned[2] == None If `slice_spec` is slice(start=9, stop=3, step=-2), we will have: returned[0] == None v1[returned[1]] = [5] v2[returned[2]] = [9, 7] zsShardedVariable: using Tensor in slice_spec is not allowed. Please file a feature request with the TensorFlow team.Nr3rzslice step cannot be zero)rqstartrrstoprrr>rrr4rrymathceilrr5rr) rrr8 slice_step slice_start slice_endcurr var_startvar_endrends r#rz*ShardedVariableMixin._decompose_slice_specs!@ :##Z%6%67 jooz'8'8 9 jooz'8'8 9  =  F$.OO$?QJQ 2 33""K#aAT[[^a-?k qT[[^#kI%/N$++a.i Q4;;q>!i CA~S**+,7!%%a(+ 3t(()A--   a!e $Q 'Q  ? c$))Y_ ,J"KLL L# '>SI- --  /%Iw')3# --eS*5 67J M+S**+a/R87!%%a(+ 3t(()A--   a!e $Q 'Q  '> c$))Ws]Q->*,L"MNN N# ?cY. --  /% ) #i'CC --eS*5 6#7& nn Mr%c4td|jDS)Nc3pK|].}tj|j|j0ywrVr VariableSpecr r!rts r#rwz2ShardedVariableMixin._type_spec...s-  " . .qww @ 46rSrrYs r# _type_speczShardedVariableMixin._type_spec+s  __  r%c\tjr |jgS|jS)z?The list of `Variable`s that make up the shards of this object.)rin_save_contextrrrYs r#rzShardedVariableMixin.variables4s)##%## $$ ??r%c|jS)z0The name of this object. Used for checkpointing.rrYs r#r}zShardedVariableMixin.name;s ::r%c|jS)z,The dtype of all `Variable`s in this object.)rrYs r#r!zShardedVariableMixin.dtype@ ;;r%c|jS)z7The overall shape, combining all shards along axis `0`.)rrYs r#r zShardedVariableMixin.shapeErr%c t|jD]P\}}|jtj||j ||j jR|SrV)rrassignr rrr rz)rr7 use_lockingr} read_valuerrvs r#rzShardedVariableMixin.assignJsT$//*P1hhyud&7&7&:AGGOOrr4rrrzrrmaximumr wherecastrint32rdynamic_partition) rindicesrextraexpect_first_dimrrvactual_first_dimpartition_assignments local_indicesper_var_indicess r#_decompose_indicesz'ShardedVariableMixin._decompose_indices]s}}Q  $]]O -  ;;q>S1 1D KKNS1 1EvDOO 44 5\4,Q/!3q46:ooF)!,FF++  & %,,D1H%D8OO%4!8 5D M %MM*?N#55,c$//.BO 1 11?Gs>$Fc>|j|j\}}tj|j|t |j }tt |j Dcgc]}tj||||!c}Scc}w)zDDecompose a global `IndexedSlices` into a list of per-variable ones.)rr) rrrrrr4rrindexed_slices_lib IndexedSlices)rrrrper_var_valuesrs r#_decompose_indexed_slicesz.ShardedVariableMixin._decompose_indexed_slicess-1-D-D.*O*#444c$//6JNs4??+,    ((!!$oa.@   s3$Bc|j|}t|jD]1\}}d}|dj||}|j |||3|S)z#Implements tf.Variable.scatter_add.N {}/part_{}r})rrrformat scatter_addr sparse_deltarr}per_var_sparse_deltarrvnew_names r#rz ShardedVariableMixin.scatter_addi99,G$//*<1h &&tQ/mm(+(m; < Kr%c|j|}t|jD]1\}}d}|dj||}|j |||3|S)z#Implements tf.Variable.scatter_div.Nrr)rrrr scatter_divrs r#rz ShardedVariableMixin.scatter_divrr%c|j|}t|jD]1\}}d}|dj||}|j |||3|S)z#Implements tf.Variable.scatter_max.Nrr)rrrr scatter_maxrs r#rz ShardedVariableMixin.scatter_maxrr%c|j|}t|jD]1\}}d}|dj||}|j |||3|S)z#Implements tf.Variable.scatter_min.Nrr)rrrr scatter_minrs r#rz ShardedVariableMixin.scatter_minrr%c|j|}t|jD]1\}}d}|dj||}|j |||3|S)z#Implements tf.Variable.scatter_mul.Nrr)rrrr scatter_mulrs r#rz ShardedVariableMixin.scatter_mulrr%c|j|}t|jD]1\}}d}|dj||}|j |||3|S)z#Implements tf.Variable.scatter_sub.Nrr)rrrr scatter_subrs r#r z ShardedVariableMixin.scatter_subrr%c|j|}t|jD]1\}}d}|dj||}|j |||3|S)z&Implements tf.Variable.scatter_update.Nrr)rrrrscatter_updaters r#r z#ShardedVariableMixin.scatter_updatesl99,G$//*?1h &&tQ/+A.X> ? Kr%c|j|}t|jD]1\}}d}|dj||}|j |||3|S)z,Implements tf.Variable.batch_scatter_update.Nrr)rrrrbatch_scatter_updaters r#r z)ShardedVariableMixin.batch_scatter_updatesn99,G$//*E1h &&tQ/1!48D E Kr%c |j|\}}g}t|jD]@\}}d}|dj||}|j |j |||Bt j|dS)z#Implements tf.Variable.sparse_read.Nrrrr)rrrrr sparse_readr r) rrr}rrhr8rrvrs r#rz ShardedVariableMixin.sparse_reads009OQ F$//*F1h &&tQ/ mmAMM/!"48MDE F   F ++r%cJjffd }tj|iS)z4Return a `Saveable` for each shard. See `Trackable`.c2g}tjdj}t|Dcgc]}d}}jD]}tj j jjjtj||jj}|jtj||j||dxxt|jdz cc<|Scc}w)z5Creates `SaveableObject`s for this `ShardedVariable`.r) full_name full_shape var_offset var_shape)r4rr rrrrs SaveSliceInfor}rzcopyrrResourceVariableSaveablespecry)r} saveablesr6rhrrvrrs r#_saveable_factoryzPShardedVariableMixin._gather_saveables_for_checkpoint.._saveable_factorysi #)) *d$T{+!A+j+ )!'00>>iizz))+yy,ggoo' ?   9 9?''  1 QWWQZ(  ),s D)r} trackableVARIABLE_VALUE_KEY)rrs` r# _gather_saveables_for_checkpointz5ShardedVariableMixin._gather_saveables_for_checkpoints% $yy(  ( (*; <>!  ' ' / /) DDr%c4|jdjSNr)r_distribute_strategyrYs r#r1z)ShardedVariableMixin._distribute_strategy1s >>!  1 11r%c|jSrVrrYs r# _shared_namez!ShardedVariableMixin._shared_name5s ::r%cy)NTr+rYs r#is_sharded_variablez(ShardedVariableMixin.is_sharded_variable9 r%c4t|jS)a-Copies the values in this ShardedVariable to a NumPy array. First converts to a single Tensor using the registered conversion function, which concatenates the shards, then uses Tensor.numpy() to convert to a NumPy array. Returns: A NumPy array of the same shape and dtype. )rnumpyrYs r#r8zShardedVariableMixin.numpy=s $  % % ''r%rW)NNT)FNT)FNrV)%r'r(r)r*r1rrrrkrrr}r!r rrrrrrrrrrr r r rrr"r%r-r1r3r5r8 __classcell__)rs@r#rnrn s1" Yv!K ZjX            .2`$ ,=2<  E E 2 2     (r%rnz'__internal__.distribute.ShardedVariablecXeZdZdZedZedZedZdZ dZ dZ dZ y ) rXaA container for `Variables` that should be treated as shards. Variables that are too large to fit on a single device (e.g., large embeddings) may need to be sharded over multiple devices. This class maintains a list of smaller variables that can be independently stored on separate devices (eg, multiple parameter servers), and saves and restores those variables as if they were a single larger variable. Objects of this class can be saved with a given number of shards and then restored from a checkpoint into a different number of shards. Objects of this class can be saved to SavedModel format using `tf.saved_model.save`. The SavedModel can be used by programs like TF serving APIs. It is not yet supported to load the SavedModel with `tf.saved_model.load`. Since `ShardedVariable` can be saved and then restored to different number of shards depending on the restore environments, for example, TF serving APIs would restore to one shard for serving efficiency, when using `ShardedVariable` in a tf.function, one should generally not assume it has the same number of shards across save and load. Sharding is only supported along the first dimension. >>> class Model(tf.Module): ... def __init__(self): ... self.sharded_variable = ShardedVariable([ ... tf.Variable([3.0], dtype=tf.float32), ... tf.Variable([2.0], dtype=tf.float32) ... ]) ... ... @tf.function(input_signature=[tf.TensorSpec([], dtype=tf.int32)]) ... def fn(self, x): ... return tf.nn.embedding_lookup(self.sharded_variable.variables, x) ... ... @tf.function(input_signature=[tf.TensorSpec([], dtype=tf.int32)]) ... def serve_fn(self, x): ... return tf.nn.embedding_lookup(self.sharded_variable.variables, x) >>> >>> model = Model() >>> model.fn(1).numpy() 2.0 >>> tf.saved_model.save(model, export_dir='/tmp/saved_model', ... signatures=model.serve_fn) c4td|jDS)Nc3pK|].}tj|j|j0ywrVrrts r#rwz-ShardedVariable._type_spec..~s- $ , ,QWWagg > $rrrYs r#rzShardedVariable._type_spec{s  $?? $ %%r%cptjjD]}|dk(r |j|y)z%Register overloads for all operators.rN)rrOVERLOADABLE_OPERATORS_overload_operator)clsoperators r#_overload_all_operatorsz'ShardedVariable._overload_all_operatorss8%%<<' ] " X& 'r%c^ttj|fd}t|||y)z5Delegate an operator overload to `tensor_lib.Tensor`.c.t|g|i|SrV)r)rvargsr(tensor_operators r# _operatorz5ShardedVariable._overload_operator.._operators ^A. @ @ @@r%N)getattrrrsetattr)r@rArGrFs @r#r?z"ShardedVariable._overload_operators+j//:OA C9%r%cyrVr+)rconcrete_functioninternal_captures r##__tf_experimental_restore_capture__z3ShardedVariable.__tf_experimental_restore_capture__s r%cy)z6Pass resource_variable_ops.is_resource_variable check.Tr+rYs r# _should_act_as_resource_variablez0ShardedVariable._should_act_as_resource_variabler6r%cJtj|j||dy)NF)enforce_naming)r(write_object_proto_for_resource_variabler)rprotor's r#_write_object_protoz#ShardedVariable._write_object_protosBB uger%c||vr#|jD]}|j|yg}|jD]'}|j||j||)t||j}|||<y)r rN)rr"rrXr})rr!rv copied_varsnew_vars r#r"z&ShardedVariable._copy_trackable_to_cpus z-!   ,-k*!   ,:a=)* $))rr}rrget_name_scoperr rr)rr!r}as_refs r#rrs  u77 B  vejj#))..9  D  3--//      #--a 00r%ctt|tr|d}tj|j|||||Sr0)rqrrrZr)paramsidspartition_strategyr}validate_indicesmax_norms r#rZrZsB AYF  ' '     r%embedding_weightsc Rtj|j|||||||| S)z.Pass the individual shard variables as a list.)sparse_weightscombiner default_idr}rarcallow_fast_lookup)rsafe_embedding_lookup_sparser) rd sparse_idsrfrgrhr}rarcris r#rjrjs8  3 3!!# +)  r%)NNF)modNTN)NmeanNNdivNF):r*rrtypingrrr8rtensorflow.python.frameworkrrrrrrr rr r r tensorflow.python.opsr rrrrrrrrtensorflow.python.saved_modelrtensorflow.python.trackablerr!tensorflow.python.training.savingrtensorflow.python.utilr tensorflow.python.util.tf_exportrobjectrr-r;rLTypeSpecrS TrackablernCompositeTensorrXr#register_tensor_conversion_functionrBdispatch_for_typesrZdispatch_for_apirjr+r%r#r~s 83.L+<B41+//*77<69B+6 ="E!&!F!H HRP[Q< DL99M9x DL??M?D),,6z(9..z(z 4<i!*,<,L,Li!=i!X1F?>>^'')];;_M   N,=EEF   &Gr%