AVhZdZddlZddlmZddlmZddlm Z ddl m Z ddl m Z Gdd eZdd Zdd Z dd Z dd Z ddZ ddZdZdZdZ ddZ ddZ ddZ ddZy)z0Experimental support for defining XLA shardings.N)xla) xla_data_pb2)attr_value_pb2)context)resource_variable_opsceZdZdZddZedZedZedZedZ edZ ed Z edd Z dd Z d Zed ZdZdZy)ShardingzA class to support adding sharding attributes to Ops. Use the factory constructors and then call apply_to_tensor: Sharding.replicate().apply_to_tensor(tensor) Nc||_y)z=Do not use this constructor; use the factory functions below.N_proto)selfprotos h/home/dcms/DCMS/lib/python3.12/site-packages/tensorflow/python/compiler/xla/experimental/xla_sharding.py__init__zSharding.__init__!s DKcrttjtjjS)zReturns a replicated sharding attribute. This causes an op to be computed in its entirety independently on all cores in the XLA device. typer)r r OpSharding REPLICATEDclss r replicatezSharding.replicate%s- %%<+B+B+M+MN PPrcrttjtjjS)zReturns a manuall sharding attribute. This means the op is manually partitioned by the user and XLA will not change the shapes. rr)r rrMANUALrs rmanualzSharding.manual/s- %%<+B+B+I+IJ LLrczttjtjjdg|gS)zReturns an AssignDevice sharding attribute. This causes an op to be computed in its entirety only on one core in the XLA device. Args: core: The core to assign this Op to. rtile_assignment_dimensionstile_assignment_devicesr)r rrMAXIMAL)rcores r assign_devicezSharding.assign_device9s8 %%((00()s%)F, --rc "t|tjs tdt |j }|j dd}ttjtjj|t |S)aReturns a Tiled sharding attribute. This causes an op to be partially computed on multiple cores in the XLA device. Args: tile_assignment: An np.ndarray describing the topology of the tiling and which device will compute which part of the topology. Raises: TypeError: tile_assignment was not of np.array type. TODO(jmolloy): This concept is nefarious and is not something we really want to expose to users (especially as the contract for tile_assignment is very strict). z*Tile assignment must be of type np.ndarrayCorderr r isinstance_npndarray TypeErrorlistshapereshaper rrOTHERrtile_assignmentdimsflattened_devicess rtilez Sharding.tileHsz$ os{{ 3 B CC %% &D'//#/> %%((..'+$():$;= >>rc Tt|tjs tdt|ts tdt |j t |kr td|D]C}|tjjtjjfvs:tdt |j }|jdd}ttjtjj|t |t | S) aiReturns a subgroup manual sharding attribute. This is similar to tile(), but tile_assignment has one or more dimension than the tensor, and subgroup_modes define the sharding types in the last dimensions of tile_assignment. Args: tile_assignment: An np.ndarray describing the topology of the tiling and which device will compute which part of the topology. subgroup_modes: sharding types for the dimension more than the tensor shape rank. Raises: TypeError: tile_assignment was not of np.array type or subgroup_modes has unsupported sharding type. z2SubgroupTile assignment must be of type np.ndarrayz6subgroup_modes in subgroup manual must be of type listzKSubgroupTile assignment must have rank larger than length of subgroup_modeszEach sharding_type in subgroup_modes in subgroup manual must be of type xla_data_pb2.OpSharding.REPLICATED or xla_data_pb2.OpSharding.MANUALr'r(r))rr!r"last_tile_dimsr)r,r-r.r/r0lenr1rrrrr2r r3)rr5subgroup_modes sharding_typer6r7s r subgroup_tilezSharding.subgroup_tileds$ os{{ 3 J KK nd + N OO ? !C$77 2 33(2   ! ! , ,l.E.E.L.L  12 2 2 %% &D'//#/> %%((..'+$():$;/ 1 22rc $t|tjs tdt |j }|j dd}ttjtjj|t |dS)aReturns a partially tiled sharding attribute. This is similar to tile(), but tile_assignment has one more dimension than the tensor, and tiles in the last dimension of tile_assignment are replicated. Args: tile_assignment: An np.ndarray describing the topology of the tiling and which device will compute which part of the topology. Raises: TypeError: tile_assignment was not of np.array type. z1PartialTile assignment must be of type np.ndarrayr'r(r)T)rr!r"replicate_on_last_tile_dimrr+r4s r partial_tilezSharding.partial_tiles} os{{ 3 I JJ %% &D'//#/> %%((..'+$():$;'+ - ..rc 0|r|}n|jj}|||||krtd|d|d|dgt|z}|||<t t j t j j|t|S)a/Returns a Sharding that splits a tensor across a dimension. This creates a Tiled attribute, similar to tile(), but easier to use for the common case of tiling a tensor N ways in one dimension. Args: tensor: A tf.Tensor to split. split_dimension: The dimension number to split. num_devices: The number of cores to split `tensor` over. input_shape: The shape of the original tensor. Raises: ValueError: The tensor to split was smaller in the split dimension than the number of devices to split over. zFSplit dimension was smaller than the required number of splits: shape=z , dimension=z, num_devices=rr r) r1as_list ValueErrorr;r rrr3range)rtensorsplit_dimension num_devices input_shaper1tile_assignment_dimss rsplitzSharding.splits"ell""$e o* o,  = >>3U+,7) %%((..';$)+$68 99rcZ|r|r|rJ|j}t|tjr;t j r'|r|j d}|j||S|re|r8|j d}tj||j}ntj||j|xsg}n|s"t|jjdkDrx|j|j}t|j }|j||j"<t%j&t$j&j(|}|jj+dt-j.|j|S)aeApplies this Sharding attribute to `tensor`. Args: tensor: A tf.Tensor to split. assign_tuple_sharding: If the sharding type should be a tuple. use_sharding_op: Whether to create a sharding op on `tensor`. unspecified_dims: An optional list of dimensions unspecified. Returns: The tensor with Sharding attribute. r) num_outputssharding)rOunspecified_dimsrtuple_shardings _XlaShardings)r r,rBaseResourceVariabler+xla_sharding_for_resource_variables_enabled_create_tuple_proto_set_xla_shardingtf2xlarOSerializeToStringr;opoutputs_get_or_create_tuple_protor0rR value_indexrrTUPLE _set_attrr AttrValue)r rFassign_tuple_shardinguse_sharding_oprPrrRs rapply_to_tensorzSharding.apply_to_tensorse  %:: : KKE 60EEF  ? ? A ((Q(7 u% m ((Q(7%2I2I2KL ,,.-35 #fii&7&7"81"<--fii8eU223o,0KKof(()%%&&,,oOe  II&0053J3J3LMO Mrctj|jj}|j d|y)z{Applies this Sharding attribute to `operation`. Args: operation: A tf.Operation to add sharding annotation. rTrSN)rrbr r[ra)r operation attr_values rapply_to_operationzSharding.apply_to_operation s1  ))DKK,I,I,KLJ  3rc|jS)z=Return the sharding protobuf of type xla_data_pb2.OpSharding.r )r s rrzSharding.protos ;;rc |jd}tj}|j||S#t$r'|j t |jcYSwxYw)NrS)get_attrrrParseFromStringrDrXr;r])r r\attrrs rr^z#Sharding._get_or_create_tuple_protos^7 [[ (d%%'e D! l 7  % %c"**o 667s7:-A*)A*ctjtjjg|z}tjtjj|S)NrrQ)rrrr`)r rM shardingss rrXzSharding._create_tuple_proto"sS\%<%<%G%GHI  " "  $ $ * *I GGrNFFN)__name__ __module__ __qualname____doc__r classmethodrrr%r8r>rArKreripropertyrr^rXrrr r s PPLL - ->>6*2*2X..2!9!9J-2&+'+ 9v4  7Grr ct|}||St|tjrLt j r8t j}|j||j||S|rtj||}tj|}|jjd||S)a@Copies the a tensor's sharding to another. Args: from_tensor: Source tensor. Must be the sole output of an op. to_tensor: the tensor the annotate with the copy. use_sharding_op: whether to create a sharding op on `to_tensor`. Returns: A tensor with sharding annotation copied from `from_tensor`. rNrTrS)get_tensor_shardingr,rrVrrWrrrmrYrZrOrrbr\ra) from_tensor to_tensorrdrOrrhs r copy_shardingr~*s! -(  1FFG  = = ?  # # %E (# &  H=I''(3* ,,4 rcNtjj|||S)Nrcrd)r rre)rFrcrds rrrSs,     - - 1% . ''rcPtj|j|||S)z:Returns a tensor that has AssignDevice sharding attribute.r)r r%re)rFdevicercrds rr%r%Zs0    ' 7 7 1% 8 ''rcZtj|j||||xsgS)aReturns a tensor that has tiled sharding. Args: tensor: A tf.Tensor to shard. tile_assignment: An np.ndarray describing the topology of the tiling and which device will compute which part of the topology. assign_tuple_sharding: If the sharding type should be a tuple. use_sharding_op: If true, adds a sharding op to set the sharding. unspecified_dims: An optional list of dimensions unspecified. )rcrdrP)r r8re)rFr5rcrdrPs rr8r8es6  ' 7 7 1%'-2 8 //rcVtj||||j|||S)aReturns a tensor that is split along the given dimension. Args: tensor: A tf.Tensor to split. split_dimension: The dimension to split. num_devices: The number of devices to partition the dimension. assign_tuple_sharding: If the sharding type should be a tuple. use_sharding_op: If true, adds a sharding op to set the sharding. input_shape: The full shape of the input tensor. r)r rKre)rFrGrHrcrdrIs rrKrK{s6 # %%4_"2G,;&5&==rcXtj|j|||xsgS)aReturns a tensor that has tiled sharding. Args: tensor: A tf.Tensor to shard. tile_assignment: An np.ndarray describing the topology of the tiling and which device will compute which part of the topology. It must have one more dimension than tensor, and the last dimension represents partially replicated tiles. use_sharding_op: If true, adds a sharding op to set the sharding. unspecified_dims: An optional list of dimensions unspecified. rdrP)r rAre)rFr5rdrPs rrArAs6    / ? ? %'-2 @ //rcX |jdS#t$rYyt$rYywxYw)zReturns sharding attribute of an op. Args: op: a TensorFlow op. Returns: The attribute representing XLA sharding on this op. rSN)rlrDAttributeError)r\s rget_op_shardingrs3 ;;~ &&    s )))ct|tjr7tjr#|j }|y|j S t|jS#t$rYywxYw)zReturns sharding attribute of a Tensor. Args: tensor: a Tensor. Returns: The attribute representing XLA sharding on tensor's op. N) r,rrVrrW_get_xla_shardingr[rr\r)rFrOs rr{r{so.CCD  = = ?'')H   ' ' )) 699 %%  sA(( A43A4c|ytj}|j||jr |jSy)a_Returns the tile assignment shape for a sharded Tensor. Args: sharding: a serialized OpSharding message describing the layout of a sharded Tensor. Returns: A list, for each dimension of the sharded Tensor, of the number of shards into which it has been split. Returns None if the input indicates no tile assignments. N)rrrmr!)rOsharding_messages rget_sharding_tile_shapersB !,,.""8,00  6 66 rc<tj||||xsgS)aaSwitches from automatic SPMD partitioning to manual partitioning. Converts a full-shaped tensor (to be automatically partitioned by SPMD partitioner) to a shard-shaped tensor to be consumed by manually partitioned ops. Args: tensor: A tf.Tensor in full shape. manual_sharding: A serialized string of OpSharding to be used in manual partitioning. single_dim: If >= 0, the conversion will happen only on this dim in subgroups. unspecified_dims: An optional list of dimensions unspecified. Returns: A shard-shaped tensor to be consumed by manually partitioned ops. )manual_shardingdimrP)rZspmd_full_to_shard_shape)rFr single_dimrPs rauto_to_manual_spmd_partitionrs(*  ( ( % '-2  //rc>tj|||||xsgS)aSwitches from manual partitioning to automatic SPMD partitioning. Converts a shard-shaped tensor (manually partitioned in SPMD-style) to a full-shaped tensor to be partitioned automatically by the SPMD partitioner. Args: tensor: A tf.Tensor in shard shape. manual_sharding: a serialized string of OpSharding to be used in manual partitioning. full_shape: the shape of tensor before partitioning. single_dim: If >= 0, the conversion will happen only on this dim in subgroups. unspecified_dims: An optional list of dimensions unspecified. Returns: A full-shaped tensor to be partitioned automatically by the SPMD partitioner. )r full_shaperrP)rZspmd_shard_to_full_shape)rFrrrrPs rmanual_to_auto_spmd_partitionr s+.  ( ( % '-2  //rc |xsg}|Dcgc] }|dk\s | c}|z}t|t|jkDr=tdt|dt|jd|d|j|tt|jDcgc] }||vs| c}z}t j ||}||zDcgc]}|dkrdn|j|}}t jjgt|z}t|t|jk} | rm|jt j|jt j|z|jt jjt j||}|rtj||S| rtj|Stj!|Scc}wcc}wcc}w)aReturns a Sharding object representing sharding along multiple dimensions. Args: device_mesh: An np.ndarray describing the topology of the device mesh and each element is the ID of the device in the topology. tensor_split_dims_mapping: A list of integers that map each tensor axis to the device mesh axis along which it is sharded. Its length is the tensor rank, and tensor_split_dims_mapping[i] is device mesh axis for tensor dimension i. Use -1 for tensor dimensions that are not sharded. manual_mesh_dims: An optional list of mesh dims for manual subgroups. Raises: ValueError: The number of tensor split dimensions is larger than device mesh rank. rz#Number of tensor split dimensions (z#) is larger than device mesh rank (z). tensor_split_dims_mapping: z, device_mesh.shape: r)r;r1rDrEr- transposerrrappendprodrr2r r>rAr8) device_meshtensor_split_dims_mappingmanual_mesh_dimsd permutationtranspose_permutationr5 tile_shaper<partials rmesh_split_shardingr(s$&+5qa$%+K--..  [ 3    9;;L;L N OO &s;,,-.) !;2Fa)MM+/DE/*,<<  q5ak''***!++223c:J6KK.  s;#4#45 5' chh{001SXXj5IIJ,11<<=KK BB   11  '';)s G?G? H)HH cLt|||}|j|||xsgS)alReturns a tensor that is split along multiple dimensions in a device mesh. Args: tensor: A tf.Tensor to split. device_mesh: An np.ndarray describing the topology of the device mesh and each element is the ID of the device in the topology. tensor_split_dims_mapping: A list of integers that map each tensor axis to the device mesh axis along which it is sharded. Its length is the tensor rank, and tensor_split_dims_mapping[i] is device mesh axis for tensor dimension i. Use -1 for tensor dimensions that are not sharded. use_sharding_op: If true, adds a sharding op to set the sharding. manual_mesh_dims: An optional list of mesh dims for manual subgroups. unspecified_dims: An optional list of dimensions unspecified. Raises: ValueError: The number of tensor split dimensions is larger than device mesh rank. r)rre)rFrrrdrrPrOs r mesh_splitr[s<0!.G!13(  ! ! %'-2 " //r)F)FFrr)FN)r'Nrq)FNN)rvnumpyr-!tensorflow.compiler.tf2xla.pythonrrZtensorflow.compiler.xlartensorflow.core.frameworkrtensorflow.python.eagerrtensorflow.python.opsrobjectr r~rr%r8rKrArr{rrrrrryrrrs7;04+7MGvMG`"R')."'' % /2!& =2"'"&/*$<0.037/>.037 /B*.0(l % $ $ /r