AVhwOdZddlZddlZddlZddlmZmZmZmZddl Z ddl m Z ddl mZddlmZddlmZddlmZdd lmZd Zd Zd Zed gj3ededgj3edej6dddgZdeedeefdZedgGddej>Zej@Z edgGddejBZ!y)z+Python definitions for `Mesh` and `Layout`.N)ListDictOptionalUnion) layout_pb2)_pywrap_dtensor_device)device)ops)tensor) tf_export unshardedmatchFzexperimental.dtensor.UNSHARDED)v1 UNSHARDEDzexperimental.dtensor.MATCHMATCH MeshDimensionnamesizeshapereturncdg}tt|ddD]\}}|j|||z|j|S)N) enumeratereversedappendreverse)rstridesidxdim_sizes P/home/dcms/DCMS/lib/python3.12/site-packages/tensorflow/dtensor/python/layout.py_compute_mesh_stridesr!-sP C' %)!45,mc8 NN73<(*+, // .zexperimental.dtensor.Meshc6eZdZdZddefdeedejdee dee e jefded e ee e jefd effd Zefd Zd ejffd Zded efdZdZd efdZdZde d ej2fdZedej8d dfdZeded dfdZeded dfdZede jBfdZ"e#jHd#fd Z%d#dZ&d ee'ee ffd Z(e)d ee fd!Z*d"Z+xZ,S)$MeshaRepresents a Mesh configuration over a certain list of Mesh Dimensions. A mesh consists of named dimensions with sizes, which describe how a set of devices are arranged. Defining tensor layouts in terms of mesh dimensions allows us to efficiently determine the communication required when computing an operation with tensors of different layouts. A mesh provides information not only about the placement of the tensors but also the topology of the underlying devices. For example, we can group 8 TPUs as a 1-D array for data parallelism or a `2x4` grid for (2-way) data parallelism and (4-way) model parallelism. Refer to [DTensor Concepts](https://www.tensorflow.org/guide/dtensor_overview) for in depth discussion and examples. Note: the utilities `dtensor.create_mesh` and `dtensor.create_distributed_mesh` provide a simpler API to create meshes for single- or multi-client use cases. N dim_namesglobal_device_idslocal_device_ids local_devices mesh_nameglobal_devices use_xla_spmdc t|tjs td|jdk(r td|j }|dt fdt|Drtd|zt||jk7r tdt|ts tdt|ts td|rt|ts td |s |s td |j } |j} d tfd } d tjfd } |D cgc] } | |  }} |D cgc] } | |  }} |sg}|D cgc] } | |  }} |D cgc] } | |  }} t|}t|dk(xrt|dj dk(}|sHt|t|k7r1td|D cgc]} |j#| dkDs| c} zt|t|k7r tdt|ttj$|kDr tdt|Dcgc]}|j c}}|s#t|Dcgc]}|j c}}d|vr tdt|dkDrtd|z|j'}|r|dk7rtd|zt(|U||| | ||||ycc} wcc} wcc} wcc} wcc} wcc}wcc}w)ayBuilds a Mesh. The `dim_names` and `global_device_ids` arguments describe the dimension names and shape for the mesh. For example, ```python dim_names = ('x', 'y'), global_device_ids = [[0, 1], [2, 3], [4, 5]] ``` defines a 2D mesh of shape 3x2. A reduction over the 'x' dimension will reduce across columns (0, 2, 4) and (1, 3, 5), and a reduction over the 'y' dimension reduces across rows. Note: the utilities `dtensor.create_mesh` and `dtensor.create_distributed_mesh` provide a simpler API to create meshes for single- or multi-client use cases. Args: dim_names: A list of strings indicating dimension names. global_device_ids: An ndarray of global device IDs is used to compose DeviceSpecs describing the mesh. The shape of this array determines the size of each mesh dimension. Values in this array should increment sequentially from 0. This argument is the same for every DTensor client. local_device_ids: A list of local device IDs equal to a subset of values in global_device_ids. They indicate the position of local devices in the global mesh. Different DTensor clients must contain distinct local_device_ids contents. All local_device_ids from all DTensor clients must cover every element in global_device_ids. local_devices: The list of devices hosted locally. The elements correspond 1:1 to those of local_device_ids. mesh_name: The name of the mesh. Currently, this is rarely used, and is mostly used to indicate whether it is a CPU, GPU, or TPU-based mesh. global_devices (optional): The list of global devices. Set when multiple device meshes are in use. use_xla_spmd (optional): Boolean when True, will use XLA SPMD instead of DTensor SPMD. z.Variable global_device_ids must be an ndarray.rz-Variable global_device_ids must be non-empty.c34K|]\}}||z k7ywN).0igiddistances r z Mesh.__init__..s% O"(!SqH  Osz0global_device_ids must sequentially increase: %szCNumber of mesh dimensions does not match number of dimension names.z5Variable local_device_ids must be a list of integers.z5Variable local_devices must be a list of DeviceSpecs.z6Variable global_devices must be a list of DeviceSpecs.z"Empty list of devices not allowed.rcZt|tjr|jS|Sr/) isinstance tf_device DeviceSpec to_stringds r to_strzMesh.__init__..to_strs# Ay++ ,{{} hr"cxt|tjstjj|S|Sr/)r7r8r9 from_stringr;s r to_speczMesh.__init__..to_specs. 9// 0##//22 hr"rCPUz1Duplicate devices found in mesh specification %s.zCVariable local_device_ids does not have same size as local_devices.z-Cannot have more local than gobal device IDs.Nzdevice_type is requiredz-Devices containing multiple device_types : %sTPUz4XLA SPMD is not currently not supported for %s mesh.)r7npndarray ValueErrorrflattenanyrlenndimlistrstrr8r9set device_typecountravelpopsuper__init__)selfr&r'r(r)r*r+r,flat_global_device_idsglobal_device_ids_flattenglobal_device_ids_shaper=r@r<local_devices_strlocal_devices_specglobal_devices_strglobal_devices_speclocal_devices_set#local_device_only_contains_host_cpur device_typesrMr4 __class__s @r rRz Mesh.__init__Ks~j ' 4 G HH" F GG.668&a(H  O,56L,M OO I() ** 9~*///  O QQ & - N OO mT * N OOj> O PP  ; <<!2 9 9 ;/55S i** -::q::.;<'!*<< n-;<&)<!71:>>./ !# 8 "..%7( /3}3EJ4 J#0OaM4G4G4JQ4NOP QQ M 22  O QQ s288,=#>?? F GG9KLv**LML ;NO&,,OPl | 0 11 <1 F#$ %%""$K u, M"# $$ G! I;<=>PMOs0LL$3L)L.2L3 L3 9L8L=cltjj|}t||g|i||Sr/)rr$__new__rQrRclsargskwargsrSr^s r _new_objectzMesh._new_objects7 " & & . .s 3D GT+D+F+ Kr"rctjt| tjj |j S)z)Returns a global device list as an array.)dtype)rCarrayrQr'int64reshaperrSr^s r r'zMesh.global_device_idss6 88EG-/rxx @ H H  r"dim_namec:t||j|S)N)rr)rr)rSrls r __getitem__zMesh.__getitem__s hT]]8-D EEr"cTt|jjdSNT) deterministichashas_protoSerializeToStringrSs r __hash__z Mesh.__hash__!  111E FFr"c*d|jdS)NzMesh.from_string()r:rvs r __repr__z Mesh.__repr__s t~~/0 22r"cDtj|jffSr/)r$r?r:rvs r __reduce__zMesh.__reduce__s   dnn.0 00r" device_idxctj|j}tj|j}||z|zS)z*>*C$DEFSSj F#F-FG331+s+v}}+ -Z11-f--....1s1v11&277&& 00Id4S>&:I /tCy/ /.r"r$zexperimental.dtensor.Layoutc eZdZdZdeedeffd ZefdZ defdZ dZ d Z e fd Ze d Ze dded ed ededdf dZdeeddfdZedej(ddfdZededdfdZd fd Zededed eddfdZededdfdZededdfdZdZdZeded eddfdZxZS)!LayoutaRepresents the layout information of a DTensor. A layout describes how a distributed tensor is partitioned across a mesh (and thus across devices). For each axis of the tensor, the corresponding sharding spec indicates which dimension of the mesh it is sharded over. A special sharding spec `UNSHARDED` indicates that axis is replicated on all the devices of that mesh. Refer to [DTensor Concepts](https://www.tensorflow.org/guide/dtensor_overview) for in depth discussion and examples. For example, let's consider a 1-D mesh: ``` Mesh(["TPU:0", "TPU:1", "TPU:2", "TPU:3", "TPU:4", "TPU:5"], [("x", 6)]) ``` This mesh arranges 6 TPU devices into a 1-D array. `Layout([UNSHARDED], mesh)` is a layout for rank-1 tensor which is replicated on the 6 devices. For another example, let's consider a 2-D mesh: ``` Mesh(["TPU:0", "TPU:1", "TPU:2", "TPU:3", "TPU:4", "TPU:5"], [("x", 3), ("y", 2)]) ``` This mesh arranges 6 TPU devices into a `3x2` 2-D array. `Layout(["x", UNSHARDED], mesh)` is a layout for rank-2 tensor whose first axis is sharded on mesh dimension "x" and the second axis is replicated. If we place `np.arange(6).reshape((3, 2))` using this layout, the individual components tensors would look like: ``` Device | Component TPU:0 [[0, 1]] TPU:1 [[0, 1]] TPU:2 [[2, 3]] TPU:3 [[2, 3]] TPU:4 [[4, 5]] TPU:5 [[4, 5]] ``` sharding_specsrcft|ts tdt|D]g\}}|tk(s |t k(r|j |dkDrtdj||||vsNtdj|t|%tj||y) aBuilds a Layout from a list of dimension names and a Mesh. Args: sharding_specs: List of sharding specifications, each corresponding to a tensor axis. Each specification (dim_sharding) can either be a mesh dimension or the special value UNSHARDED. mesh: A mesh configuration for the Tensor. Returns: A valid Layout built with given layout & mesh. z mesh is not a valid Mesh object.rz~Mesh dimension {mesh_dim} was repeated in sharding specification {sharding_specs}. Mesh dimensions must be unique in a layout.)mesh_dimrzX{dim_sharding}: A dimension sharding must either be a valid mesh dimension or UNSHARDED.) dim_sharding)typerrN) r7r$rErrrrNformatrQrR LayoutTypeSTATIC)rSrr_rr^s r rRzLayout.__init__s dD ! 9 ::%^4-<  "le&;   l +a /#V&~$GH H T !2396*4:4,- --$ G   ~Dr"cltjj|}t||g|i||Sr/)rrr`rQrRras r rezLayout._new_objects7 " ( ( 0 0 5D GT+D+F+ Kr"rc*d|jdS)NzLayout.from_string(rzr{rvs r r|zLayout.__repr__s !1 2! 44r"cTt|jjdSrprrrvs r rwzLayout.__hash__rxr"cDtj|jffSr/)rr?r:rvs r r~zLayout.__reduce__s    02 22r"c@tjt| S)Nr)r$rrQrrks r rz Layout.meshs ?? ? --r"c6|jjSr/)rrrvs r rz Layout.shapes 99?? r" batch_dimrankaxisc,|j||||S)z,Returns a layout sharded on batch dimension.)rrrrr)rbrrrrs r batch_shardedzLayout.batch_shardeds& ??     r"dimsct|ts|g}t|jDcgc] \}}||vs |}}}t ||j Scc}}w)z4Returns the layout with the give dimensions deleted.)r7rJrrrr)rSrr2spec new_specss r deletez Layout.deletesZ dD !Vd%d&9&9:DatmI )TYY ''s AA layout_protoc&|j|S)z'Creates an instance from a LayoutProto.)rr)rbrs r rzLayout.from_protos ?? ? 55r" layout_strc&|j|S)z1Creates an instance from a human-readable string.)rr)rbrs r r?zLayout.from_strings ??j? 11r"cHtjt| S)zReturns a "parted" layout from a static layout. A parted layout contains axes that are treated as independent by most of SPMD expanders. FIXME(b/285905569): The exact semantics is still being investigated. )layout)rrerQ to_partedrks r rzLayout.to_parteds!   UW%6%8  99r" inner_dimc2|j||||dz S)z,Returns a layout sharded on inner dimension.r)r)r)rbrrrs r inner_shardedzLayout.inner_shardeds"   T9d  BBr"c&|j|S)|jj}dg|jjz}|jD]U\}}g}|jD]1}|t k(r|j d|j ||3t|||<W|S)z7Mapping from offset in a flattened list to shard index.Nr)rrritemsrrrtuple)rSr locationsoffsetmesh_loclocrs r offset_to_shardzLayout.offset_to_shard sII++-M'I)//1% c---, 9 $ **Q- **Xl+ , -  *i% r"cd}t|D]A\}}d}t|dz|jD]}||j|z}|||zz}C|S)z.Mapping from offset to index in global tensor.rr)rrr num_shards)rS offset_tupleindexr2omxs r offset_tuple_to_global_indexz#Layout.offset_tuple_to_global_indexsi E,'1 aQUDII&#! " "#a!eme  Lr"c(|j||S)z+Returns a replicated layout of rank `rank`.)rrr)rbrrs r replicatedzLayout.replicated#s ??4? 00r")r)rr)rrrrrrKr$rRrrer|rwr~rrrrrrr LayoutProtorr?rrrrrrrrrs@r rr_s*X%T#Y%d%N55G3 . .  >?  "% -0 8;   (c(x(6J$:$:6x66232822:CtCC3C8CC&&(&&AsAxAA  1D1111r"r)"r collectionsrrtypingrrrrnumpyrCtensorflow.dtensor.protortensorflow.pythonrtensorflow.python.frameworkr r8r r tensorflow.python.util.tf_exportr rrrexport_constantr namedtuplerrr!r$rrr0r"r rs2../4;+.6   $  ?8[ 1  R))87)K& &&8HI ctCy &2.b ! & &b/bJ $ . .  (R0F1 # * *F11F1r"