AVh}/2dZddlmZmZmZmZmZddlmZddl Z ddl m Z ddl m Z ddl mZddl mZdd l mZdd lmZdd lmZdd lmZdd lmZddlmZdededededej:ddf dZ d&deeeej>efdeedeeej>effdZ edgddddejBfdeeeeeefeeeffdedeeeej>efdeede"dej:f dZ#edgdddejBfdeeeeefeeeffded eeeej>efdeede"dej:f d!Z$iZ%ed"g d&dej:d#eed$eefd%Z&y)'z%Utilities to help with mesh creation.)DictListOptionalTupleUnion)loggingN)accelerator_util)api)config)layout)tpu_util)context)device) array_ops)math_ops) tf_exportnum_global_devices num_clients client_id device_typemeshreturncTtjd||tjd|j|tjd|jtjd|j tjd|j y)NzThis is client %d of %d clientszNumber of global %s devices: %dzGlobal device IDs: %szLocal device IDs: %szLocal devices: %s)rinfoupperglobal_device_idslocal_device_ids local_devicesrrrrrs S/home/dcms/DCMS/lib/python3.12/site-packages/tensorflow/dtensor/python/mesh_util.py_print_contextr!"sz ,,0)[I ,,0+2C2C2E!# ,,&(>(>(@A ,,%t'<'<'>? ,,"D$6$6$89devicesc||d}tj|}||fSt|dtr,|Dcgc]!}tj j |#}}||dj}|j|djjk7rtdt|d|||fScc}w)zAMakes device specs for all local devices or from a provided list.CPUrzConflicting devices z and device_type ) r r isinstancestr tf_device DeviceSpec from_stringrr ValueError)r#rds r _make_device_specsr-.s  _k"";/G + '!*c">EF%%11!4FgFAJ**kgaj44::<<  W.? } M  + Gs&B<z experimental.dtensor.create_mesh)v1 mesh_dims mesh_name use_xla_spmdcjt||\}}tjtjdd}|Dcgc]}|j |}}t |trt|j}|dt|fg}n/t|dk(r!|dddk(r|ddt|f|d<|Dcgc]}|d }}|Dcgc]}|d } }tj| t|k7rtdt|d| tjt|j| } tj | j#} t%j&|| | |||} t)t|dd|| | Scc}wcc}wcc}w) a6Creates a single-client mesh. If both `mesh_dims` and `devices` are specified, they must match each otehr. As a special case, when all arguments are missing, this creates a 1D CPU mesh with an empty name, assigning all available devices to that dimension. Args: mesh_dims: A dict of dim_name: dim_size, or a list of (dim_name, dim_size) tuples. Defaults to a single batch-parallel dimension called 'x' usin all devices. As a special case, a single-element mesh_dims whose dim_size is -1 also uses all devices. e.g. `{'x' : 4, 'y' : 1}` or `[('x', 4), ('y', 1)]`. mesh_name: Name of the created mesh. Defaults to ''. devices: String representations of devices to use. This is the device part of tf.DeviceSpec, e.g. 'CPU:0'. Defaults to all available logical devices. device_type: If `devices` is missing, the type of devices to use. Defaults to 'CPU'. use_xla_spmd: Boolean when True, will use XLA SPMD instead of DTensor SPMD. Returns: A single-client mesh created from specified or default arguments. rjobreplicataskxzlength of devices (3) must be equal to total size of the mesh of shape  dim_namesrrrr1r2r)r-r(r)r job_namemake_merged_specr&dictlistitemslennpprodr+arangereshaperaveltolistr Meshr!) r0r1r#rr2 device_specs local_specr,r=shaperrrs r create_meshrNFs<1+F, ##(911M*:FGQ*--a0G,G 4 Y__&'Is<()*I 9~y|A"4aLOS%67IaL&'qt')'" #A1Q4 #% #WWU^s<(( *3|+<*=>AAFI JJiiL 12::5AXX/0779 )'   !$\*   +AH( #sF& F+ F0z,experimental.dtensor.create_distributed_meshrc t|trt|j}t |\}}t j s td|rC|jdk(r0|.td|jd|jdt||\}}|jdvrtjtjdtj}|D cgc]} |j| }} t!|tj"z} t%j&|| k7r2td t!|d tj"d | d |t%j(| j+|} t%j,| j/} t!|tjz} | | | t!|z}t1j2|| |||| }t5| tj"tj|||S|jdk(rat7j8||||}t5tj:|tj"tj|||Std|dcc} w)a"Creates a distributed mesh. This is similar to `create_mesh`, but with a different set of arguments to create a mesh that spans evenly across a multi-client DTensor cluster. For CPU and GPU meshes, users can choose to use fewer local devices than what is available `local_devices`. For TPU, only meshes that uses all TPU cores is supported by the DTensor runtime. Args: mesh_dims: A dict of dim_name: dim_size, or a list of (dim_name, dim_size) tuples. e.g. `{'x' : 4, 'y' : 1}` or `[('x', 4), ('y', 1)]`. mesh_name: Name of the created mesh. Defaults to ''. local_devices: String representations of devices to use. This is the device part of tf.DeviceSpec, e.g. 'CPU:0'. Defaults to all available local logical devices. device_type: Type of device to build the mesh for. Defaults to 'CPU'. Supported values are 'CPU', 'GPU', 'TPU'.6 use_xla_spmd: Boolean when True, will use XLA SPMD instead of DTensor SPMD. Returns: A mesh that spans evenly across all DTensor clients in the cluster. zYAccelerators are uninitialized, please run dtensor.initialize_accelerator_system() first.TPUzDo not specify devices for z- meshes. Using a partial list of devices for z is not supported.)r%GPUrr4zGlobal number of devices (z per client * z clients = r;r<)mesh_dim_names mesh_shaper1r2z Device type z is not CPU, GPU or TPU)r&r@rArBzipr is_initializedr+rr-r(r)r r>rr?rCrrDrErFrGrHrIr rJr!r create_tpu_meshr)r0r1rrr2r=rMrKrLr,rr flattened start_idxrrs r create_distributed_meshrYsB 4 Y__&'I)_)U  ( ( * F GG[&&(E1  ' (9(9(;'<=11<1B1B1D0EF   1 L, N* %% OO qv/?/?/ACJ ??  "45==eD*+224IL!F$4$4$66I 9s<7H+HI ;;+)"!  #D%v'9'9';V=M=M=O& KE!  # # !  #D !!+.0B0B0DK/ K\+.EFGGOJs;K zexperimental.dtensor.barrier barrier_name timeout_in_msc|d}tjd|tjt j ddgt |jz}tj|g|jztj|j|}tj|}||j k7r%t#dj%||j tjtjj&rQ|d}t(j+|d}|dzt(|<|d |}tjj-||tjd |y) aRuns a barrier on the mesh. Upon returning from the barrier, all operations run before the barrier would have completed across all clients. Currently we allocate a fully sharded tensor with mesh shape and run an all_reduce on it. Example: A barrier can be used before application exit to ensure completion of pending ops. ```python x = [1, 2, 3] x = dtensor.relayout(x, dtensor.Layout.batch_sharded(mesh, 'batch', 1)) dtensor.barrier(mesh) # At this point all devices on all clients in the mesh have completed # operations before the barrier. Therefore it is OK to tear down the clients. sys.exit() ``` Args: mesh: The mesh to run the barrier on. barrier_name: The name of the barrier. Mainly used for logging purpose. timeout_in_ms: The timeout of the barrier in ms. If omitted, blocks indefinitely till the barrier is reached from all clients. Nz (barrier)zentering barrier before op: %sg?r9zMGlobal barrier produced wrong mesh size : {0} while mesh has actualsize : {1}i\&r:z8finished running barrier across all clients after op: %s)rrr async_waitrrGrCrMr packnum_local_devicesr Layoutr=r reduce_sumsizer+formatcoordination_service _BARRIER_DICT setdefaultwait_at_barrier)rrZr[ componentones mesh_size num_calls barrier_ids r barrierrnsI@L ,,/> aS3tzz|+<%<=) 9+ 6 6 88--5 7$!!$')$))  VItyy1 33   __++)m((q9I"+a-M, >9+.J OO%%j-@ ,,%'r")NN)'__doc__typingrrrrrabslrnumpyrDtensorflow.dtensor.pythonr r r r r tensorflow.python.eagerrtensorflow.python.frameworkrr(tensorflow.python.opsrr tensorflow.python.util.tf_exportrintr'rJr!r)r- USE_XLA_SPMDboolrNrYrfrnr"r r|s,556),,.+;+*6:s::: #:+1;;:;?:AE!% d5!5!5s!:;< =# 4 $$ %s *+0 -"5HL@D!%,, @d5c?3T#s(^CDE@@d5!5!5s!:;< =@# @  @  [[ @6@F 9bAFJ!%,, bHT%S/*DcN:;bHbHDy';';S'@!ABCbH# bH  bH  [[ bHBbHJ  )b1*.+/B'&++B'"3-B'#C=B'2B'r"