AVhjdZddlZddlZddlmZmZmZddlZddl m Z ddl m Z ddl m Z ddl m ZddlmZdd lmZdd lmZdd lmZdd lmZdd lmZddlmZddlmZddlmZddlmZddl m!Z!dZ"dZ#da$da%da&da'GddZ(d@dZ)dee(de*de*dejVfdZ,dZ- dAd Z.dBd!Z/d"ee*d#ee*d$ee*d%ee*d&ee*deee*f d'Z0d"ee*d#ee*d(ee1d)e2d*e*dee(f d+Z3 dBdee(d,e2dee*fd-Z4dee(d*e*d.e2dee(fd/Z5e!d0g1ddddddejlfd2ee1d3ee*d4e1d5ee*d6eee1d#eee*d)e2d7e2d.e2d8e2dejnfd9Z8 d@d:ejnd;ee*dee*fd<Z9 d@d:ejnd;ee*deee1e*ffd=Z:dBd>Z;d?ZReturns ID and device lists that can be used to create a mesh.)rnum_global_devicesnparangereshape local_deviceslenrange) shape device_typehost_idlocal_device_idsr;global_device_idslocal_device_listnum_local_devicesrs r _create_device_arrayrIUs00=ii 23;;EB**;7+, 055F/G*+a'% %%!1 ,.? ?? s)Bcore_locations num_tasksnum_devices_per_taskreturnct|Dcgc]}|jc}dk(sJt|Dcgc]}|jc}dk(sJt|Dcgc]}|jc}dk(sJt|Dcgc]}|jc}dk(sJt |Dcgc]}|jc}}t |Dcgc]}|jc}}t |Dcgc]}|jc}}t |Dcgc]}|jc}}|dz|dz|dz|dzg}|Dcgc]0}|j|j|j|jg2} }t j| j||d} tj|| Scc}wcc}wcc}wcc}wcc}wcc}wcc}wcc}wcc}w)a?Returns a Topology object build from a _CoreLocation list. Args: core_locations: A list of _CoreLocation objects sorted first by TF task ID and then by per-task device ordinals. num_tasks: The number of TF tasks in the cluster. num_devices_per_task: The number of TPU devices local to each task. r) mesh_shapedevice_coordinates) minrrrrmaxr np_asarrayr>rTopology) rJrKrLlx_maxy_maxz_maxcore_maxrQrRs r _create_tpu_topologyr\ds > *aacc * +q 00 0 > *aacc * +q 00 0 > *aacc * +q 00 0 n -aff - .! 33 3 N+qqss+ ,% N+qqss+ ,% N+qqss+ ,% .1Q!&&1 2( 519eaiA>*9GHAacc133/HH#../ABJJ%q*   0B DD+ * * -+++1Is4F7F<GG*G  G0GG5Gctjd}tjr|nd}|rt j dyt j dy)zShuts down the TPU system.c*tjSr)rshutdown_tpu_systemr9r"r _shutdown_tpu_systemz1shutdown_tpu_system.._shutdown_tpu_systems  . . 00r"TzTPU system shut down.zTPU system fails to shut down.N)r functionr is_tfrt_enabledlogginginfowarning)r`successs r r_r_sQ11'.&=&=&? "T'  LL() OO45r"TFc &tjfd}tjd}tjdt j zdz5|}ddd|rt jddz}t jdt||z }tjtggt|ftt j} tjttj g| } t#j$| g } t jd | j'|t)j*|gt(j, } t/t1|D]} || | ||z| z<t3j4| g} t7j8| }| g|g|d z zz} tj| j:5| j=| | } t?j@| d g} | jC| }dddtEd jG} t jd| | jId| | a%| j'| }|Dcgc]}tM|d |d |d|d}}|a't jd|tQ|||}||jS|| fS#1swYxYw#1swYxYwcc}w)z8A helper function to initialize multi-client tpu system.c0tjS)Nuse_tfrt_host_runtime)r#configure_and_initialize_global_tpurisr _tpu_init_fnz,tpu_system_init_helper.._tpu_init_fns  > >3 55r"c.tj|yr)rd_tensor_set_global_tpu_array)topology_protos r _set_global_tpu_array_fnz8tpu_system_init_helper.._set_global_tpu_array_fns11.Ar"z/job:z/device:TPU_SYSTEM:0NzUsing TPU megacorezTPU core IDs: %s)mesheszTPU core locations: %s)dtyperOr)axiszAll TPU core IDs: %szAll TPU core locations: %s)*r rardevicer full_job_namercrdr8 layout_libMesh _MESH_DIM_XrI_TPU_DEVICE_TYPE client_idLayout UNSHARDEDr DTensorDevicetpu_core_ids_to_locationsr<zerosint32rAr@r constantr zeros_likenamepackr reduce_sumunpacklistnumpyset_tpu_core_ids _all_core_idsr_all_core_locationsr\ serialized)task_idrK num_devicesrj use_megacorerlrp my_core_idsrLmeshr rw all_core_idsirunpacked_all_tpu_idsall_core_locationsrW tpu_topologys ` r tpu_system_init_helperrs55BB zz'F00225KKL!.K! LL%&/K ,,!;/[945 + D. ~?O/5/?/?/AC D$   k:+?+?@$ G&  ' 'v 6& ,,'// <>;-rxx8, [! "Fa7B1~L//!34F%%|n5,   | ,%5'-AA-E"FF,  zz&++7;;|V4L&&|1#>L!==67 *1-3356, ,,%|4 "l+ -77 E5G01mAaD!A$!ad++ ,,+-?@%&8)&:<,<2245 v o!!j77.sK54 z -> y -> x order (fatest to slowest varying): [_CoreLocation(0, 0, 0, 0), _CoreLocation(0, 0, 0, 1), _CoreLocation(0, 1, 0, 0), _CoreLocation(0, 1, 0, 1), _CoreLocation(1, 0, 0, 0), _CoreLocation(1, 0, 0, 1), _CoreLocation(1, 1, 0, 0), _CoreLocation(1, 1, 0, 1)] Raises: ValueError: If ring_size cannot be fulfilled without splitting hosts. rvrOrqzUnsupported TPU slice size: %srr)rOrOrOrOrz;Rings too small for can_split_host_across_rings = False: %d ring_bounds z should be >= host_bounds rP) ValueErrorr<prodrpoprrAindexr)rrrrrnum_cores_per_chiprtrrr host_bound host_sizer ring_boundcoresrJrs r _enumerate_core_locationsrQsDHay1!3q!8 5> ??AE E1 -d 3 E$ E# $!F1I $& $!K wwv!=ABAq!/03BKB>BBAq!/03BKBs*3jjnz123nn)** * E   *..AQ.+.; !;0 11s*3jjnz123 .. 6; K% '%.Md).q 2ADA  2D 2-Qa$q'47KLM _ F $CC/" 3s"F! F&*F+F0 F5F:rotatecFttt}|s|Stjddt Dcgc]}|j c}t Dcgc]}|jc}ttDcgc]}|jc}dk(}tjddtjddtjdd|dtdtd tffd dtdtd tffd }|r&|jtj n%|jtj| tjdd ||Scc}wcc}wcc}w)aReorders a list of TPU cores to optimize for AllReduce performance. This is ported from the C++ tensorflow::BuildAllReduceRing function, mixed with some logic from TF TPU's device_assignment._ring_3d. Args: core_locations: A list of core locations expressed as [x, y, z, core]. rotate: If true, scan the cores in a column-major order. False by default. Returns: A permutation of the input list such that neighbors in the sequence are nearby in the TPU topology. rqzCore locations in: %srOzfirst_column: %dz first_row: %dz same_z: %siaibrMcj s|}|}|jk(xr|jk7}|jk(xr|jk7}||k7r|rdSdS|j|jk7r4|r|j|jz S|j|jz S|j|jk7rD|jdzdk(r|j|jz S|j|jz S|j|jz S|}|}|jk(xr|jk7}|jk(xr|jk7}||k7r|rdSdS|j|jk7r4|r|j|jz S|j|jz S|j|jk7rD|jdzdk(r|j|jz S|j|jz S|j|jz SNrrOrqr)rrr) rraba_firstb_firstrJ first_column first_rowrs r _cmp_2dz'_build_all_reduce_ring.._cmp_2ds   a  a $9 )9g $9 )9g G r#!# #qssQSSy2qss2 CC!GqLqssQSSy7accACCi7VVaff_  a  a !9acc\&9g !9acc\&9g G r#!# #qssQSSy2qss2 CC!GqLqssQSSy7accACCi7VVaff_r"c|}|}|jk(xr|j k(}|jk(xr|j k(}|rM|rK|j|jk7r|j|jz S|j|jz S||k7r|rdSdS|j|jk(r%|jdzdk(r ||S|| S|j|jz Srr) rrrra_cornerb_cornerrrJrrs r _cmp_3dz'_build_all_reduce_ring.._cmp_3dsrArA|#8y(8H|#8y(8HH##*QSS133Y9!&&166/98R"" ssaccz !a1 WR_B72r?2BB 339r")keyzPermutation out: %s)rrAr@rcrrSrrsetrr8sort functools cmp_to_key) rJr permutationrWsame_zrrrrs `` @@@r _build_all_reduce_ringrsX U3~./0+   ,,q)>:>2aacc23,/1133/0) .1QQSS12 3q 8& ,,q$l3 ,,q/9- ,,q,'(#(3(3((T#332 --g67--g67 ,,q'5 _3/1sF*FF rotate_ring_across_ringsct|}t|d|}td||D]*}t|Dcgc] }||||zc}||||z,tjdd|g}t|D]&}|td||Dcgc] }|||z c}z }(t ||z } t|d| |}td|| D]*}t| Dcgc] }||||zc}|||| z,g} t| D]&}| td|| Dcgc] }|||z c}z } (tjdd| | Scc}wcc}wcc}wcc}w)aBuild two all-reduce rings orthogonal to each other. One ring includes every `ring_size` consecutive core locations. It is usually applied to the model-parallel dimension of a mesh to achieve best 1D all-reduce performance. The other ring includes core locations separated by a stride of `ring_size`. It is usually applied to the data-parallel dimension of a mesh to get predictable strided all-reduce performance. Args: core_locations: A list of core locations expressed as [x, y, z, core]. ring_size: The number of core locations in the consecutive ring. rotate_ring_across_rings: Build column-major secondary rings. Returns: A permutation of the input list forming the described rings. NrrOPermutated core locations: %s)rz$Stride-permutated core locations: %s)r@rrArcrr8) rJrr num_coresrrr transposedg num_rings untransposeds r _build_orthogonal_ringsrs*.!)&~jy'AB+ Iy )a49)4D'/0q;q>)*'N1Q]# ,,q1>B *  a',Q 9'E"#q1uJ )i'()&)%=?+ Iy )a05i0@#+, 1{1~%&#JqY ,  PaaI0NO1ZA&OOLP ,,q8,G 7'# PsE E EEz$experimental.dtensor.create_tpu_mesh)v1mesh_dim_namesrQ mesh_name ring_dims ring_axesbuild_ring_across_rings use_xla_spmdc tjd||tjd|tjd|tjd|tjd||sdtjz}tjd||d t|z }n+|t| ks|t|kDrt d |ztjd ||gd }n:t|d k7rt d|zt |gd k7rt d|ztjd|t t dtt j} || }n2t|d k7rt d|z|| kDrt d|d| tjd||dkDrtj|d|} n |dkrtj||d} nd } tjd| t| |||| } tjd d| t| } tj|}|| k7rt d|| fz|rt| | |} n_t| d| }td| | D]*}t| Dcgc] }| |||zc}| ||| z,tjd d| | Dcgc]}|j!} }t" t dt"j%| }t"j'||t j(t+j,}|Dcgc]}| j/t|}}t1|t2d|\}}}t5j6|||||| Scc}wcc}wcc}w)aJReturns a distributed TPU mesh optimized for AllReduce ring reductions. Only as many as leading axes specified by `ring_axes` as necessary will be used to build rings, as long as the subslice formed by these axes have enough cores to contain a ring of the required size. The leftover axes in `ring_axes` won't affect results. This function always uses all TPU devices, and offers more customization than `tf.experimental.dtensor.create_distributed_mesh`. Args: mesh_dim_names: List of mesh dimension names. mesh_shape: Shape of the mesh. mesh_name: A unique name for the mesh. If empty, internally generate one. ring_dims: Optional; The number of leading (ring_dims > 0) or trailing (ring_dims < 0) mesh dimensions to build rings for. If unspecified, build rings for all but the first dimension. ring_axes: Optional; A permutation of ["x", "y", "z", "core"], specifying the order of TPU topology axes to build rings in. If unspecified, default to ["core", "x", "y", "z"]. ring_bounds: Optional; The maximum number of devices on each axis, in the x, y, z, core order. If unspecified, default to physical topology limits. can_split_host_across_rings: Optional; If true, devices attached to the same host (i.e., DTensor client) may get assigned to different rings. Setting it to false may cause some combinations of arguments to be infeasible; see DeviceAssignmentTest.testCreateMesh[No]SplittingHosts* for examples. build_ring_across_rings: Optional; If true, also build a data-parallel ring across model-parallel rings. This ring could be strided. rotate_ring_across_rings: Optional; If true, build the data-parallel ring in column-major instead of row-major order. use_xla_spmd: Boolean when True, will use XLA SPMD instead of DTensor SPMD. z"Building a TPU mesh %s of shape %szRequested ring_dims: %szRequested ring_axes: %szRequested ring_bounds: %sz)Requested can_split_host_across_rings: %szmesh_%fzRequested mesh_name: %sNrOzInvalid ring_dims value: %dzActual ring_dims: %s)rrrrrPz(Expected 4 elements in ring_axes, got %szInvalid ring_axes value: %szActual ring_axes: %sz?Invalid TPU topology, run dtensor.initialize_tpu_system() firstz*Expected 4 elements in ring_bounds, got %srz should be <= topology sizes zActual ring_bounds: %srzActual ring_size: %dzEnumerated core locations: %sz?Invalid mesh size: mesh shape %s cannot 1:1 map to %d TPU coresrzHInvalid system device, run dtensor.initialize_accelerator_system() first)rE)r)rcrdtimer@rsortedrrrQr<rrrrrrAr4rtpu_core_locations_to_idsrrRrr}rrIr|ryrz)rrQrrrrrrrrtopology_shaperglobal_core_locationsr mesh_sizerrrrWglobal_core_idslocal_core_locationslocal_core_locationindexesrFrErGs r create_tpu_meshrIs\ ,,3Y K ,,()4 ,,()4 ,,*K8 ,,:*, DIIK'I ,,()4C O#IC O##y3z?'B 2Y> ?? ,,%y1'I 9~ ?)K LL i33 2Y> ?? ,,%y1 I KK 001. K ;1 AKO PP^# !>3 44 ,,'5] :I./I1} 9:./II ,,%y14k9.I ,,q13HI'()ggj!)) I Y  !! 34I4=4LN))>z )JKK 1i +=B9=M089 KN 2 30Aa)m, LL35JK1FF1199;FF I JJ#==/""9o>'99&:J:J:LM"6  !!$':";< ' rr}c|jtk7r td||tjk(r|j St d)aReturns the device IDs of all TPU cores local to the given client. A device ID is a non-negative integer that uniquely identifies a device in the mesh. For example, for a 2x2 mesh ('x', 'y'), this function returns a permutation of [0, 1, 2, 3]. Note that device IDs and device locations are equivalent. The former is a linearization of the latter along mesh dimensions. Args: mesh: A TPU mesh. client_id: Optional; A DTensor client ID. If empty, query this client. The mesh must be a TPU meshz5Looking up other clients' device IDs is not supported)rCr|rrr}rENotImplementedErrorrr}s r get_device_idsrsY  ++ 2 33)v'7'7'99  "" = ??r"c|jtk7r td||tjk(r|j St d)aMReturns the device locations of all TPU cores local to the given client. A device location is a dictionary from dimension names to indices on those dimensions. For example, for a 2x2 mesh ('x', 'y'), this function returns a permutation of this list: [{'x': 0, 'y': 0}, {'x': 0, 'y': 1}, {'x': 1, 'y': 0}, {'x': 1, 'y': 1}]. Note that device IDs and device locations are equivalent. The former is a linearization of the latter along mesh dimensions. Args: mesh: A TPU mesh. client_id: Optional; A DTensor client ID. If empty, query this client. rz;Looking up other clients' device locations is not supported)rCr|rrr}local_device_locationsrrs r get_device_locationsrs[, ++ 2 33)v'7'7'99  & & (( C EEr"c6ddlm}|jd|y)z,Deprecated way to initialize the TPU system.rOaccelerator_utilr)enable_coordination_serviceN)rur initialize_accelerator_system)r r s r dtensor_initialize_tpu_systemr "s 00 )D1Fr"c0ddlm}|jy)z*Deprecated way to shutodwn the TPU system.rOr N)rur shutdown_accelerator_systemr s r dtensor_shutdown_tpu_systemr)s ..0r"r)TF)F)=r7rrtypingrrrrr<tensorflow.dtensor.pythonrrrr rytensorflow.python.eagerr r tensorflow.python.frameworkr r rtensorflow.python.opsrrtensorflow.python.platformrrctensorflow.python.tpurtensorflow.python.utilr tensorflow.python.util.tf_exportrr{r|rrrrrrIr8rVr\r_rrrrboolrrr USE_XLA_SPMDrzrrrr rr9r"r rs* '',45:+03.++*<*/6   //: @Dm)<DD/2D7?7H7HD: 6"26(- gT.$b$T#Y$T#Y$!%c$9=c$!%c$/3DI$NXd3iXd3iX$(IX;?X*-X26m1DXx+0d4 +>d#'d48IdN3'3473"3'+M':3l 1b9 $%)'+(,$)%*#00UIUS UU} U S " U $s)$ U"&U"U#UU6@__U:Ur/3??&sm?7;Cy?< $E //E}E(,T#s(^(<EHF1r"