AVh~)dZddlZddlZddlZddlZddlZddlZddlmZddl m Z ddl m Z ddl m Z ddl mZddl mZdd l mZdd lmZdd lmZdd lmZdd lmZddlmZddlmZddlmZ ddlZej<GddZdZ dZ!dZ"dZ#dZ$d#dZ%dZ&dZ'dZ(dZ)dZ*dZ+d$dZ,d Z-d!Z.d"Z/y#e$rdZYUwxYw)%zTest utilities.N)app) v2_compat)collective_all_reduce_strategy)multi_process_runner)multi_worker_test_base) tpu_strategy)values)context)config)ops)tensor) array_ops)array_ops_stack)nestc,eZdZUeed<eed<eed<y)TestClusterParamsclustermax_num_worker max_num_psN)__name__ __module__ __qualname__dict__annotations__intV/home/dcms/DCMS/lib/python3.12/site-packages/tensorflow/python/distribute/test_util.pyrr.s -/rrc||jkDs||jkDr td|j0t j |j|j|_|jdd||jdd|dS)Nz]Requesting more servers than the maximum, adjustcluster params' max_num_ps and max_num_worker) num_workersnum_psworkerps)r"r#)rr ValueErrorrrcreate_in_process_cluster)cluster_paramsr r!s rget_cluster_defr'5sN111 ~((( E FF#3MM"11((*N&&x0+>  " "4 (& 1 rc^tjtjt||S)aGathers value from all workers. This is intended for tests before we implement an official all-gather API. Args: strategy: a `tf.distribute.Strategy`. value: a nested structure of n-dim `tf.distribute.DistributedValue` of `tf.Tensor`, or of a `tf.Tensor` if the strategy only has one replica. Cannot contain tf.sparse.SparseTensor. Returns: a (n+1)-dim `tf.Tensor`. )r map_structure functoolspartial_gather)strategyvalues rgatherr/Ds$   I--gx@% HHrc.t|tjs)tjt j |g}t|j tjstj|jSt|j jt|jk(sJ|jDcgc]}tj|d}}|j!tj|dScc}w)zGathers a single value.r)axis) isinstancer DistributedValues PerReplicar convert_to_tensorextendedrCollectiveAllReduceExtendedrstack_valueslenworker_devicesrexpand_dims_v2r/)r-r.vinputss rr,r,Us E633 4   s44U;< =E H%%2NN P   // X   - - .#emm2D DD D9> GAI $ $QQ / G& G **62 ;; Hs Dc|dkrtd|tj|}|stdj |t ||k\ry|t |z dz}g}t |D]^}|jdk(r&|jtjd<|jtj`tj|d|y) z2Create logical devices of at least a given number.z`num` must be at least 1 not z No {} foundNGPUi) memory_limit) r$r list_physical_devices RuntimeErrorformatr:rangeupperappendr LogicalDeviceConfiguration set_logical_device_configuration)devicenumphysical_deviceslogical_devices_s rset_logical_devices_to_at_leastrQcs1W #? @@11&9  }++F3 44 c!  c"##a'#/ :Ca ||~  , ,$ ?AW??AB C ))*:2*>Prctjdr tddtjdr tddyy)NrACPU)r rDrQrrr_set_logical_devicesrU|s7 !!%(#E1- !!%(#E1-)rc|rtjt|rtjntj t jy)z1All-in-one main function for tf.distribute tests.N)rcall_after_initrUrenable_v2_behaviordisable_v2_behaviorr test_main)rXconfig_logical_devicess rmainr\s;,-   " !!#  "rcg}tj|j|jD]U}t |t j r |j}t |tjsJ|j|W|S)zEReturns the data and control dependencies of a tf.Operation combined.) itertoolschainr>control_inputsr2r Tensoropr OperationrI)rbdepsnodes r_op_dependenciesrfsi $oobii):):;d$ & WWd dCMM ** *KK  +rctj}t|D]<}||vrd||<tt|D]}|j |ddz||<>g}|j D]\}}|dk(s |j |i}d}|rW|d|dd}}|||<|dz }tt|D])}||xxdzcc<||dk(s|j |+|rWt|t|k(sJ|S)aTopological sorts a list of operations. This does a topological sort of the operations in a graph. The edges include both data dependencies and control dependencies. Note that the edge goes from an operation to its dependencies. The sort is intentionally unstable, reversing orders of operations and dependencies on ties. Args: operations: a list of tf.Operation in the same graph. Returns: A map from a tf.Operation to its topological order. rr@N) collections OrderedDictreversedrfgetitemsrIr:) operations in_degreesrbnext_opnexts in_degreeorder next_orders rtopological_sort_operationsrts: &&(* Z ;b jn,R01;&NN7A6:j;; %!'')mb)A~ ll2 %* a%)BE"I!OJ,R01Q G  ! W  Us: && & ,rcx|g}|r5|d|dd}}t|D]}||k(ry|j||r5y)zBReturns whether there exists a dependency chain from start to end.rr@NTF)rfrI)startendrprbros r_exists_dependencyrxsS '% a%)B#B' C ll7  rc Dt|fd}tt|dz D]u}t||||dzrt ||j j tdj||j||dzjy)aAsserts there's a deterministic execution order between the operations. Args: order: a map from a tf.Operation to its topological order. operations: a list of operations that should be executed sequentially. It can be given in any order. c|S)Nr)rbrrs rz-assert_sequential_execution..s rr)keyr@z;No dependency between {} and {}. Graph is dumped to stdout.N) sortedrGr:rxprintgraph as_graph_defAssertionErrorrFname)rrrmis` rassert_sequential_executionrsj&:;* Z1$ %;a jmZA-> ? JqM   , , ./  G N Nm  *QU"3"8"8 : ;;;rct}tjD]*}|j|j |j,|S)z*Returns a set of all running thread names.)set threading enumerateradd)running_threadsthreads rget_running_threadsrsEE/##%'f {{&++&' rc8|D]}|j|syy)zReturns whether any 'running_threads' is prefixed with 'prefix'. Args: prefix: The prefix of the expected thread name. running_threads: A collection of the running thread names. TF) startswith)prefixrrs r has_threadrs) f   rct tdtj}tj||||j }|j |S)aBReturns a dot graph of all the objects that are referencing the target. A object referencing graph is useful to debug memory leak like circular reference. objgraph provides a good visualization of the memory graph than most python built-in utilities like gc.get_referrers(), which are not human-readable sometimes. The dot graph will be written to a string IO object, and can be rendered with graphviz in operating system. E.g. dot -Tpng {$dot_graph} -o output.png Args: target: The target object for the memory graph. max_depth: The maximum depth of the graph. By default 3 layers of references are used. Increases this a lot may result in the graph growing too big. Returns: A string that contains the object reference graph. Raises: NotImplementedError: if objgraph is not installed. zobjgraph is not installed.) max_depthoutput)objgraphNotImplementedErrorioStringIO show_backrefsgetvalueclose)targetr string_iors r show_backrefrsP* : ;;kkm) 9YG    % // ,rct|jjt|k7r tdg}t |jj|D]G\}}t j |5|jt j|dddItj|S#1swYhxYw)z4Creates a PerReplica of Tensors from the value_list.zEthe length of values must be the same as the number of worker devicesN) r:r6r;r$zipr rLrIr5r r4)r- value_listtensorsrLr.s rcreate_per_replicars   ) )*c*o= O QQ '8,,;;ZH3mfe F 3 nnS**512333   7 ##33s 4%B88C ctt|tjtjtjfS)z/Returns whether the strategy is a TPU strategy.)r2r TPUStrategy TPUStrategyV1 TPUStrategyV2)r-s ris_tpu_strategyr$s0 H!--|/I/I!//1 22rc,tjy)zResets eager context.N)r _reset_contextrrr reset_contextr+s r)TT))0__doc__rh dataclassesr*rr^rabslrtensorflow.python.compatrtensorflow.python.distributerrrrr tensorflow.python.eagerr tensorflow.python.frameworkr r r tensorflow.python.opsrrtensorflow.python.utilrr ImportError dataclassrr'r/r,rQrUr\rfrtrxrrrrrrrrrrrs .G=?5/+.+.+1'   I" <Q2.#%P ;. < $2C (s0CC  C