AVh)vdZddlmZddlmZddlmZdZdZ dZ dd Z d Z d Z d Zd ZdZdZdZdZy)z3Utilities for multi-worker distribution strategies.) cluster_pb2)distribute_coordinator_context) server_libct|ttjfrt j |St|tj s t d|S)a4Makes `cluster_spec` into a `ClusterSpec` object. Args: cluster_spec: a dict, ClusterDef or ClusterSpec object specifying the cluster configurations. Returns: a `ClusterSpec` object. Raises: ValueError: if `cluster_spec` is not a dict or a `ClusterSpec` or a `ClusterDef`. z[`cluster_spec' should be dict or a `tf.train.ClusterSpec` or a `tf.train.ClusterDef` object) isinstancedictr ClusterDefr ClusterSpec ValueError cluster_specs ^/home/dcms/DCMS/lib/python3.12/site-packages/tensorflow/python/distribute/multi_worker_util.pynormalize_cluster_specrsT t[%;%;<=  ! !, // lJ$:$: ;  ' (( cD |j|S#t$rYywxYw)Nr) num_tasksr r task_types r task_countr-s*  ! !) ,,   s  cdt|}tfd|jDrtdj ||vrtdj ||r!||jvr|dk7rtd|zt |ddkDr td t |ddkDr td ||jvr |t ||k\rtd ||fzy y ) aMValidates `cluster_spec`. It checks: 1) task type is one of "chief", "worker", "ps", "evaluator", or not provided (None). 2) whether there is such a task type as `task_type` in the `cluster_spec`. The only exception is `evaluator`. In other words, it is still a valid configuration when `task_type` is `evaluator` but it doesn't appear in `cluster_spec`. 3) whether there is at most one "chief" job. 4) whether there is at most one "evaluator" job. 5) whether the `task_id` is smaller than the number of tasks for that particular `task_type`. Args: cluster_spec: a dict, `ClusterDef` or `ClusterSpec` object to be validated. task_type: string indicating the type of the task. task_id: the id of the `task_type` in this cluster. Raises: ValueError: if `cluster_spec` fails any check. )chiefworker evaluatorpsNc3&K|]}|v yw)N).0joballowed_task_typess r z)_validate_cluster_spec..QsD3& &Dsz\Disallowed task type found in cluster spec. Allowed types are {} and the cluster spec is {}.z4Unrecognized task_type: {}, valid task types are: {}rz)`task_type` %r not found in cluster_spec.rz&There must be at most one 'chief' job.z*There must be at most one 'evaluator' job.z.The `task_id` %d exceeds the maximum id of %s.N)ranyjobsr formatr)r rtask_idrs @r_validate_cluster_specr&4s!2D' 5,D,2C2CDD @@F+\A; <<(( >EE ) + ,,I\%6%66; @9L MM g&* = >> k*Q. A BB,###:I4) 8GY;OO QQ)#rNctrtjjSt |||t |j }|dk(s|dk(ryd|vr |dk(r|dk(ryy)a Returns whether the given task is chief in the cluster. Since there is at most one evaluator and the evaluator itself should be independent of the training cluster, the evaluator job is also a chief job on its own. If this is currently running under a `_WorkerContext` of distribute coordinator, the arguments can be omitted as the result is already available. Args: cluster_spec: a dict, `ClusterDef` or `ClusterSpec` object specifying the cluster configurations. task_type: the task type in the cluster. task_id: the task id in the cluster. Returns: a boolean indicating whether the given task is chief. Raises: ValueError: if `task_type` is not in the `cluster_spec` or `task_id` exceeds the maximum id of the `task_type`. rrTrrF)has_worker_context dc_contextget_current_worker_contextis_chiefr&ras_dictr rr%s rr+r+lsm.  0 0 2 ; ;;y':' 5==?,'Y+5 \!i8&;1  rct|}|jsyt||||dk(ryd|jvryd|jvsJy)aqReturn the job name for the leader of for collective ops. Args: cluster_spec: a dict, `ClusterDef` or `ClusterSpec` object specifying the cluster configurations. task_type: the task type in the cluster. task_id: the task id in the cluster. Returns: a string indicating the leader job name or empty string if no need to set leader job. rr/job:chief/replica:0/task:0r/job:worker/replica:0/task:0)rr,r&r#r-s rcollective_leaderr2sd( 5,     y':+  !!! ( \&& && & 'rct|}|jsyd|jvryd|jvryd|jvsJy)aReturn the task name of the coordination service leader. Args: cluster_spec: a dict, `ClusterDef` or `ClusterSpec` object sxpecifying the cluster configurations. Returns: a string indicating the task name of the coordination service leader. r/rz/job:ps/replica:0/task:0rr0rr1)rr,r#r s rcoordination_leaderr4s^( 5,      \   % !!! ( \&& && & 'rct||dt|j}|dvrtd|z|dk(rt |dSt |j dgt |j dgzS)z-Returns the number of workers in the cluster.r)r%)rrrzUnexpected `task_type` %rrrr)r&rr,r lengetrs r worker_countr8sy!<' 5==?,66 09< ==+ |K( ))    "- .2&2( ()rct|||t|j}|dk(ry|dk(r|t|j dgzS|dk(r|St d|z)aEReturns a unique id for the task in the `task_type`'s cluster. It returns an id ranging from [0, `worker_count(task_type, task_id)`). Note: this function assumes that "evaluate" job is in its own cluster or its own partition of a cluster. Args: cluster_spec: a dict, `ClusterDef` or `ClusterSpec` object to be validated. task_type: string indicating the type of the task. task_id: the id of the `task_type` in this cluster. Returns: an int indicating the unique id. Throws: ValueError: if `task_type` is not "chief", "worker" or "evaluator". rrrrzThere is no id for task_type %r)r&rr,r6r7r r-s r id_in_clusterr:sw&y':' 5==?,' ( S))'267 77+ N 4y@AArc>tjjS)aReturns whether the current worker should save checkpoints. In multi-worker training, if saving checkpoint is requested by user, or needed for fault-tolerance, the cluster should save checkpoint but not necessarily every worker in the cluster should. TODO(rchao): Consider generalizing this util to be `should_save_file` as there can be other files to save such as summary. Returns: Whether this particular worker in the cluster should save checkpoints. )r)r*should_checkpointrrrshould_save_checkpointr=s  . . 0 B BBrc>tjjS)a^Returns whether the current worker should load checkpoints. In multi-worker training, if loading checkpoint is requested by user, or needed for fault-tolerance, the cluster should load checkpoint but not necessarily every worker in the cluster should. Returns: Whether this particular worker in the cluster should load checkpoints. )r)r*experimental_should_initrrrshould_load_checkpointr@s  . . 0 I IIrcFtjjS)z>Waits for other workers to reach the same call to this method.)r)r*wait_for_other_workersrrrrBrB+s  . . 0 G G IIrc.tjduS)z2Returns whether a worker context has been entered.N)r)r*rrrr(r(0s  . . 0 <rHsX:0U1. 5Qp%P(D(:)*#BL C JJ =r