AVhAYdZddlmZ ddZdZ ddZ ddZ ddZ dd Z dd Z dd Z dd Z dd Z ddZ ddZddZy)zTensorFlow collective Ops.)gen_collective_opsc |dkrtd|dtj||||||||j| S)aReduces tensors collectively, across devices. Args: t: the tensor to be reduced. group_size: the total number of tensors to be collectively reduced. Each must reside on a different device. Should be a positive integer. group_key: an integer identifying the group of devices. instance_key: an integer identifying the participating group of Ops. merge_op: string naming the binary Op to be applied to compute each partial reduction. final_op: string naming the unary Op to be applied to each fully reduced value. Can be 'Id' for no operation. subdiv_offsets: a list of integer offsets into the tensor at which each independent subdivision should begin. Use [0] if no subdivision should be done. communication_hint: preferred collective communication. The implementation may fall back to another mechanism. Options include `auto`, `ring`, and `nccl`. timeout: a float. If set to a non zero, set a completion timeout to detect staleness. If the timer goes off, a DeadlineExceededError is raised. The timeout value in seconds. This feature is experimental. Returns: An Op implementing the distributed reduction. Raises: ValueError: if any of the input parameter constraints are not met. zCParameter `group_size` to all_reduce must be at least 1. Received: .) group_size group_key instance_keymerge_opfinal_opsubdiv_offsetscommunication_hinttimeout_seconds) ValueErrorrcollective_reducelower) trrr r r r r timeouts T/home/dcms/DCMS/lib/python3.12/site-packages/tensorflow/python/ops/collective_ops.py all_reducersaJ!^ "",Q0 11  - -#+113  c@tj|||\}}||fS)aAssign group key based on group_assignment. Args: group_assignment: a 2 dimensional integer Tensor that encodes which devices belong to the same group. The values are indices of the devices within 0 to number of devices. device_index: integer for the index of the current device base_key: integer to offset the resulted group_key. The base key shall be unique for different values of group_assignment in the same tf.function. Notes: The device_index argument must be consistent with the index of the device of this Op in the device assignment list. The behavior of this Op is undefined if they are inconsistent. Returns: group_size, group_key: The group size and group key for the current device. )group_assignment device_indexbase_key)rcollective_assign_group_v2)rrrrrs rassign_group_v2rGs/"-GG'*i Y rNc p||g}ng}tj|||||||j|d|| |  S)aReduces tensors collectively, across devices. Args: t: the tensor to be reduced. group_size: an int32 tensor. The total number of tensors to be collectively reduced. Each must reside on a different device. Should be a positive integer. group_key: an int32 tensor identifying the group of devices. instance_key: an int32 tensor identifying the participating group of Ops. merge_op: string naming the binary Op to be applied to compute each partial reduction. final_op: string naming the unary Op to be applied to each fully reduced value. Can be 'Id' for no operation. communication_hint: preferred collective communication. The implementation may fall back to another mechanism. Options include `auto`, `ring`, and `nccl`. timeout: a float. If set to a non zero, set a completion timeout to detect staleness. If the timer goes off, a DeadlineExceededError is raised. The timeout value in seconds. This feature is experimental. ordering_token: a resource tensor on the same device as the op to order the collectives in a per-device manner by auto control dependency. This argument can be omited when there is one collective Op per `tf.function`, or when explicit control dependency is used instead of auto control dependency. max_subdivs_per_device: int specifying the maximum number of subdivisions a tensor on a device can be divided into. The runtime uses this contraint to parallelize processing of each per-device tensor. Setting to -1 disables subdivision and reverts to previous behavior of not sub-dividing tensor. Setting to 0 uses sytem defaults. name: name of the Op. Returns: An Op implementing the distributed reduction. F) rrr r r r r is_statelessordering_tokenmax_subdivs_per_devicename)rcollective_reduce_v2r) rrrr r r r rrr r!s r all_reduce_v2r#_sXZ$%NN  0 0+113#3   rc |dkrtd|dtj|dg||||j|S)aAccumulates tensors collectively, across devices, along first dimension. Args: t: the tensor to participate in the accumulation. group_size: the total number of tensors to be collectively accumulated. Each must reside on a different device. Should be a positive integer. group_key: an integer identifying the group of devices. instance_key: an integer identifying the participating group of Ops. communication_hint: preferred collective communication. The implementation may fall back to another mechanism. Options include `auto`, `ring`, and `nccl`. timeout: a float. If set to a non zero, set a completion timeout to detect staleness. If the timer goes off, a DeadlineExceededError is raised. The timeout value in seconds. This feature is experimental. Returns: An Op implementing the distributed operation. Raises: ValueError: if any of the input parameter constraints are not met. rzCParameter `group_size` to all_gather must be at least 1. Received: rrshaperrr r r)rrcollective_gatherrrrrr r rs r all_gatherr)s\6!^ ##-,a1 22  - - C+113 rc j||g}ng}tj|||||j|d|| S)aAccumulates tensors collectively, across devices, along first dimension. Args: t: the tensor to participate in the accumulation. group_size: an int32 tensor, the total number of tensors to be collectively accumulated. Each must reside on a different device. Should be a positive integer. group_key: an int32 tensor identifying the group of devices. instance_key: an int32 tensor identifying the participating group of Ops. communication_hint: preferred collective communication. The implementation may fall back to another mechanism. Options include `auto`, `ring`, and `nccl`. timeout: a float. If set to a non zero, set a completion timeout to detect staleness. If the timer goes off, a DeadlineExceededError is raised. The timeout value in seconds. This feature is experimental. ordering_token: a resource tensor on the same device as the op to order the collectives in a per-device manner by auto control dependency. This argument can be omited when there is one collective Op per `tf.function`, or when explicit control dependency is used instead of auto control dependency. name: name of the Op. Returns: An Op implementing the distributed operation. Frrr r rrrr!)rcollective_gather_v2rrrrr r rrr!s r all_gather_v2r.sOB$%NN  0 0+113#   rc *|dkrtd|d|j|k7rtd|jd|d|j|k7rtd|jd|dtj||||||j |S)aBroadcasts one tensor to a group of others, across devices. Args: t: the tensor to be sent. shape: the shape of the tensor being sent, which must agree with t. dtype: the type of the tensor being sent, which must agree with t. group_size: one plus the number of receiving tensors, i.e. the total number of devices participating. Each tensor must reside on a different device. group_key: an integer identifying the group of devices. instance_key: an integer identifying the participating group of Ops. communication_hint: preferred collective communication. The implementation may fall back to another mechanism. Options include `auto`, `ring`, and `nccl`. timeout: If set to a non zero, set a completion timeout to detect staleness. If the timer goes off, a DeadlineExceededError is raised. The timeout value in seconds. This feature is experimental. Returns: An Op implementing the distributed broadcast send. Raises: ValueError: if any of the input parameter constraints are not met. Note that the shape and dtype arguments appear redundant since they should be obtainable from t. The are two reasons for including them. First, the shape and type of tensors passed via broadcast must be known ahead of time in their most specific form so that the receive side can allocate memory for the operation and shape/type inference can carry forward from there. Including the same declarations on the send side clarifies a commitment already made. Secondly, having nearly identical use syntax for send and receive sides may simplify tool-driven generation of broadcast. rGParameter `group_size` to broadcast_send must be at least 2. Received: rzIShape of broadcast_send tensor `t` not equal to declared shape. Received z , expected zGType of broadcast_send tensor `t` not equal to declared type. Received r%)rr&dtypercollective_bcast_sendr)rr&r1rrr r rs rbroadcast_sendr3sT1_  L # $$WW  GG9Kwa 1 22WW  GG9Kwa 1 22  1 1 +113 rcTtj|||||j|S)asBroadcasts one tensor to a group of others, across devices. Args: t: the tensor to be sent. group_size: an int32 tensor. One plus the number of receiving tensors, i.e. the total number of devices participating. Each tensor must reside on a different device. group_key: an int32 tensor identifying the group of devices. instance_key: an int32 tensor identifying the participating group of Ops. communication_hint: preferred collective communication. The implementation may fall back to another mechanism. Options include `auto`, `ring`, and `nccl`. timeout: If set to a non zero, set a completion timeout to detect staleness. If the timer goes off, a DeadlineExceededError is raised. The timeout value in seconds. This feature is experimental. Returns: An Op implementing the distributed broadcast send. )rrr r r)rcollective_bcast_send_v2rr(s rbroadcast_send_v2r6:s22  4 4+113  rc ~|dkrtd|dtj||||||j|S)aReceives a broadcasts tensor, across devices. Args: shape: Shape of the tensor to be received. dtype: Type of the tensor to be received. group_size: one plus the number of receiving tensors, i.e. the total number of devices participating. Each tensor must reside on a different device. group_key: an integer identifying the group of devices. instance_key: an integer identifying the participating group of Ops. communication_hint: preferred collective communication. The implementation may fall back to another mechanism. Options include `auto`, `ring`, and `nccl`. timeout: If set to a non zero, set a completion timeout to detect staleness. If the timer goes off, a DeadlineExceededError is raised. The timeout value in seconds. This feature is experimental. Returns: An Op implementing the broadcast receive. Raises: ValueError: if any of the input parameter constraints are not met. rr0r)r&Trrr r r)rrcollective_bcast_recvrr&r1rrr r rs rbroadcast_recvr;\s\<1_  L # $$  1 1  +113 rc Vtj||||||j|S)aReceives a broadcasts tensor, across devices. Args: shape: an int tensor. Shape of the tensor to be received. dtype: Type of the tensor to be received. group_size: an int32 tensor. One plus the number of receiving tensors, i.e. the total number of devices participating. Each tensor must reside on a different device. group_key: an int32 tensor identifying the group of devices. instance_key: an int32 tensor identifying the participating group of Ops. communication_hint: preferred collective communication. The implementation may fall back to another mechanism. Options include `auto`, `ring`, and `nccl`. timeout: If set to a non zero, set a completion timeout to detect staleness. If the timer goes off, a DeadlineExceededError is raised. The timeout value in seconds. This feature is experimental. Returns: An Op implementing the broadcast receive. )r8rrr r&r r)rcollective_bcast_recv_v2rr:s rbroadcast_recv_v2r>s56  4 4  +113 rc6tj|||||S)aInitializes a collective communicator. This creates a collective communicator, which represents membership to a collective group identified by the group_key. It should be called once per member of the group, and each member needs to be on a different device. It blocks until all members of the group run this op. Communicators of a group can only be initialized once. Trying to initialize communicators for an existing group key will result in an error. Args: group_key: an int32 `tf.Tensor` identifying the group. rank: an `tf.Tensor` specifying the rank of this device in the group. If specified, the rank is required to be unique in the group. group_size: an int32 `tf.Tensor`. The size of the group. communication_hint: preferred collective communication. The implementation may fall back to another mechanism. Options include `auto`, `ring`, and `nccl`. timeout_seconds: If set to a non zero, set a completion timeout to detect staleness. If the timer goes off, a DeadlineExceededError is raised. The timeout value in seconds. This feature is experimental. Returns: A resource `tf.Tensor`. rrankrr r)r"collective_initialize_communicatorr@s rinitialize_communicatorrCs&>  > > +%  ''rc>|g}tj|||||S)aReduces tensors mutually. Args: communicator: the resource `tf.Tensor` returned from `initialize_communicator`. t: the `tf.Tensor` to be reduced. reduction: a string. The name of the operation to reduce the values. Accpeted values are `"min"`, `"max"`, `"mul"`, `"add"`. group_assignment: Optional int32 `tf.Tensor` with shape [num_groups, num_ranks_per_group]. `group_assignment[i]` represents the ranks in the `ith` subgroup. timeout_seconds: If set to a non zero, set a completion timeout to detect staleness. If the timer goes off, a DeadlineExceededError is raised. The timeout value in seconds. This feature is experimental. Returns: The reduced `tf.Tensor`. ) communicatorinputr reductionr)rcollective_reduce_v3)rErrGrrs r all_reduce_v3rIs2.  0 0 '%  ''rc j||g}ng}tj|||||j|d|| S)aExchanges tensors mutually. Args: t: a `tf.Tensor`. The first dimension should have the length as the size of the group. `t[i]` is sent to `rank i` within the group. group_size: an int32 tensor, the total number of tensors to be mutually exchanged. Each must reside on a different device. Should be a positive integer. group_key: an int32 tensor identifying the group of devices. instance_key: an int32 tensor identifying the participating group of Ops. communication_hint: preferred collective communication. The implementation may fall back to another mechanism. Options include `auto` and `nccl`. timeout: a float. If set to a non zero, set a completion timeout to detect staleness. If the timer goes off, a DeadlineExceededError is raised. The timeout value in seconds. This feature is experimental. ordering_token: a resource tensor on the same device as the op to order the collectives in a per-device manner by auto control dependency. This argument can be omited when there is one collective Op per `tf.function`, or when explicit control dependency is used instead of auto control dependency. name: name of the Op. Returns: An Op implementing the distributed operation. Fr+)rcollective_all_to_all_v2rr-s r all_to_all_v2rLsOF$%NN  4 4+113#   rc<|g}tj||||S)aExchanges tensors mutually. Args: communicator: the resource `tf.Tensor` returned from `initialize_communicator`. t: a `tf.Tensor`. The first dimension should have the length as the size of the group. `t[i]` is sent to `rank i` within the group. group_assignment: Optional int32 `tf.Tensor` with shape [num_groups, num_ranks_per_group]. `group_assignment[i]` represents the ranks in the `ith` subgroup. timeout_seconds: If set to a non zero, set a completion timeout to detect staleness. If the timer goes off, a DeadlineExceededError is raised. The timeout value in seconds. This feature is experimental. Returns: a `tf.Tensor`. `t[i]` is sent from `rank i` within the group. )rErFrr)rcollective_all_to_all_v3)rErrrs r all_to_all_v3rO*s/$  4 4 '%  ''r)AddId)rautor)rPrQrRrNN)rRr)rRrNN)rPNN)NN)__doc__tensorflow.python.opsrrrr#r)r.r3r6r;r>rCrIrLrOrrrWs!4""(1h8!%+!%)+>J#) %X&,!%/p'-=H*0 N'- )b*0 "P06,- $'R"#'"& 'L  2j'r