AVh"dZddlZddlZddlmZddlmZeddGddejZeZ ed Gd d e Z Gd d e Z edGdde Z y)zUtilities for collectives.N) deprecation) tf_exportz3distribute.experimental.CommunicationImplementationz/distribute.experimental.CollectiveCommunicationceZdZdZdZdZdZy)CommunicationImplementationu Cross device communication implementation. Warning: The alias `tf.distribute.experimental.CollectiveCommunication` is deprecated and will be removed in a future version. Use `tf.distribute.experimental.CommunicationImplementation` instead. * `AUTO`: Automatically chosen by Tensorflow. * `RING`: TensorFlow's ring algorithms for all-reduce and all-gather. * `NCCL`: NVIDIA®'s NCCL library. This is now only used for all-reduce on GPUs; all-reduce on CPU, all-gather and broadcast fallbacks to RING. AUTORINGNCCLN)__name__ __module__ __qualname____doc__rrr \/home/dcms/DCMS/lib/python3.12/site-packages/tensorflow/python/distribute/collective_util.pyrrs  $ $ $rrz,distribute.experimental.CommunicationOptionsc8eZdZdZdZddej fdZy)_OptionsExportedaOptions for cross device communications like All-reduce. This can be passed to methods like `tf.distribute.get_replica_context().all_reduce()` to optimize collective operation performance. Note that these are only hints, which may or may not change the actual behavior. Some options only apply to certain strategy and are ignored by others. One common optimization is to break gradients all-reduce into multiple packs so that weight updates can overlap with gradient all-reduce. Examples: ```python options = tf.distribute.experimental.CommunicationOptions( bytes_per_pack=50 * 1024 * 1024, timeout_seconds=120.0, implementation=tf.distribute.experimental.CommunicationImplementation.NCCL ) grads = tf.distribute.get_replica_context().all_reduce( 'sum', grads, options=options) optimizer.apply_gradients(zip(grads, vars), experimental_aggregate_gradients=False) ``` ct|i|S)NOptions)clsargskwargss r__new__z_OptionsExported.__new__Os D #F ##rrNcy)aCreates a CollectiveHints. Args: bytes_per_pack: a non-negative integer. Breaks collective operations into packs of certain size. If it's zero, the value is determined automatically. This hint is respected by all multi-replica strategies except `TPUStrategy`. timeout_seconds: a float or None, timeout in seconds. If not None, the collective raises `tf.errors.DeadlineExceededError` if it takes longer than this timeout. Zero disables timeout. This can be useful when debugging hanging issues. This should only be used for debugging since it creates a new thread for each collective, i.e. an overhead of `timeout_seconds * num_collectives_per_second` more threads. This only works for `tf.distribute.experimental.MultiWorkerMirroredStrategy`. implementation: a `tf.distribute.experimental.CommunicationImplementation`. This is a hint on the preferred communication implementation. Possible values include `AUTO`, `RING`, and `NCCL`. NCCL is generally more performant for GPU, but doesn't work for CPU. This only works for `tf.distribute.experimental.MultiWorkerMirroredStrategy`. Raises: ValueError: When arguments have invalid value. Nrselfbytes_per_packtimeout_secondsimplementations r__init__z_OptionsExported.__init__Vs8 r)r r r r rrrr rrrrr2s"6$ #9>> rrcteZdZdZddej fdZej je_dZdZ y)rz#Implementation of OptionsInterface.rNc|dkrtd|dt|trt|j }t|ts td||_||_||_y)Nrz0Argument `bytes_per_pack` must be >=0, Received .zgArgument `implementation` must be instance of `tf.distribute.experimental.CommunicationImplementation`.) ValueError isinstancestrrupperrrrrs rr zOptions.__init__xs  <^3G3G3IJn n&A B  F GG)D*D(Drctj|}||S|jdk7r|j|_|j|j|_|jt j k7r|j|_|S)a1Merges with another options and returns a new one. Values specified in the `options` takes precedence if they're not the default. Args: options: a `tf.distribute.experimental.CollectiveCommunication`. Returns: A new `tf.distribute.experimental.CollectiveCommunication`. r)copydeepcopyrrrrr)roptionsmergeds rmergez Options.mergesz]]4 F m"%44f*&66f!%d&9&9%:;#3345"112! 56r) r r r r rrr rr-r0rrrrrus;+ #9>>)"&..66(.6rrz'distribute.experimental.CollectiveHintscLeZdZdZej ddddZddZy)HintsanHints for collective operations like AllReduce. This can be passed to methods like `tf.distribute.get_replica_context().all_reduce()` to optimize collective operation performance. Note that these are only hints, which may or may not change the actual behavior. Some options only apply to certain strategy and are ignored by others. One common optimization is to break gradients all-reduce into multiple packs so that weight updates can overlap with gradient all-reduce. Examples: - bytes_per_pack ```python hints = tf.distribute.experimental.CollectiveHints( bytes_per_pack=50 * 1024 * 1024) grads = tf.distribute.get_replica_context().all_reduce( 'sum', grads, experimental_hints=hints) optimizer.apply_gradients(zip(grads, vars), experimental_aggregate_gradients=False) ``` - timeout_seconds ```python strategy = tf.distribute.MirroredStrategy() hints = tf.distribute.experimental.CollectiveHints( timeout_seconds=120.0) try: strategy.reduce("sum", v, axis=None, experimental_hints=hints) except tf.errors.DeadlineExceededError: do_something() ``` Nz8use distribute.experimental.CommunicationOptions insteadct||S)N)rrr)rrrs rrz Hints.__new__s % HHrcy)aqCreates a CollectiveHints. Args: bytes_per_pack: a non-negative integer. Breaks collective operations into packs of certain size. If it's zero, the value is determined automatically. This only applies to all-reduce with `MultiWorkerMirroredStrategy` currently. timeout_seconds: a float or None, timeout in seconds. If not None, the collective raises `tf.errors.DeadlineExceededError` if it takes longer than this timeout. This can be useful when debugging hanging issues. This should only be used for debugging since it creates a new thread for each collective, i.e. an overhead of `timeout_seconds * num_collectives_per_second` more threads. This only works for `tf.distribute.experimental.MultiWorkerMirroredStrategy`. Raises: ValueError: When arguments have invalid value. Nr)rrrs rr zHints.__init__s& r)rN)r r r r r deprecatedrr rrrr2r2s8$L; FHHHH rr2)r r)enumtensorflow.python.utilr tensorflow.python.util.tf_exportrEnumrCollectiveCommunicationobjectrrr2rrrr<s ! .6 @ <>$))>&6 9:? v? ;? D06f06f 45@ F@ 6@ r