AVhLdZddlmZddlmZddlmZ ddZ ddZy) z9Utils to create distributed datasets based on TF version.)tf2) input_libNc tjrtj||||||||St j ||||||S)adReturns a distributed dataset from the given tf.data.Dataset instance. This is a common function that is used by all strategies to return a distributed dataset. The distributed dataset instance returned is different depending on if we are in a TF 1 or TF 2 context. The distributed dataset instances returned differ from each other in the APIs supported by each of them. Args: dataset: a tf.data.Dataset instance. input_workers: an InputWorkers object which specifies devices on which iterators should be created. strategy: a `tf.distribute.Strategy` object, used to run all-reduce to handle last partial batch. num_replicas_in_sync: Optional integer. If this is not None, the value is used to decide how to rebatch datasets into smaller batches so that the total batch size for each step (across all workers and replicas) adds up to `dataset`'s batch size. input_context: `InputContext` for sharding. Only pass this in for between graph multi-worker cases where there is only one `input_worker`. In these cases, we will shard based on the `input_pipeline_id` and `num_input_pipelines` in the `InputContext`. options: Default is None. `tf.distribute.InputOptions` used to control options on how this dataset is distributed. build: whether to build underlying datasets when a DistributedDataset is created. This is only useful for `ParameterServerStrategy` now. replica_order: the order of the replicas, which will be used to reorder the iterators to match the device order. Returns: A distributed dataset instance. )num_replicas_in_sync input_contextbuildoptions replica_order)rrr )renabledrDistributedDataset input_lib_v1DistributedDatasetV1)dataset input_workersstrategyrrr rr s W/home/dcms/DCMS/lib/python3.12/site-packages/tensorflow/python/distribute/input_util.pyget_distributed_datasetrs`T [[]  ' '1##    , ,1#  c |>|jtjjk7r|jr t d|J|jtjjk(r#|j r|jr t dtjrtj|||||||Stj|||||S)aReturns a distributed dataset from the given input function. This is a common function that is used by all strategies to return a distributed dataset. The distributed dataset instance returned is different depending on if we are in a TF 1 or TF 2 context. The distributed dataset instances returned differ from each other in the APIs supported by each of them. Args: dataset_fn: a function that returns a tf.data.Dataset instance. input_workers: an InputWorkers object which specifies devices on which iterators should be created. input_contexts: A list of `InputContext` instances to be passed to call(s) to `dataset_fn`. Length and order should match worker order in `worker_device_pairs`. strategy: a `tf.distribute.Strategy` object, used to run all-reduce to handle last partial batch. options: Default is None. `tf.distribute.InputOptions` used to control options on how this dataset is distributed. build: whether to build underlying datasets when a `DistributedDatasetFromFunction` is created. This is only useful for `ParameterServerStrategy` now. replica_order: the order of the replicas, which will be used to reorder the iterators to match the device order. Returns: A distributed dataset instance. Raises: ValueError: if `options.experimental_replication_mode` and `options.experimental_place_dataset_on_device` are not consistent zWhen `experimental_place_dataset_on_device` is set for dataset placement, you must also specify `PER_REPLICA` for the replication modez`experimental_place_dataset_on_device` can not be set to True when experimental_fetch_to_device is True and replication mode is set to `PER_REPLICA`)input_contexts dataset_fnr rr ) experimental_replication_moderInputReplicationMode PER_REPLICA$experimental_place_dataset_on_device ValueErrorexperimental_fetch_to_devicerr DistributedDatasetsFromFunctionr !DistributedDatasetsFromFunctionV1)rrrrr rr s r&get_distributed_datasets_from_functionr UsRgCC$$001 22    gCC  ' ' 3 34 ** 22  3 44  [[]  4 4%#   9 9xW FFr)NNNTN)NTN) __doc__tensorflow.pythonrtensorflow.python.distributertensorflow.python.distribute.v1r rr rrr&s=@!2E  <H  FFr