BVh *dZddlmZddlmZddlmZddlmZddlm Z ddl m Z e jdd e d  dd Z e jdd e d g ddZe jdd e d g ddZeje_e jdde dg ddZe jdde dg ddZeje_ej$reaeaneaeadZej,ey)z*Non-deterministic dataset transformations.)tf2) v2_compat) dataset_ops)readers) deprecation) tf_exportNzUse `tf.data.Dataset.interleave(map_func, cycle_length, block_length, num_parallel_calls=tf.data.AUTOTUNE)` instead. If sloppy execution is desired, use `tf.data.Options.deterministic`.z%data.experimental.parallel_interleavec&fd}|S)a6A parallel version of the `Dataset.interleave()` transformation. `parallel_interleave()` maps `map_func` across its input to produce nested datasets, and outputs their elements interleaved. Unlike `tf.data.Dataset.interleave`, it gets elements from `cycle_length` nested datasets in parallel, which increases the throughput, especially in the presence of stragglers. Furthermore, the `sloppy` argument can be used to improve performance, by relaxing the requirement that the outputs are produced in a deterministic order, and allowing the implementation to skip over nested datasets whose elements are not readily available when requested. Example usage: ```python # Preprocess 4 files concurrently. filenames = tf.data.Dataset.list_files("/path/to/data/train*.tfrecords") dataset = filenames.apply( tf.data.experimental.parallel_interleave( lambda filename: tf.data.TFRecordDataset(filename), cycle_length=4)) ``` WARNING: If `sloppy` is `True`, the order of produced elements is not deterministic. Args: map_func: A function mapping a nested structure of tensors to a `Dataset`. cycle_length: The number of input `Dataset`s to interleave from in parallel. block_length: The number of consecutive elements to pull from an input `Dataset` before advancing to the next input `Dataset`. sloppy: A boolean controlling whether determinism should be traded for performance by allowing elements to be produced out of order. If `sloppy` is `None`, the `tf.data.Options.deterministic` dataset option (`True` by default) is used to decide whether to enforce a deterministic order. buffer_output_elements: The number of elements each iterator being interleaved should buffer (similar to the `.prefetch()` transformation for each interleaved iterator). prefetch_input_elements: The number of input elements to transform to iterators before they are needed for interleaving. Returns: A `Dataset` transformation function, which can be passed to `tf.data.Dataset.apply`. c :tj|SN)rParallelInterleaveDataset)dataset block_lengthbuffer_output_elements cycle_lengthmap_funcprefetch_input_elementssloppys f/home/dcms/DCMS/lib/python3.12/site-packages/tensorflow/python/data/experimental/ops/interleave_ops.py _apply_fnz¶llel_interleave.._apply_fnQs)  , ,Wh -96-C-D FF)rrrrrrrs`````` rparallel_interleaversrFF rz0Use `tf.data.Dataset.sample_from_datasets(...)`.z&data.experimental.sample_from_datasets)v1cHtjj||||S)aKSamples elements at random from the datasets in `datasets`. Creates a dataset by interleaving elements of `datasets` with `weight[i]` probability of picking an element from dataset `i`. Sampling is done without replacement. For example, suppose we have 2 datasets: ```python dataset1 = tf.data.Dataset.range(0, 3) dataset2 = tf.data.Dataset.range(100, 103) ``` Suppose also that we sample from these 2 datasets with the following weights: ```python sample_dataset = tf.data.Dataset.sample_from_datasets( [dataset1, dataset2], weights=[0.5, 0.5]) ``` One possible outcome of elements in sample_dataset is: ``` print(list(sample_dataset.as_numpy_iterator())) # [100, 0, 1, 101, 2, 102] ``` Args: datasets: A non-empty list of `tf.data.Dataset` objects with compatible structure. weights: (Optional.) A list or Tensor of `len(datasets)` floating-point values where `weights[i]` represents the probability to sample from `datasets[i]`, or a `tf.data.Dataset` object where each element is such a list. Defaults to a uniform distribution across `datasets`. seed: (Optional.) A `tf.int64` scalar `tf.Tensor`, representing the random seed that will be used to create the distribution. See `tf.random.set_seed` for behavior. stop_on_empty_dataset: If `True`, sampling stops if it encounters an empty dataset. If `False`, it skips empty datasets. It is recommended to set it to `True`. Otherwise, the distribution of samples starts off as the user intends, but may change as input datasets become empty. This can be difficult to detect since the dataset starts off looking correct. Default to `False` for backward compatibility. Returns: A dataset that interleaves elements from `datasets` at random, according to `weights` if provided, otherwise with uniform probability. Raises: TypeError: If the `datasets` or `weights` arguments have the wrong type. ValueError: - If `datasets` is empty, or - If `weights` is specified and does not match the length of `datasets`. datasetsweightsseedstop_on_empty_dataset)rDatasetsample_from_datasetsrs rsample_from_datasets_v2r"Zs/v    1 1 1 2 33rcDtjt||||Sr )rDatasetV1Adapterr"rs rsample_from_datasets_v1r%s(  % %h7LM OOra$Use `tf.data.Dataset.choose_from_datasets(...)` instead. Note that, unlike the experimental endpoint, the non-experimental endpoint sets `stop_on_empty_dataset=True` by default. You should set this argument explicitly in case you would like to match the behavior of the experimental endpoint.z&data.experimental.choose_from_datasetscFtjj|||S)aCreates a dataset that deterministically chooses elements from `datasets`. For example, given the following datasets: ```python datasets = [tf.data.Dataset.from_tensors("foo").repeat(), tf.data.Dataset.from_tensors("bar").repeat(), tf.data.Dataset.from_tensors("baz").repeat()] # Define a dataset containing `[0, 1, 2, 0, 1, 2, 0, 1, 2]`. choice_dataset = tf.data.Dataset.range(3).repeat(3) result = tf.data.experimental.choose_from_datasets(datasets, choice_dataset) ``` The elements of `result` will be: ``` "foo", "bar", "baz", "foo", "bar", "baz", "foo", "bar", "baz" ``` Args: datasets: A non-empty list of `tf.data.Dataset` objects with compatible structure. choice_dataset: A `tf.data.Dataset` of scalar `tf.int64` tensors between `0` and `len(datasets) - 1`. stop_on_empty_dataset: If `True`, selection stops if it encounters an empty dataset. If `False`, it skips empty datasets. It is recommended to set it to `True`. Otherwise, the selected elements start off as the user intends, but may change as input datasets become empty. This can be difficult to detect since the dataset starts off looking correct. Default to `False` for backward compatibility. Returns: A dataset that interleaves elements from `datasets` according to the values of `choice_dataset`. Raises: TypeError: If `datasets` or `choice_dataset` has the wrong type. ValueError: If `datasets` is empty. rchoice_datasetr)rr choose_from_datasetsr's rchoose_from_datasets_v2r*s,f    1 1#1 2 33rcBtjt|||Sr )rr$r*r's rchoose_from_datasets_v1r,s&  % %h8MN PPrc^tjr tatayt atayr )renabledr*r)r"r!r,r%rrr _tf2_callbackr/s#[[]2222r)FNN)NNF)F)__doc__tensorflow.pythonrtensorflow.python.compatrtensorflow.python.data.opsrrtensorflow.python.utilr tensorflow.python.util.tf_exportr deprecatedrr"r%r*r,r.r)r!r/register_data_v2_callbackrrrr9s1!.2..6AB  23&'$/304 94 B 9xJL 3;$(!%27<3<L<3~JL 789$(!%27O:LO#:"A"A   3;38/3< /3d   78938P: P#:"A"A3;;=00003$ ##M2r