BVh-JdZddlZddlZddlmZddlmZddlm Z ddl m Z ddl m Z ddl m Z dd lmZdd lmZdd lmZdd lmZdd lmZddlmZGddej.Zej2ddedddZdZdZddej:fdZy)zExperimental shuffle ops.N) random_access) dataset_ops) random_seed) constant_op)dtypes)ops) array_ops)gen_dataset_ops)math_ops)stateless_random_ops) deprecation) tf_exportc$eZdZdZdfd ZxZS)_ShuffleAndRepeatDatasetz.A `Dataset` that fuses `shuffle` and `repeat`.cJ||_tj|tjd|_|,t jdtjd|_n+tj|tjd|_tj|\|_ |_ tj|jjf|j |j|j|jd|j }t"t$|O||y)N buffer_size)dtypenamecount)rrseedseed2)_input_datasetrconvert_to_tensorrint64 _buffer_sizerconstant_countrget_seed_seed_seed2r shuffle_and_repeat_dataset_variant_tensor_flat_structuresuperr__init__)self input_datasetrrrvariant_tensor __class__s c/home/dcms/DCMS/lib/python3.12/site-packages/tensorflow/python/data/experimental/ops/shuffle_ops.pyr&z!_ShuffleAndRepeatDataset.__init__%s'D--6<>> d = tf.data.Dataset.from_tensor_slices([1, 2, 3]) >>> d = d.apply(tf.data.experimental.shuffle_and_repeat(2, count=2)) >>> [elem.numpy() for elem in d] # doctest: +SKIP [2, 3, 1, 1, 3, 2] ```python dataset.apply( tf.data.experimental.shuffle_and_repeat(buffer_size, count, seed)) ``` produces the same output as ```python dataset.shuffle( buffer_size, seed=seed, reshuffle_each_iteration=True).repeat(count) ``` In each repetition, this dataset fills a buffer with `buffer_size` elements, then randomly samples elements from this buffer, replacing the selected elements with new elements. For perfect shuffling, set the buffer size equal to the full size of the dataset. For instance, if your dataset contains 10,000 elements but `buffer_size` is set to 1,000, then `shuffle` will initially select a random element from only the first 1,000 elements in the buffer. Once an element is selected, its space in the buffer is replaced by the next (i.e. 1,001-st) element, maintaining the 1,000 element buffer. Args: buffer_size: A `tf.int64` scalar `tf.Tensor`, representing the maximum number elements that will be buffered when prefetching. count: (Optional.) A `tf.int64` scalar `tf.Tensor`, representing the number of times the dataset should be repeated. The default behavior (if `count` is `None` or `-1`) is for the dataset be repeated indefinitely. seed: (Optional.) A `tf.int64` scalar `tf.Tensor`, representing the random seed that will be used to create the distribution. See `tf.random.set_seed` for behavior. Returns: A `Dataset` transformation function, which can be passed to `tf.data.Dataset.apply`. c t|S)N)r)datasetrrrs r+ _apply_fnz%shuffle_and_repeat.._apply_fnms #G[% FFr,)rrrr6s``` r+shuffle_and_repeatr8:sfG r,cg}d}tjg}d}tjg}d}d}|D]}|j|dd} d|vrE|ddkrtdj |d|ddk(r|d} nt |d|d} |d| z } d|vr<|ddkrtd j |d|ddk7rt |d| } |d| z | z } | |dk7rd }|| z }tj||| z}|| | zz }tj||}|| z }||d } |r || d <|| d <| S)aPComputes aggregate information about files to read. The method collects information about the files to read, the total number of elements, and arrays that can be used to account for elements to be skipped, which can be specified via the "skip" and "take" keys. To account for elements to skip, the range of each file can be divided into three regions: - S (elements to skip) - T (elements to read) - R (remainder of elements that will also be skipped) The `thresholds` and `offsets` arrays are initialized as follows: `thresholds = [0, T_1, T_1 + T_2, ...]` and `offsets = [S_1, S_1 + R_1 + S_2, S_1 + R_1 + S_2 + R_2 + S_3, ...]` This makes it possible to map an index from a contiguous range `(0...num_elements_to_read)` to an index in the range of all elements, skipping over elements as per the "skip" and "take" keys values. In particular, for a given input index `X`, we find the greatest `thresholds` value that is smaller or equal to `X`. Let `t(X)` denotes such index in the `thresholds` array. The output index is computed as `X + offsets[t(X)]`. Args: file_infos: See `file_infos` argument of `index_shuffle` for details. Returns: A dictionary containing the following keys: - `files`, the vector of pathnames of files to read - `num_elements`, an integer identifying the total number of elements - `offsets`, the vector of offsets to use for index adjustment (in case any elements should be skipped) - `thresholds`, the vector of thresholds to use for index adjustment (in case any elements should be skipped) rFpathskiprz-`skip` should be greater than `-1` but got {} num_elementstakez-`take` should be greater than `-1` but got {}T)filesr<offsets thresholds)nprappend ValueErrorformatmin) file_infosr>r<r? offset_sumr@ threshold_sumadjustment_needed file_infor;r= remainderresults r+_process_file_infosrMssH %, HHRL'*xx|*-i LL6"# D  6 R HOO f   6 b (9V$i&?@ ^ $t +D  6 R HOO f    6 b 9V$d+.)D047I y((DLiid!23G$""J:}5JTM56L 9&F9%F< -r,c tjtj|tj||ddz }|tj ||zS)z4Adjusts index to account for elements to be skipped.r)r shape boolean_maskr less_equalgather)indexr@r?t_indexs r+ _adjust_indexrVs_ OO    j% 02 345 69: :' !!'73 33r,Fct|fd}tjj||}|j dj dd}|j |S)a(Creates a (globally) shuffled dataset from the given set of files. Unlike `tf.data.Dataset.shuffle()`, which uses an in-memory buffer to shuffle elements of input dataset in a streaming fashion, `tf.data.experimental.index_shuffle()` performs a global shuffle of element indices and then reads the data in a shuffled order. The advantage of `index_shuffle()` is that it can perform global shuffle of datasets that do not fit into memory (as long as the array of their indices does) and that the shuffling logic it provides is compatible with symbolic checkpointing. The disadvantage of `index_shuffle()` is that reading data in a shuffled random order will in general not be as efficient as reading data sequentially. Args: file_infos: A list of dictionaries that describe each file of the input dataset. Each dictionary is expected to contain the "path" key, which identifies the path of the file and the "num_elements" key, which identifies the number of elements in the file. In addition, the "skip" and "take" keys can be used to identify the number of elements to skip and take respectively. By default, no elements are skipped and all elements are taken. reader_factory: A function that maps a sequence of filenames to an instance of `tf.data.Dataset` that reads data from the files. seed: (Optional.) A `tf.int64` scalar `tf.Tensor`, representing the random seed that will be used to shuffle the order of elements. Default to non-deterministic seed. reshuffle_each_iteration: (Optional.) A `tf.bool` scalar `tf.Tensor`, that determines whether to change the shuffle order each iteration. Defaults to `False`. num_parallel_calls: (Optional.) A `tf.int64` scalar `tf.Tensor`, that determines the maximum number of random access operations to perform in parallel. By default, the tf.data runtime uses autotuning to determine the value dynamically. Returns: A `tf.data.Dataset` object, representing a globally shuffled dataset of the input data. ctjjd}fd}tj|d}|j |S)Nr<ctj|ddz }dvrdvrt|dd}tj||S)Nr<rOr@r?)r index_shufflerVrat)r5rTshuffled_indexrLseedss r+ read_elementzEindex_shuffle..sequential_index_shuffle..read_elementse+99 ~.24n  I$7&~vl7K'-i'8:  g~ 66r,r>)num_parallel_calls)rDatasetrange functoolspartialmap)r]r5r^map_funcr_reader_factoryrLs` r+sequential_index_shufflez/index_shuffle..sequential_index_shufflesT!!''~(>?G 7  ~fWo/NOH ;;x4F; GGr,)rrerandomize_each_iterationT)drop_remainder)rMrr`randomr=batchflat_map)rFrfrreshuffle_each_iterationr_rgrng_dsrLs ` ` @r+rZrZshV z *&H&    % % !9 & ;& ;;q>  $  7& 1 22r,r-)r1rbnumpyrA'tensorflow.python.data.experimental.opsrtensorflow.python.data.opsrtensorflow.python.data.utilrtensorflow.python.frameworkrrrtensorflow.python.opsr r r r tensorflow.python.utilr tensorflow.python.util.tf_exportrUnaryUnchangedStructureDatasetr deprecatedr8rMrVAUTOTUNErZr7r,r+r{s A233.++1*6.6C{IIC0 ./ 1213 / 1hJZ4+0%0%9%9 D3r,