BVh?rdZddlmZddlmZddlmZddlmZddlm Z ddl m Z ddl m Z dd lmZdd lmZdd lmZGd d e j*ZGdde j.ZGdde j2ZGddeZGddeZGdde j:ZGddeZ ddZ!y)z0Various classes representing distributed inputs.) cardinality) dataset_ops)multi_device_iterator_ops) optional_ops) input_lib)context)ops)control_flow_ops)data) deprecatedcPeZdZdZ d fd ZdZdZdZd dZdZ dZ xZ S) DistributedDatasetV1zBDistributed dataset that supports prefetching to multiple devices.cF||_tt|||||||y)N)num_replicas_in_sync input_contextoptions)_input_workerssuperr__init__)selfdataset input_workersstrategyrrr __class__s Y/home/dcms/DCMS/lib/python3.12/site-packages/tensorflow/python/distribute/v1/input_lib.pyrzDistributedDatasetV1.__init__ s4(D .1# /c"|jS)zGet a one time use iterator for DistributedDatasetV1. Note: This API is deprecated. Please use `for ... in dataset:` to iterate over the dataset or `iter` to create an iterator. Returns: A DistributedIteratorV1 instance. )_make_one_shot_iteratorrs rmake_one_shot_iteratorz+DistributedDatasetV1.make_one_shot_iterator0s  ' ' ))rc`tjs td|jS)z)Get an iterator for DistributedDatasetV1.VCannot create a one shot iterator. Please use `make_initializable_iterator()` instead.rexecuting_eagerly ValueError _get_iteratorrs rrz,DistributedDatasetV1._make_one_shot_iterator;4  $ $ & B CC    rc"|jS)aGet an initializable iterator for DistributedDatasetV1. Note: This API is deprecated. Please use `tf.compat.v1.data.make_initializable_iterator(dataset)` to create an initializable iterator. Returns: A DistributedIteratorV1 instance. )_make_initializable_iteratorrs rmake_initializable_iteratorz0DistributedDatasetV1.make_initializable_iteratorEs  , , ..rc`tjr td|jS)z7Get an initializable iterator for DistributedDatasetV1.PCannot create initializable iterator in Eager mode. Please use `iter()` instead.r#r shared_names rr)z1DistributedDatasetV1._make_initializable_iteratorQs1  " 6 77    rclt|j|j|j}t j |jd}t |j||j||j}|j|_ tjrtj|SNr)_create_iterators_per_worker_cloned_datasetsr_optionsr _cardinalityDistributedIteratorV1 _strategy_enable_get_next_as_optional element_spec _element_specrr$ async_wait)rworker_iteratorsriterators rr&z"DistributedDatasetV1._get_iteratorZs3D4I4I484G4G48MMC(()>)>q)ABK$T%8%8:J%)^^[%)%F%FHH"..H   "  Orctjstjjr|j St dNzV__iter__() is only supported inside of tf.function or when eager execution is enabled.r #executing_eagerly_outside_functionsget_default_graphbuilding_functionr& RuntimeErrorrs r__iter__zDistributedDatasetV1.__iter__mA //1 11    !! = >>r)NNNN) __name__ __module__ __qualname____doc__rr rr*r)r&rD __classcell__rs@rrrs4J %)!  *  / &>rrc*eZdZdZddZdZdZdZy)!DistributedDatasetsFromFunctionV1z%Inputs created from dataset function.Ncb~tjr td|jS)zDGet an initializable iterator for DistributedDatasetsFromFunctionV1.r,r#r-s rr)z>DistributedDatasetsFromFunctionV1._make_initializable_iterator|s4  " 6 77    rc`tjs td|jS)zEGet an iterator for iterating over DistributedDatasetsFromFunctionV1.r"r#rs rrz9DistributedDatasetsFromFunctionV1._make_one_shot_iteratorr'rclt|j|j|j}t j |jd}t |j||j||j}|j|_ tjrtj|Sr0) r1 _datasetsrr3rr4r5r6r7r9rr$r:)r iteratorsrr<s rr&z/DistributedDatasetsFromFunctionV1._get_iterators,T^^-1-@-@$--QI(():;K$T%8%8)%)^^[%)%F%FHH"//H   "  Orctjstjjr|j St dr>r?rs rrDz*DistributedDatasetsFromFunctionV1.__iter__rErrF)rGrHrIrJr)rr&rDrrrNrNxs-  $>rrNceZdZdZedZedddZedZedZ edZ ed Z d Z ed Z y) r5z)Input Iterator for a distributed dataset.cg}|jD]!}|j|j#tj|SrF) _iteratorsextend initializer group)rinit_opsits r _initializerz"DistributedIteratorV1._initializers=Hoo'oobmmo&'  ! !( ++rNz2Use the iterator's `initializer` property instead.c|jS)zkInitialize underlying iterators. Returns: A list of any initializer ops that should be run. )r^rs rrZz DistributedIteratorV1.initializes   rc"|jS)z3Returns a list of ops that initialize the iterator.)rZrs r initializerz!DistributedIteratorV1.initializers ?? rc4|jdjSr0)rXoutput_classesrs rrcz$DistributedIteratorV1.output_classess ??1  , ,,rc4|jdjSr0)rX output_shapesrs rrez#DistributedIteratorV1.output_shapess ??1  + ++rc4|jdjSr0)rX output_typesrs rrgz"DistributedIteratorV1.output_typess ??1  * **rc~t|jjD]\}}||k(s |j|cSyrF) enumeraterworker_devicesrX)rworkeriws r get_iteratorz"DistributedIteratorV1.get_iteratorsA$--<<="1 1q!!" rc|jS)z6The type specification of an element of this iterator.)r9rs rr8z"DistributedIteratorV1.element_specs   r)rGrHrIrJpropertyr^r rZrarcrergrnr8rUrrr5r5s1  , , dHIJ    - - , , + +   rr5c(eZdZdZ dfd ZxZS)DatasetIteratorz$Iterator created from input dataset.ct|||||}t|j|}tt|||||j |j|j|_ y)aiMake an iterator for the dataset on given devices. If `num_replicas_in_sync` is not None, we split each batch of the dataset into `num_replicas_in_sync` smaller batches, to be distributed among that worker's replicas, so that the batch size for a global step (across all workers and replicas) is as expected. Args: dataset: `tf.data.Dataset` that will be used as the input source. input_workers: an `InputWorkers` object. strategy: a `tf.distribute.Strategy` object, used to run all-reduce to handle last partial batch. num_replicas_in_sync: Optional integer. If this is not None, the value is used to decide how to rebatch datasets into smaller batches so that the total batch size for each step (across all workers and replicas) adds up to `dataset`'s batch size. input_context: `InputContext` for sharding. Only pass this in for between graph multi-worker cases where there is only one `input_worker`. In these cases, we will shard based on the `input_pipeline_id` and `num_input_pipelines` in the `InputContext`. )rrN) rr1r2rrrrrr7r8r9) rrrrrr dist_datasetr;rs rrzDatasetIterator.__init__ss6(1# %L4%%}6 / &(8(%11%BBD&22Dr)NNrGrHrIrJrrKrLs@rrrrrs, %)! (3(3rrrc"eZdZdZfdZxZS)InputFunctionIteratorz%Iterator created from input function.ct|tjsJ|jt |k7r#t d|jt |fzg}t |D]\}}|j|}tj|5||} |j|} t| tjrt| || } n$t| rt| || } n t d|j!| dddt"t$|O|||t(j*dd|_y#1swYxYw)a|Make an iterator for input provided via an input function. Currently implements PER_WORKER mode, in which the `input_fn` is called once on each worker. TODO(priyag): Add other replication modes. Args: input_fn: Input function that returns a `tf.data.Dataset` object. input_workers: an `InputWorkers` object. input_contexts: A list of `InputContext` instances to be passed to call(s) to `input_fn`. Length and order should match worker order in `worker_device_pairs`. strategy: a `tf.distribute.Strategy` object, used to run all-reduce to handle last partial batch. zINumber of input workers (%d) is not same as number of input_contexts (%d)z5input_fn must return a tf.data.Dataset or a callable.NF)renable_get_next_as_optional) isinstancer InputWorkers num_workerslenr%rirjr devicecompute_devices_for_worker data_types DatasetV2_SingleWorkerDatasetIteratorcallable_SingleWorkerCallableIteratorappendrrwrcardinality_libUNKNOWNr7) rinput_fnrinput_contextsrrSrlctxrkresultdevicesr<rs rrzInputFunctionIterator.__init__sS" mY%;%; << <  C$77 -%113~3FGH IIIN+ #3++A.f ::f  ##::1= fj22 31&&'J( f 2667K(EG G" # # # /#++$) 0+ ).D%% # #s A7D??E rurLs@rrwrws-,.,.rrwcLeZdZdZdZdZedZedZedZ y)rz4Iterator for a single DistributedDatasetV1 instance.ctj|j5|j[t j |j |j|jj|jj|_ n/t j |j |j|_ dddy#1swYyxYw)z)Make appropriate iterator on the dataset.N)max_buffer_sizeprefetch_buffer_size) r r~_workerr3rMultiDeviceIterator_dataset_devices$experimental_per_replica_buffer_size _iteratorrs r_make_iteratorz+_SingleWorkerDatasetIterator._make_iteratorHs DLL !   "2FF MM MM MMNN!% 1 1 33FF MM MM     s BCC ctjr|jjgS|jjgS)a Initialize underlying iterator. In eager execution, this simply recreates the underlying iterator. In graph execution, it returns the initializer ops for the underlying iterator. Returns: A list of any initializer ops that should be run. )r r@r _eager_resetrars rrZz'_SingleWorkerDatasetIterator.initializeXs8 ..0 nn!!# inn(( ))rc@tj|jSrF)rget_legacy_output_classesrrs rrcz+_SingleWorkerDatasetIterator.output_classeshs  0 0 @@rc@tj|jSrF)rget_legacy_output_shapesrrs rrez*_SingleWorkerDatasetIterator.output_shapesls  / / ??rc@tj|jSrF)rget_legacy_output_typesrrs rrgz)_SingleWorkerDatasetIterator.output_typesps  . .t~~ >>rN) rGrHrIrJrrZrprcrergrUrrrrEsO< *  A A @ @ ? ?rrc2eZdZdZdZddZddZdZdZy) rz0Iterator for a single tensor-returning callable.c.||_||_||_yrF)_fnrr)rfnrkrs rrz&_SingleWorkerCallableIterator.__init__xsDHDLDMrNc~~tj|j5|jcdddS#1swYyxYw)z8Get next element for the given device from the callable.N)r r~rr)rr~names rget_nextz&_SingleWorkerCallableIterator.get_next}s5 DLL ! XXZs <Ac~tj|j5|jDcgc]}|j }}|cdddScc}w#1swYyxYw)z#Get next element from the callable.N)r r~rrr)rr_ data_lists rget_next_as_listz._SingleWorkerCallableIterator.get_next_as_listsR DLL !'+}}5!488:5i5 5sAAAAA"ctj|j5|jDcgc]/}tj j |j1}}|cdddScc}w#1swYyxYwrF)r r~rrrOptional from_valuer)rrrs rget_next_as_optional_listz7_SingleWorkerCallableIterator.get_next_as_optional_listsk DLL !@D ;<,   * *488: 6i sA54A0#A50A55A>cgSrFrUrs rrZz(_SingleWorkerCallableIterator.initializes IrrF) rGrHrIrJrrrrrZrUrrrrus8  rrNc|t|tjsJt|t|jk(sJg}t |jD]V\}}t j|5|j|}t|||||}|j|dddX|S#1swYdxYw)z5Create a multidevice iterator on each of the workers.N) rzrr{r}rjrir r~rrr)worker_datasetsrrrSrlrkrjr<s rr1r1s M9#9#9 :: : _ ]%A%A!B BB B)]99:!ia F !$??Bn- !     h x !!! !!s 24B22B; rF)"rJ'tensorflow.python.data.experimental.opsrrtensorflow.python.data.opsrrrtensorflow.python.distributertensorflow.python.eagerrtensorflow.python.frameworkr tensorflow.python.opsr tensorflow.python.typesr r"tensorflow.python.util.deprecationr DistributedDatasetrDistributedDatasetsFromFunctionrNDistributedIteratorBaser5rrrw _SingleWorkerDatasetIteratorBaserobjectrr1rUrrrs7R2@32++269V>977V>v0> --0>j4I==4n+3++3^/.1/.d-?9#M#M-?`FBr