AVh4`dZddlZddlZddlZddlZddlmZddlmZddlm Z ddlm Z ddl m Z ddl mZdd l mZdd l mZdd lmZdd lmZdd lmZddlmZddlmZddlmZddlmZddlmZddlmZddlmZddlmZ ddlm!Z!ddlm"Z"ddlm#Z#ddlm$Z$ddlm%Z%ddlm&Z&ddlm'Z'ddl(m)Z)ddl(m*Z+ddl(m,Z,dd l(m-Z-dd!l.m/Z/dd"l0m1Z2ddl3m Z4dd#l5m6Z6dd$l7m8Z8ejrd%ejtd&d'd()d*d+d,Z;ejrd-ejtd&d'd()d.d+d,ZZ?d2Z@d3ZAGd4d5e8je4jZDGd6d7e'jZFGd8d9eFZGGd:d;eDejZIGd<d=e8je4jZLGd>d?eFZMGd@dAeLejZNGdBdCeFZOGdDdEeLejZPdFZQdGZRGdHdIe>ZSGdJdKe'jZTGdLdMeSejZU d^dNZVdOZWdPZXdQZYdRZZdSZ[GdTdUe>Z\dVZ]dWZ^dXZ_dYZ`dZZad[Zbd\Zcd]ZdejjeIedejjeNedy)_z0Various classes representing distributed inputs.N) py_builtins)batching) cardinality) distribute) dataset_ops) iterator_ops)multi_device_iterator_ops) optional_ops) device_util)distribute_lib)distribute_utils) input_ops) reduce_util)values)InputReplicationMode)context) monitoring)composite_tensor)device)dtypes)errors)ops) sparse_tensor) tensor_shape) tensor_util) type_spec) array_ops)cond)math_ops) while_loop) ragged_tensor) tf_logging)nest)collections_abczZ/tensorflow/api/distribution_strategy/distributed_dataset_initialization_time_milliseconds)scale growth_factor bucket_countzDTrack the time (in milliseconds) to initialize distributed datasets.strategyworkerszh/tensorflow/api/distribution_strategy/distributed_dataset_from_function_initialization_time_millisecondszRTrack the time (in milliseconds) to initialize distributed datasets from function.c|j}t|jttfrMt |j j||j jd|jd}|S|j jr,tj|j jdtj|}|S)aReturns an iterator spec from dataset function. This function constructs type spec for iterator obtained from iter(dataset). Args: strategy: a `tf.distribute.Strategy` object, used to run all-reduce to handle last partial batch. dataset: A tf.data.Dataset instance. If using a function that returns a tf.data.Dataset instance, pass dataset_fn.structured_outputs. Returns: A type_spec for iterator for dataset instance. NT)optionsrenable_get_next_as_optionalz GPUs are allocated per worker. Please use DistributedDataset by calling strategy.experimental_distribute_dataset or strategy.distribute_datasets_from_function to make best use of GPU resources) element_spec isinstance _type_specDistributedDatasetSpec#DistributedDatasetsFromFunctionSpecDistributedIteratorSpecextended_input_workers_with_options_container_strategyr_num_gpus_per_workerloggingwarningr IteratorSpec)r+datasetoutput_element_speciterator_type_specs V/home/dcms/DCMS/lib/python3.12/site-packages/tensorflow/python/distribute/input_lib.pyget_iterator_spec_from_datasetrANs" ,,""'4671557--/''$( *" -- oo    3 3 45 &223FG cReZdZdZ d dZedZedZdZdZ dZ dZ y ) InputWorkerszAA 1-to-many mapping from input worker devices to compute devices.c||_td|jD|_||_|r"td|jD|_ytd|jD|_y)ajInitialize an `InputWorkers` object. Args: worker_device_pairs: A sequence of pairs: `(input device, a tuple of compute devices fed by that input device)`. canonicalize_devices: Whether to canonicalize devices for workers fully or partially. If False, it will partially canonicalize devices by removing job and task. c3&K|] \}}| ywN).0d_s r@ z(InputWorkers.__init__..s&OTQq&Osc3FK|]\}}td|Dyw)c3FK|]}tj|ywrGr canonicalizerIrJs r@rLz2InputWorkers.__init__...s#((+!NtuplerIrKfs r@rLz(InputWorkers.__init__..0 1a   1rRc3FK|]\}}td|Dyw)c3FK|]}tj|ywrGr !canonicalize_without_job_and_taskrQs r@rLz2InputWorkers.__init__...s#==a@rRNrSrUs r@rLz(InputWorkers.__init__..rWrRN)_worker_device_pairsrT_input_worker_devices_canonicalize_devices _fed_devices)selfworker_device_pairscanonicalize_devicess r@__init__zInputWorkers.__init__}sv!4D!&&OT5N5N&O!OD!5D 1// 11d   1// 11drBc,t|jSrG)lenr]r`s r@ num_workerszInputWorkers.num_workerss t)) **rBc|jSrG)r]rfs r@worker_deviceszInputWorkers.worker_devicess  % %%rBc |j|SrGr_)r` worker_indexs r@compute_devices_for_workerz'InputWorkers.compute_devices_for_workers   \ **rBcjdjfdttD}jj d|dS)Nz, c3NK|]}d||j|fzyw)z %d %s: %sNrk)rIidevicesr`s r@rLz(InputWorkers.__repr__..s8: !* D,=,=a,@AB:s"%z:{ })rijoinrangere __class____name__)r` debug_reprrqs` @r@__repr__zInputWorkers.__repr__sI!!G:%*3w<%8::J..11: >>rBc2|j|jfSrG)r\r^rfs r@ serializezInputWorkers.serializes  % %t'A'A BBrBct|SrG)rD)r` serializeds r@ deserializezInputWorkers.deserializes  ##rBNT) rv __module__ __qualname____doc__rcpropertyrgrirmrxrzr}rHrBr@rDrDxsLI %)14 + + & &+?C$rBrDc ~g}t|j|D]\}}tj|5|Dcgc]4}t j |j tj6}}|jt j|ddddt j|d}|jjrB|jtjj |d} t#j$| gSt#j$|gScc}w#1swY#xYw)a7Computes the number of replicas that have values. Args: strategy: the `tf.distribute.Strategy`. input_workers: the `InputWorkers`. optional_list: a list of lists `tf.experimental.Optional`. The values from each compute device grouped by the input device. Returns: A scalar Tensor. T)keepdimsNaxis)ziprirrrcast has_valuerint64append reduce_sumr6_in_multi_worker_modereducerReduceOpSUMrreshape) r+ input_workers optional_listworker_has_valuesworker optionalsvdevice_has_valuesclient_has_valuesglobal_has_valuess r@_calculate_replicas_with_valuesrs(};;]KAfi F A>G9:(-- v|| 4   /$ ?A AAA))*;dK ,,.   "3$(@   . 33   . 33AAsD29D-2(D2-D22D< c\tj|D]}t|tjt j fr:|jjdkDsG|jjdey|jD]}|jjryy)a,Test if an iterator output is statically shaped. For sparse and ragged tensors this only tests the batch dimension. Args: element_spec: a nest structure of `tf.TypeSpec`. The element spec of the dataset of the iterator. Returns: True if the shape is static, false otherwise. rFT) r#flattenr1rSparseTensorSpecr!RaggedTensorSpecshaperankas_list_flat_tensor_specsis_fully_defined)r0spec components r@_is_statically_shapedrsll<( d }--}/M/MNP 1 !3!3!5a!8!@..)//1  rBcHeZdZdZ d dZdZdZdZdZd dZ d d Z d Z y) DistributedIteratorBasez.Common implementation for all input iterators.Nct|tsJ|js td||_||_||_||_||_||_ y)Nz3Should have at least one worker for input iterator.) r1rDri ValueError _iterators_input_workers _strategy _cardinality_enable_get_next_as_optional_replica_order)r`r iteratorsr+rr/ replica_orders r@rcz DistributedIteratorBase.__init__sW m\ 22 2  ' ' L MMDO'DDN#D(CD%'DrBc"|jSrG)__next__rfs r@nextzDistributedIteratorBase.nexts ==?rBc^ |jS#tj$rtwxYwrG)get_nextrOutOfRangeError StopIterationrfs r@rz DistributedIteratorBase.__next__s. ]]_  ! ! s,c|SrGrHrfs r@__iter__z DistributedIteratorBase.__iter__ s KrBcjtjk(r-tjj j Sjdk(rMjjjs)tjjjSgtjjD]O\}}tj |5j#j$|j'dddQfd}fd}t)jj}t+j,|dkD||dS#1swYxYw)Nrctjd}jj|}t |j }t jj|SNT) produce_dummy) _get_value_or_dummyrr_reorder_replicas_create_per_replicarr Optional from_value) value_list per_replicarr`s r@_create_optional_with_dummyzQDistributedIteratorBase.get_next_as_optional.._create_optional_with_dummy%s`&   }DBj    (++J7 ' DNNCk  " " - -k ::rBcVtjjjSrG)r rempty _element_specrfsr@_create_empty_optionalzLDistributedIteratorBase.get_next_as_optional.._create_empty_optional/s  " " ( ();); <._value_or_dummyNsJ&   }DBj    (++J7 T^^ <._eofWs 5 5 77rBrTr)r enter_or_assert_strategyrget_replica_contextrrrrrrirrrrrrrr)r`namerprrrrrs` @r@rz DistributedIteratorBase.get_next;s0  0 0  4  + + - 934 4 :4  , ,  5 5d ;;Mt22AABM 6 ::f MT__Q/IIKLMMM ? ++] <=8 << 1$otD JJ?44MMs D#$-D/#D,/D8 cg}t|jjD]\}}|> D   #h-  "&!4!4 5AHQK 55 5s# A2rG) rvrrrrcrrrrrrrrHrBr@rrs56(( *X"JH9"6rBrcdeZdZdZgdZej ddfdZdZdZ dZ dZ d Z d Z d Zy) !DistributedDatasetAndIteratorSpeczVCommon Type specification for `DistributedDataset and DistributedDatasetsFromFunction.)rrrrr_optionsr^Nct|tr td||_||_||_||_||_||_|j rt|j dd|_ nd|_ ||_ y)NzBDistributedIteratorSpec does not have support for deserialization.r^T) r1rTNotImplementedErrorrrrrrrgetattrr^r)r`rr0r+r.rr/rs r@rcz*DistributedDatasetAndIteratorSpec.__init__s-' !7 88*d'ddn%d*Ed'dm %,T^^-Dd&L"&*")drBc|jj|jt|jt|j fSrG)rrzridrrrfs r@ _serializez,DistributedDatasetAndIteratorSpec._serializes?    ) ) +T-?-? t~~ 4== 1 33rBc2tdt|d)Nz-Deserialization is currently unsupported for .)rtyperfs r@ _deserializez.DistributedDatasetAndIteratorSpec._deserializes!  7T |1E GGrBc0t|t|urtd|d||jj|jjk7rtd|d||j|jurtd|d|y)zReturns the most specific TypeSpec compatible with `self` and `other`. Args: other: A `TypeSpec`. Raises: ValueError: If there is no TypeSpec that is compatible with both `self` and `other`. z$No TypeSpec is compatible with both z and z+_input_workers is not compatible with both z3tf.distribute strategy is not compatible with both N)rrrrzr)r`others r@sanity_check_typez3DistributedDatasetAndIteratorSpec.sanity_check_types Dze$ e% && $$&%*>*>*H*H*JJ #'0 11 ~~U__, #'0 11-rBcX |j|tj|j|jtj |j}tj |j}tdt||DS#tt f$rYywxYw)zXReturns True if `self` is subtype of `other`. Args: other: A `TypeSpec`. Fc3DK|]\}}|j|ywrG) is_subtype_of)rI self_element other_elements r@rLzBDistributedDatasetAndIteratorSpec.is_subtype_of..s)Q )\= ""=1Qs ) rr#assert_same_structurer TypeErrorrrallr)r`r self_elementsother_elementss r@rz/DistributedDatasetAndIteratorSpec.is_subtype_ofs  U#   !3!3U5H5HILL!3!34M\\%"5"56N Q-0-OQ QQ z " s;BB)(B)c |D]=}|j|tj|j|j? tj |j}|Dcgc]!}tj |j#}}dgt|z}t|D]1\}}|j|Dcgc]}|| c}||<||1ytj|j|} t||j| |j|j|j|j S#tt f$rYywxYwcc}wcc}w)zReturns the most specific supertype of `self` and `others`. Args: others: A Sequence of `TypeSpec`. Returns `None` if a supertype does not exist. Nrr/)rr#rrrrrrermost_specific_common_supertypepack_sequence_asrrrrrr) r`othersrrothers_elementscommon_elementsrprrcommon_element_specs r@rz@DistributedDatasetAndIteratorSpec.most_specific_common_supertypes^L% u% ""4#5#5u7J7JKL LL!3!34MFLMUt||E$7$78MOMfs=11O$]3<'FF3B C>!  CEoa   #  //0B0B0?A 4:   %%$($E$E  GG z " N DsAE(&E? E! EEctjd|j}t||j||j |j |j|jS)Nc"|jSrG)_with_tensor_ranks_onlyss r@zKDistributedDatasetAndIteratorSpec._with_tensor_ranks_only..s!++-rBr r# map_structurerrrrrrrr`r0s r@rz9DistributedDatasetAndIteratorSpec._with_tensor_ranks_onlysa%%- L 4:   %%$($E$E  GGrBctjd|j}t||j||j |j |j|jS)Nc"|jSrG)_without_tensor_namesrs r@rzIDistributedDatasetAndIteratorSpec._without_tensor_names..s!))+rBrrrs r@rz7DistributedDatasetAndIteratorSpec._without_tensor_namessa%%+ L 4:   %%$($E$E  GGrB)rvrrr __slots__rUNKNOWNrcrrrrrrrrHrBr@rrysK^)"))"&*<3 G1*Q& GD G GrBrcLeZdZdZedZedZdZdZe dZ y)r5z-Type specification for `DistributedIterator`.ctSrG)DistributedIteratorrfs r@ value_typez"DistributedIteratorSpec.value_types rBc 8g}|jj}t|D]s\}\}}tjt j t||j}|jt||||j|ju|SN)rp) rr\rr#r functoolspartial_replace_per_replica_specrr _SingleWorkerDatasetIteratorSpecrr^)r`specsrarp input_devicecompute_devicesr0s r@_component_specsz(DistributedIteratorSpec._component_specss E--BB.78K.LH* *L/''   5 ;T=O=OQl ll *<+7+/+E+E GHH LrBc|jSrG)rr`values r@_to_componentsz&DistributedIteratorSpec._to_components"s   rBc t|jd||j|j|j|j |j |j S)N) rr componentsr0r+rr/r.r)rrrrrrrrr`r.s r@_from_componentsz(DistributedIteratorSpec._from_components%sP ))''%%$($E$E ))  rBct|j|j|j|j|j |j S)Nr)r5rrrrrrr+s r@rz"DistributedIteratorSpec.from_value2sE #    &&$)$F$F  HHrBN rvrrrrrr(r,r0 staticmethodrrHrBr@r5r5 sK5       HHrBr5c jeZdZdZdddddej dddf fd ZedZedZ xZ S)rz)Input Iterator for a distributed dataset.NFc | tdd} ||_|A|| t| ||_||_||_||_||_||_| |_y| | t| tt|/|||||| y)Nz#`input_workers` should be provided.zSEither `input_workers` or both `components` and `element_spec` need to be provided.) rrrrrrrrrsuperrrc) r`rrr+r.r0rr/r.r error_messagerus r@rczDistributedIterator.__init__Bs # $$!MDM   4'''d)d"dodn%d*Ed')d  \%='' /     %  rBc|jrJ|jjjr&t j t |jdS|jS)NFexpand_compositesrrr6rr#r_rebatch_as_dynamicrrfs r@r0z DistributedIterator.element_specnsQ )) 557    t11ULL   rBc t|j|j|j|j|j |j |jSrG)r5rrrrrrrrfs r@r2zDistributedIterator._type_speczsL #      ))  rB) rvrrrrrrcrr0r2 __classcell__rus@r@rr>sW1!))"'*X        rBrc"eZdZdZdZdZdZy)_IterableInputz;Base class for iterable inputs for distribution strategies.c6t|tsJ||_yrG)r1rDr)r`rs r@rcz_IterableInput.__init__s m\ 22 2'DrBctdNz"must be implemented in descendantsrrfs r@rz_IterableInput.__iter__ B CCrBct|j}d}fd}tj||||gdd\}}|S)z9Execute a `reduce_fn` over all the elements of the input.c$~|jSrG)r) optional_datastates r@rz#_IterableInput.reduce..conds   $ $ &&rBcZ||j}j}||fS)z9Executes `reduce_fn` in a loop till the dataset is empty.) get_valuer)rJrKiterator reduce_fns r@ loop_bodyz(_IterableInput.reduce..loop_bodys1}6689e335m E !!rBr%T)parallel_iterationsreturn_same_structure)iterrr )r` initial_staterOrJrrP final_staterNs ` @r@rz_IterableInput.reducesVDzH113M'" ",!6!6 M=1" "$M; rBN)rvrrrrcrrrHrBr@rBrBsC(DrBrBcLeZdZdZedZedZdZdZe dZ y)r3z+Type specification for `DistributedDataset.ctSrG)DistributedDatasetrfs r@rz!DistributedDatasetSpec.value_types rBcg}|jj}t|D]b\}}tjt j t||j}|jtj|d|Sr  rr\rr#rr!r"r#rrr DatasetSpecr`r%rarprKr0s r@r(z'DistributedDatasetSpec._component_specsz E--BB-.:1''   5 ;T=O=OQl ll;**<89: LrBc|jSrG)_cloned_datasetsr*s r@r,z%DistributedDatasetSpec._to_componentss  ! !!rBc t|j|j||j|j|j |j S)N)rr+r.r0r/r.r)rXrrrrrrr/s r@r0z'DistributedDatasetSpec._from_componentssD ))''$($E$E )) rBct|j|j|j|j|j SN)r/r3rrrrrr2s r@rz!DistributedDatasetSpec.from_values< "    $)$F$F  HHrBNr3rHrBr@r3r3sK3    " HHrBr3ceZdZdZ d fd Zd dZdZedZdZ dZ dZ ed Z ed Z xZS) rXzBDistributed dataset that supports prefetching to multiple devices.c Htt| ||| td| | td| | td||_||_| |_||_||_| |_ |"||_ d|_ | r|jyy| s td| td||_ t|jd |_||_|J|t!|j |jd j"k7r td ||_d |_ y) aDistribute the dataset on all workers. If `num_replicas_in_sync` is not None, we split each batch of the dataset into `num_replicas_in_sync` smaller batches, to be distributed among that worker's replicas, so that the batch size for a global step (across all workers and replicas) is as expected. Args: input_workers: an `InputWorkers` object. strategy: a `tf.distribute.Strategy` object, used to run all-reduce to handle last partial batch. dataset: `tf.data.Dataset` that will be used as the input source. Either dataset or components field should be passed when constructing DistributedDataset. Use this when constructing DistributedDataset from a new `tf.data.Dataset`. Use components when constructing using DistributedDatasetSpec. num_replicas_in_sync: Optional integer. If this is not None, the value is used to decide how to rebatch datasets into smaller batches so that the total batch size for each step (across all workers and replicas) adds up to `dataset`'s batch size. input_context: `InputContext` for sharding. Only pass this in for between graph multi-worker cases where there is only one `input_worker`. In these cases, we will shard based on the `input_pipeline_id` and `num_input_pipelines` in the `InputContext`. components: datasets when DistributedDataset is constructed from DistributedDatasetSpec. Either field dataset or components should be passed. element_spec: element spec for DistributedDataset when constructing from DistributedDatasetSpec. This will be used to set the element_spec for DistributedDataset and verified against element_spec from components. enable_get_next_as_optional: this is required when components is passed instead of dataset. build: whether to build underlying datasets when this object is created. This is only useful for `ParameterServerStrategy` now. options: `tf.distribute.InputOptions` used to control options on how this dataset is distributed. replica_order: the order of the replicas, which will be used to reorder the iterators to match the device order. rNz1input_workers and strategy are required argumentsz3Only one of dataset or components should be presentz6At least one of dataset or components should be passedFz~When constructing DistributedDataset with components, build should not be False. This is an internal error. Please file a bug.zgWhen constructing DistributedDataset with components, enable_get_next_as_optional should also be passedrz2Mismatched element_spec from the passed componentsT)r7rXrcrrrr_input_context_num_replicas_in_syncr_original_dataset_builtbuildr_rr_create_distributed_tensor_specr0r) r`rr+r=num_replicas_in_sync input_contextr.r0r/rkr.rrus r@rczDistributedDataset.__init__sbj d,=,I 0 J KKz5 L MM:- O PP'DDNDM'D!5D'D&ddk      % , @A A)d&t'<'7)!V ZZ  7(0.  #':N$77c-">">?"$.    & &~ 6 7 77 7 7s AE**E3 czrtdjztj|j5t j |dddfd}|S#1swYxYw)ajReturns a callable that rebatches the input dataset. Args: dataset: A `tf.data.Dataset` representing the dataset to be distributed. num_workers: An integer representing the number of workers to distribute `dataset` among. num_replicas_in_sync: An integer representing the number of replicas in sync across all workers. ztf.distribute expects every worker to have the same number of replicas. However, encountered `num_replicas_in_sync` ({}) that cannot be divided by `num_workers` ({})Nc  fd}fd}tjj5tjt j d||cdddS#1swYyxYw#tj$rb}dt|vrJtjttdj|tjdnYd}~yd}~wwxYw)Ncttj}j|jSrG)rbatch_sizes_for_workerrebatchprefetch) batch_sizes batch_sizer=num_replicas_per_workerrgrls r@ apply_rebatchzNDistributedDataset._make_rebatch_fn..rebatch_fn..apply_rebatchs9"99+'> N+-667NO OrBcNtjjSrG)r_LegacyRebatchDatasetr)r=rmrsr@apply_legacy_rebatchzUDistributedDataset._make_rebatch_fn..rebatch_fn..apply_legacy_rebatchs(11+--5X6M-N OrB)true_fnfalse_fnzwithout encountering a batchzCall the `batch` method on the input Dataset in order to be able to split your input across {} replicas. Please see the tf.distribute.Strategy guide. {}r&)rrrrrr not_equalrInvalidArgumentErrorrxsixreraiserrsysexc_info) r=rlrrerrmrrgs `` r@rz7DistributedDataset._make_rebatch_fn..rebatch_fns P P  O  w66 7 -  R0#+- - - -  ( (  )SV 3 ++99?*A:/0 llnQ !  ! s5/A6,A* A6*A3/A63A66C+ AC&&C+)rrrrrrcompute_batch_size)r`r=rgrmrrrs `` @@r@rz#DistributedDataset._make_rebatch_fnsk)  44:F"K51 22 3kA  722 3:009j:> E::s A11A:c Dtjs)tjjs t d|j s tdt|jdd}t|j|j|j|}t|j||j|j|j |j|j"}|j$|_tjrtj&|S)NV__iter__() is only supported inside of tf.function or when eager execution is enabled.zcTo use this dataset, you need to pass this dataset to ClusterCoordinator.create_per_worker_dataset.r^Tr.rb)rr/r.r)rrtrget_default_graphbuilding_function RuntimeErrorrjrrr_create_iterators_per_workerr_rrrrrrrru)r`rbworker_iteratorsrNs r@rzDistributedDataset.__iter__s  % % '  ! ! # 5 5 ? @@ ;; G HH#4>>3J#')4   1 3 #  %%$($E$E ))H"//H   "  OrBc|jrJ|jjjr&t j t |jdS|jSz5The type specification of an element of this dataset.Fr:r<rfs r@r0zDistributedDataset.element_specQ )) 557    t11ULL   rBct|j|j|j|j|j Srbrcrfs r@r2zDistributedDataset._type_specs: !    $($E$E  GGrB) NNNNNNTNNrG)rvrrrrcrkrrrrsrrr0r2r?r@s@r@rXrXsJ "& ]~.G.   *7X4l"H     G GrBrXcLeZdZdZedZedZdZdZe dZ y)r4z8Type specification for `DistributedDatasetsFromFunction.ctSrG)DistributedDatasetsFromFunctionrfs r@rz.DistributedDatasetsFromFunctionSpec.value_types **rBcg}|jj}t|D]b\}}tjt j t||j}|jtj|d|Sr rZr\s r@r(z4DistributedDatasetsFromFunctionSpec._component_specsr]rBc|jSrG) _datasetsr*s r@r,z2DistributedDatasetsFromFunctionSpec._to_componentss ??rBcrt|j|j||j|jS)N)rr+r.r0r.)rrrrrr/s r@r0z4DistributedDatasetsFromFunctionSpec._from_components"s2 *))''  rBcpt|j|j|j|jS)N)rr0r+r.r4rrrrr2s r@rz.DistributedDatasetsFromFunctionSpec.from_value*s1 /**((   rBNr3rHrBr@r4r4 sI@ + +    rBr4cteZdZdZ d fd ZdZdZedZdZ edZ edZ xZ S) rz%Inputs created from dataset function.c tt| |||_||_||_| |_| | td| | td||jt|k7r#td|jt|fz||_ |jdj|_ ||_ d|_|r|jyy| td|s td ||_||_d|_ d |_t%|j"d|_t'|j|j"d|j$|_y) a<Makes an iterable from datasets created by the given function. Args: input_workers: an `InputWorkers` object. strategy: a `tf.distribute.Strategy` object, used to run all-reduce to handle last partial batch. input_contexts: A list of `InputContext` instances to be passed to call(s) to `dataset_fn`. Length and order should match worker order in `worker_device_pairs`. dataset_fn: A function that returns a `Dataset` given an `InputContext`. Either dataset_fn or components should be passed to construct DistributedDatasetsFromFunction. Use this when constructing DistributedDataset using a function. Use components when constructing using DistributedDatasetsFromFunctionSpec. options: `tf.distribute.InputOptions` used to control options on how this dataset is distributed. components: datasets when DistributedDatasetsFromFunction is constructed from DistributedDatasetsFromFunctionSpec. Only one of dataset or components should be passed. element_spec: element spec for DistributedDataset when constructing from DistributedDatasetSpec. This will be used to set the element_spec for DistributedDatasetsFromFunctionSpec and verified against element_spec from components. build: whether to build underlying datasets when this object is created. This is only useful for `ParameterServerStrategy` now. replica_order: the order of the replicas, which will be used to reorder the iterators to match the device order. rfNz2Only one of dataset_fn or components should be setz6At least one of dataset_fn or components should be setzINumber of input workers (%d) is not same as number of input_contexts (%d)rFz:element_spec should also be passed when passing componentszWhen constructing DistributedDatasetFromFunction with components, build should not be False. This is an internal error. Please file a bug.T)r7rrcrrrrrrgre_input_contextsrmrh _dataset_fnrjrkrrrr) r`rr+input_contexts dataset_fnr.r.r0rkrrus r@rcz(DistributedDatasetsFromFunction.__init__9s}P )49#:%'DDNDM'D*"8 K LLj0 O PP  " "c.&9 9 "  & &N(; < => >,d#'#7#7#:#O#Od #ddk      HJ J   (d!dn#'d dk&t~~a'89d*F ..$..+T->->+@d'rBc|jrJtj}t|j|j |j \|_}tjrtjtj|z dz}tj|jjjt!|j j"j%|t'|j||_t+|jd|_t-|j|jd|j*|_d|_yrp)rjrqrr1_create_datasets_from_function_with_input_contextrrrrrrtruC_distributed_dataset_from_function_initialization_time_millisecondsrwrrurvrxrgryrlrrr)r`r{r0r|s r@rkz%DistributedDatasetsFromFunction.builds{{?#||~9  $"5"5t7G7G I!DNL  "  $  8!9=F GIRR .. " " + + d!!-- .0034J0K8  &D$T^^A%67D(D q)4+<+<)>D%DKrBc t|jt|jjk(s=Jdt|jdt|jjg}t t|jjD]u}t j |j|j5|jtj|j||||jdddwt|j|j||j|jS#1swYxYw)Nr~r)r.r0r.)rerrrirtrrrrrrrhrrrrrs r@rz*DistributedDatasetsFromFunction.auto_shards1 DNNs4#6#6#E#EFF S()*d11@@AB DF  3t**99: ;    T^^A.>> ?   ( (q!:x**      +4+>+>6F8<8J8J37== BB  s ?E  E cH|js td|jSrrrfs r@rz+DistributedDatasetsFromFunction.cardinalityrrBc Dtjs)tjjs t d|j s t dt|jdd}t|j|j|j|}t|j||j|j|j|j|j }|j"|_t%j&rt%j(|S)NrzMYou need to use this dataset in ClusterCoordinator.create_per_worker_dataset.r^Tr)rrr+rr/r.r)r#executing_eagerly_outside_functionsrrrrjrrrrrrrrrrrrrrtru)r`rbrrNs r@rz(DistributedDatasetsFromFunction.__iter__s  3 3 5  ! ! # 5 5 ? @@ ;; G HH#4>>3J#')-   1 3I #))%%$($E$E ))H"//H   "  OrBc|jrJ|jjjr&t j t |jdS|jSrr<rfs r@r0z,DistributedDatasetsFromFunction.element_specrrBcnt|j|j|j|jSrGrrfs r@r2z*DistributedDatasetsFromFunction._type_specs/ .t/B/B/3/A/A/3~~t}} NNrB)NNNNNTN) rvrrrrcrkrrrrr0r2r?r@s@r@rr5st-  N@`,B*   #J     N NrBrc4d}tj||S)z:A function to create dummy tensors from `value_structure`.c t|dr|jSt|tjrE|j ddj |j d|jzd}|j}n|j}|j}|r"|jDcgc]}||nd c}ng}|rBt|tjs|jrtjd|d<t|tj rYtj"t%j&d|t%j&dt)|ft*j,|St%j&tj.||}t|tjrPt%j&d|j0}tj2j5||f|jzd}|Scc}w)z>Create a dummy tensor with possible batch dimensions set to 0._create_empty_valueNr%r)rindices dense_shapeF)validate)hasattrrr1r!r_shape concatenate _ragged_rank_dtyperdtyperrr Dimensionrr SparseTensorrzerosrerr TensorShape_row_splits_dtype RaggedTensorfrom_nested_row_splits)r feature_shape feature_typedimdims dummy_tensor row_splitss r@create_dummy_tensorz-_dummy_tensor_fn..create_dummy_tensorst*+ % % ''$ 667kk"1o11 ++q4,,,. /1m[[ljjmZZl7D6K6K6M NsCOS * N"$  D-"@"@A//1&&q)d1g$ 667  ' 'L1//1c$i.&,,? ??<#;#;D#A<PL$ 667??1d&<&<=j"//FF  (9(99EGKl + Os( H)r#r)value_structurers r@_dummy_tensor_fnrs1f   / AArBc g}t|jD]\}tj|5|j }t|D]\}tj|5|rG|j t jjfdfddn%|j jddd ddd|S#1swYxYw#1swYxYw)aReturns the value of the optionals or dummy values. Args: input_workers: the `InputWorkers`. optional_list: a list of lists `tf.experimental.Optional`. The values from each compute device grouped by the input device. produce_dummy: a bool. Whether to produce dummy tensors when the optional doesn't have a value. Returns: A flatten list of Tensors. c0jSrG)rMrpjrsr@rz%_get_value_or_dummy..FsM!,Q/99;rBc:tjSrG)rr0rsr@rz%_get_value_or_dummy..Gs,]1-=a-@-M-MNrBTrN) rrirrrmrrrrrM) rrrrrrqrrprs ` @@r@rr-s*]99:?ia F ?88;g ) ?)!V ZZ  ?     !!$Q'113;N     mA.q1;;= > ? ? ????"  ? ???s$8D.A/C4  D4C= 9DD c:eZdZdZd dZdZdZd dZd dZdZ y) _SingleWorkerDatasetIteratorBasez(Iterator for a single `tf.data.Dataset`.Nc~||_||_||_|j|_||_|j y)aCreate iterator for the `dataset` to fetch data to worker's `devices` . A `MultiDeviceIterator` or `OwnedMultiDeviceIterator` is used to prefetch input to the devices on the given worker. Args: dataset: A `tf.data.Dataset` instance. worker: Worker on which ops should be created. devices: Distribute data from `dataset` to these devices. options: options. N)_dataset_worker_devicesr0rr_make_iterator)r`r=rrqr.s r@rcz)_SingleWorkerDatasetIteratorBase.__init__Ss:DMDLDM --DDMrBctdrErFrfs r@rz/_SingleWorkerDatasetIteratorBase._make_iteratorfrGrBc|jr@|jjtjk(r|jjs|gS|S)a=Change the data in to a list type if required. The OwnedMultiDeviceIterator returns the list data type, while the PER_REPLICA iterator (when used with prefetch disabled) returns without the enclosed list. This is to fix the inconsistency. Args: data_list: data_list Returns: list )rexperimental_replication_moder PER_REPLICAexperimental_fetch_to_device)r` data_lists r@_format_data_list_with_optionsz?_SingleWorkerDatasetIteratorBase._format_data_list_with_optionsis@ $--EE(() MM 6 6[ rBc~tj|j5t|jr$|j j |cdddS|j j cdddS#1swYyxYw)z&Get next element for the given device.N)rrr!_should_use_multi_device_iteratorr _iteratorr)r`rrs r@rz)_SingleWorkerDatasetIteratorBase.get_next{sd DLL !) *4== 9~~&&v.))~~&&( )))s0A>A>>Bc~tj|j5|j|jj cdddS#1swYyxYw)aGet next element from the underlying iterator. Runs the iterator get_next() within a device scope. Since this doesn't use get_next_as_optional(), it is considerably faster than get_next_as_list(), but it raises EOFError if any of the device doesn't get any data. Args: name: not used. Returns: A list consisting of the next data from each device. N)rrrrrr)r`rs r@rz1_SingleWorkerDatasetIteratorBase.get_next_as_listsK DLL !L  0 01H1H1J KLLLs )AActj|j5|j|jj cdddS#1swYyxYwrG)rrrrrrrfs r@rz:_SingleWorkerDatasetIteratorBase.get_next_as_optional_listsE DLL !1  0 0 .. - - /1111s )AArG) rvrrrrcrrrrrrHrBr@rrPs%0&D$)L"1rBrcjeZdZdZgdZ d dZedZdZdZ edZ dZ d Z e d Zy ) r$z;Type specification for `_SingleWorkerOwnedDatasetIterator`.rrrrr^c||_|rtd|D|_ntd|D|_||_||nt j |_||_y)Nc3FK|]}tj|ywrGrOrQs r@rLz<_SingleWorkerDatasetIteratorSpec.__init__..sIAK44Q7IrRc3FK|]}tj|ywrGrZrQs r@rLz<_SingleWorkerDatasetIteratorSpec.__init__..s"M?@+ 7 7 :MrR)rrTrrr InputOptionsrr^)r`rrqr0r.rbs r@rcz)_SingleWorkerDatasetIteratorSpec.__init__seDLIIIdmMDKMMdm%D ' 3W#002 M!5DrBctSrG)!_SingleWorkerOwnedDatasetIteratorrfs r@rz+_SingleWorkerDatasetIteratorSpec.value_types ,,rBct|j|j|j|j|jfSrGrrfs r@rz+_SingleWorkerDatasetIteratorSpec._serializes0 LL$--););T]]  & & ((rBc tj|jtj}tj|}|}|j t j|j||jyN)r0) r rPrcurrentget_host_for_devicerr MultiDeviceIteratorSpecrr)r`r% device_scope host_devicers r@_get_multi_device_iterator_specz@_SingleWorkerDatasetIteratorSpec._get_multi_device_iterator_specse++DLL+:M:M:OPL11,?KF LL!99 MM60B0B DErBcg}t|jr|j||S|jt j |j |Sr)rrrrrr<r)r`r%s r@r(z1_SingleWorkerDatasetIteratorSpec._component_specssL E(7 **51 L ll<,,$:L:LMN LrBc|jgSrG)rr*s r@r,z/_SingleWorkerDatasetIteratorSpec._to_componentss OO rBc td|j|j||j|j|j S)N)r=rrqr.r0r.rb)r rrrrr^r/s r@r0z1_SingleWorkerDatasetIteratorSpec._from_componentss< ,|| '' !77 99rBct|j|j|j|j|j SrGr$rrrrr^r2s r@rz+_SingleWorkerDatasetIteratorSpec.from_values7 ,EMM5>>,1,?,?,1,G,G IIrBNr~)rvrrrrrcrrrrr(r,r0r4rrHrBr@r$r$siC) %) 6 - -(E  9IIrBr$ceZdZdZ d fd ZdZdZedZedZ edZ edZ ed Z xZ S) r z+Iterator for a DistributedDataset instance.c|| tdd}||_||_|/|| t|||_||_||_|d|_y|| t|tt|'||||jy)a0Create iterator for the `dataset` to fetch data to worker's `devices` . `OwnedMultiDeviceIterator` is used to prefetch input to the devices on the given worker. The lifetime of this iterator is tied to the encompassing python object. Once we go out of scope of the python object or return from a tf.function the underlying iterator resource is deleted. Args: dataset: A `tf.data.Dataset` instance. worker: Worker on which ops should be created. devices: Distribute data from `dataset` to these devices. components: Tensor components to construct the _SingleWorkerOwnedDatasetIterator from. element_spec: A nested structure of `TypeSpec` objects that represents the type specification of elements of the iterator. options: `tf.distribute.InputOptions` used to control options on how this dataset is distributed. canonicalize_devices: Whether to canonicalize devices for workers fully or partially. If False, it will partially canonicalize devices by removing job and task. Nz.Both `worker` and `devices` should be providedzMEither `dataset` or both `components` and `element_spec` need to be provided.r) rrr^rrrrr7r rc) r`r=rrqr.r0r.rbr8rus r@rcz*_SingleWorkerOwnedDatasetIterator.__init__s:~ G HH,MDM!5D   4'''ddldm!!}dn  L$<'' - "FGT]]CrBcrtjsHtj|jtj }tj |}n|j|j}}tj|5|j\tj|j|j||jj|jj|_n1tj|j|j||_dddy#1swYyxYw)N) source_devicemax_buffer_sizeprefetch_buffer_size)r)rinside_functionr rPrrrrrr OwnedMultiDeviceIteratorrr$experimental_per_replica_buffer_sizer)r`rrs r@#_create_owned_multi_device_iteratorzE_SingleWorkerOwnedDatasetIterator._create_owned_multi_device_iterators     --dll.9.A.A.CEl33LAk#',, Kl L ! E  "2KK MM MM% MM 1 1!% 1 133KK MM4== E E E Es BD--D6c|js tdt|jr|j yt j |j5t|j|_ dddy#1swYyxYw)z)Make appropriate iterator on the dataset.z@Worker device must be specified when creating an owned iterator.N) rrrrr#rrrSrrrfs r@rz0_SingleWorkerOwnedDatasetIterator._make_iterator7sh << ) **(7 ..0 ::dll #-dmm,---s BB c|jSrG)rrfs r@r0z._SingleWorkerOwnedDatasetIterator.element_specBs   rBct|j|j|j|j|j SrGrrfs r@r2z,_SingleWorkerOwnedDatasetIterator._type_specFs5 +DLL$--,0,>,> ,0,F,F HHrBcDtjd|jS)a Returns the class of each component of an element of this iterator. The expected values are `tf.Tensor` and `tf.SparseTensor`. Returns: A nested structure of Python `type` objects corresponding to each component of an element of this dataset. c"|jSrG)_to_legacy_output_classescomponent_specs r@rzB_SingleWorkerOwnedDatasetIterator.output_classes..Ws~GGIrBr#rrrfs r@output_classesz0_SingleWorkerOwnedDatasetIterator.output_classesLs#   I  rBcDtjd|jS)zReturns the shape of each component of an element of this iterator. Returns: A nested structure of `tf.TensorShape` objects corresponding to each component of an element of this dataset. c"|jSrG)_to_legacy_output_shapesr*s r@rzA_SingleWorkerOwnedDatasetIterator.output_shapes..cs~FFHrBr,rfs r@ output_shapesz/_SingleWorkerOwnedDatasetIterator.output_shapesZs#   H  rBcDtjd|jS)zReturns the type of each component of an element of this iterator. Returns: A nested structure of `tf.DType` objects corresponding to each component of an element of this dataset. c"|jSrG)_to_legacy_output_typesr*s r@rz@_SingleWorkerOwnedDatasetIterator.output_types..os~EEGrBr,rfs r@ output_typesz._SingleWorkerOwnedDatasetIterator.output_typesfs#   G  rB)NNNNNNN)rvrrrrcr#rrr0r2r-r1r5r?r@s@r@r r s3 $(0DdE@ -   H H             rBr c lt|tsJt|t|jk(sJg}t |jD]X\}}t j |5|j|}t||||||}|j|dddZ|S#1swYfxYw)z5Create a multidevice iterator on each of the workers.)r=rrqr.rbN) r1rDrerirrrrmr r) worker_datasetsrr.rbrrprrirNs r@rrss M< 00 0 _ ]%A%A!B BB B)]99: !ia F !$??Bn2!!$ 3 5h x !! ! !!s (6B**B3 cg}t|D]K\}}|j|}tj|5||}|j |dddM|j fS#1swYexYw)z;Create device datasets per worker given a dataset function.N)rrirrrr0)rrrdatasetsrpctxrr=s r@rrs(.)fa  ) )! ,F F 3goog 7'' ''s A**A3 c:t|tjr |j}t|tjt j fr|St|tjtjfrt|jStd)z!Get the batched dataset from `d`.zUnable to get batched dataset from the input dataset. `batch` `map_and_batch` need to be the last operations on the dataset. The batch operations can be followed by a prefetch.) r1rDatasetV1Adapterr BatchDatasetr_MapAndBatchDatasetPrefetchDataset_OptionsDataset_get_batched_dataset_input_datasetr)rJs r@rArAs;//0 AK,,h.J.JKL H!k11!1134  0 0 11< ==rBct|tjtjfsJt|tjr|j }|j }n2t|tjr|j}|j}tjrtj|}tjrtj|}||fS)z.Get `batch_size`, `drop_remainder` of dataset.) r1rr=rr> _batch_size_drop_remainder _batch_size_t_drop_remainder_tr is_tf_typeconstant_value)rJrdrop_remainders r@_get_batched_dataset_attributesrKs A --x/K/KL NN N;++,J&&N!X112J((NJ'++J7JN+ //?N ^ ##rBc8t|}t|\}}d}t|tjr |j }nTt|tj r:t|jtjr|jj }|||fS)z6Get the underlying attributes from the dataset object.N)rArKr1rr? _buffer_sizer<r)r=batched_datasetrrJprefetch_buffers r@_get_dataset_attributesrPs)1/>O*n/445**O7K889 w'')D)D E&&33O ^_ 44rBc|F|jtjk(s)|jtjk(r |jryy)z3Determine whether to use multi_device_iterator_ops.TF)rr PER_WORKERrr)r.s r@rrs@ o ++/C/N/NN,,0D0P0PP  / /  rBcJeZdZdZdZedZdZd dZedZ dZ y) MultiStepContextaA context object that can be used to capture things when running steps. This context object is useful when running multiple steps at a time using the `experimental_run_steps_on_iterator` API. For e.g. it allows the user's step function to specify which outputs to emit at what frequency. Currently it supports capturing output from the last step, as well as capturing non tensor outputs. In the future it will be augmented to support other use cases such as output each N steps. c.i|_i|_i|_y)zHInitialize an output context. Returns: A context object. N)_last_step_outputs_last_step_outputs_reduce_ops_non_tensor_outputsrfs r@rczMultiStepContext.__init__s !D)+D&!DrBc|jS)aA dictionary consisting of outputs to be captured on last step. Keys in the dictionary are names of tensors to be captured, as specified when `set_last_step_output` is called. Values in the dictionary are the tensors themselves. If `set_last_step_output` was called with a `reduce_op` for this output, then the value is the reduced value. Returns: A dictionary with last step outputs. )rVrfs r@last_step_outputsz"MultiStepContext.last_step_outputss  " ""rBcHt|ts td||_y)z3Replace the entire dictionary of last step outputs.z+Need a dictionary to set last_step_outputs.N)r1dictrrV)r`outputss r@_set_last_step_outputsz'MultiStepContext._set_last_step_outputss gt $ D EE%DrBNcBtjrWj<|j<ytj}|j |dj<yJfd}tj j||fy)aSet `output` with `name` to be outputted from the last step. Args: name: String, name to identify the output. Doesn't need to match tensor name. output: The tensors that should be outputted with `name`. See below for actual types supported. reduce_op: Reduction method to use to reduce outputs from multiple replicas. Required if `set_last_step_output` is called in a replica context. Optional in cross_replica_context. When present, the outputs from all the replicas are reduced using the current distribution strategy's `reduce` method. Hence, the type of `output` must be what's supported by the corresponding `reduce` method. For e.g. if using MirroredStrategy and reduction is set, output must be a `PerReplica` value. The reduce method is also recorded in a dictionary `_last_step_outputs_reduce_ops` for later interpreting of the outputs as already reduced or not. Nrcf|j|dj<j<y)Nr)rrVrW) distributionr+r reduce_opr`s r@merge_fnz7MultiStepContext.set_last_step_output..merge_fn's=(4(;(;IuAE)<)G% 4=**40rBargs)r in_cross_replica_contextrWrV get_strategyrr merge_call)r`routputrbrarcs`` ` r@set_last_step_outputz%MultiStepContext.set_last_step_output s(..01:d((.  (.%%224 (4(;(;IvAE)<)G% "" "=((*55 &6$rBc|jS)zAA dictionary consisting of any non tensor outputs to be captured.)rXrfs r@non_tensor_outputsz#MultiStepContext.non_tensor_outputs2s  # ##rBctjr|j<yfd}tjj ||fy)z?Set `output` with `name` to be captured as a non tensor output.cB|j|j<yrG)experimental_local_resultsrX)rar+rr`s r@rcz8MultiStepContext.set_non_tensor_output..merge_fn<s#  3 3E :   &rBrdN)r rfrXrrh)r`rrircs`` r@set_non_tensor_outputz&MultiStepContext.set_non_tensor_output7sI..0'-dt$< ((*55 &6$rBrG) rvrrrrcrrZr^rjrlrprHrBr@rTrTsD"  #  #& '$R $ $ $rBrTct|jjt|s|Sfd}t j ||S)aCreate a `tf.TypeSpec` for a given strategy and input `tensor_spec`. Args: strategy: The given `tf.distribute` strategy. tensor_spec: `tf.TensorSpec` of a given value. The batch dimension of the shape should be None if you have partial batches. Returns: A `tf.TypeSpec` that matches the values produced by a given strategy. This can be a `tf.TensorSpec` or a `PerRelicaSpec`. cbtDcgc]}|}}tj|Scc}wrG)rtrPerReplicaSpec)tensor_spec_per_inputrK value_specs num_replicass r@_get_value_per_replicaz?_create_distributed_tensor_spec.._get_value_per_replica[s327 2EFQ(FKF  + ..Gs ,)rer6ri _always_wrapr#r)r+ tensor_specrwrvs @r@rlrlEsFX&&556, h  /   2K @@rBcXt|tjr|j|S|S)zBIf `spec` is a `PerReplicaSpec`, then return its `i`th value_spec.)r1rrs _value_specs)rrps r@r#r#bs(f++,   Q  KrBc"tjrQtj|jj5|j j cdddStjS#1swYtjSxYw)z'Returns the cardinality of the dataset.N) rrtrrrrnumpyrr)r=s r@rrjsh  G++22 3+  " ( ( *++   +   s A66Bc t|jdt|jddsy|tjk(ryt |j  xs|jj S)z7Returns whether to enable using partial batch handling.enable_partial_batch_handling(experimental_enable_get_next_as_optionalF)rr6rrrr0r)r+r=rs r@rrrsz 8 h!K   O,,,  "   I'00FFHIrBcJt|}tj||}|S)ayCreates a PerReplica. For strategies other than OneDeviceStrategy, it creates a PerReplica whose type spec is set to the element spec of the dataset. This helps avoid retracing for partial batches. Retracing is problematic for multi client when different client retraces different time, since retracing changes the collective keys in the tf.function, and causes mismatches among clients. For single client strategies, this simply calls distribute_utils.regroup(). Args: value_list: a list of values, one for each replica. strategy: the `tf.distribute.Strategy`. Returns: a structure of PerReplica. ) always_wrap)rxr regroup)rr+r per_replicass r@rrs&(X&+!))*+N, rBc~|jjxs"t|jjdkDS)zAReturns whether to always wrap the values in a DistributedValues.r%)r6rreri)r+s r@rxrxs<    0 0 2 ,c&&7(*+7,,rBct|tjsJ|d}tjtj||j S)z3Rebatch the spec to have a dynamic batch dimension.cb |jjdS#t$rY|SwxYwrG)_unbatch_batchr)rs r@_rebatchz%_rebatch_as_dynamic.._rebatchs5 ]]_ # #D ))   K s ! ..)r1rrsr#rr{)per_replica_specrs r@r=r=sR $f&;&; <N>NN <    ($4$A$AB DDrBcLd|jjd}t|)Nzenumerate not supported with z` types within tf.functions. Use a for loop over the dataset and keep a separate counter instead.)rurvr)r unused_startmsgs r@_ag_enumerate_not_implementedrs0%akk&:&:%;< C  rB)NF)grr!rrqr%tensorflow.python.autograph.operatorsr'tensorflow.python.data.experimental.opsrrrrtensorflow.python.data.opsrrr r tensorflow.python.distributer r r rrr+tensorflow.python.distribute.distribute_librtensorflow.python.eagerrrtensorflow.python.frameworkrrrrrrrrrrtensorflow.python.opsrrrrr tensorflow.python.ops.raggedr!tensorflow.python.platformr"r:tensorflow.python.typesdistribute_typestensorflow.python.utilr#tensorflow.python.util.compatr$SamplerExponentialBucketsrvrrAobjectrDrrIteratorDistributedIteratorInterfacerTypeSpecrr5CompositeTensorrIterableDistributedDatasetInterfacerBr3rXr4rrrrr$r rrrArKrPrrTrlr#rrrrxr=renumerate_registryregisterrHrBr@rs7 =<R>23@347924/L+.8;..+5431+1*,6<B'99K 8J8J;!J!!LJ 95J M% %%12 7 ID&T5$65$p4::L6o66.KKL6^QG (:(:QGh.H?.HbI1*::IX _--%AA F(H>(HXqG)9)I)IqGh $ *K$ P|Nn&6&F&F|N~6Br FH1vH1VCIy'9'9CILO(H(8(H(HOh*.6;* (=$$.5*c$vc$LA:!I22, D$! ''6 ''5rB