AVhu dZddlZddlZddlZddlmZddlmZddlm Z ddlm Z ddlm Z ddlm Z dd lm Z dd lmZdd lmZdd lmZdd lmZddlmZddlmZddlmZddlddlmZddlmZddlmZdZ dEdZdZ dZ!edgdejDddgGddZ#dZ$ed gd!ejDd"d#gGd$d#e#Z%ed%d%d&gejDd&Gd'd&e#Z&Gd(d)e#Z'ed*gd+ejDd,d-gGd.d-e#Z(ed/gd0ejDd1d2gGd3d2e#Z)Gd4d5Z*ed6gGd7d6Z+ed8gGd9d8e+Z,ed:d;gGd<d;e+Z-Gd=d>Z.Gd?d@e.Z/GdAdBe.Z0GdCdDZ1y)FzData Flow Operations.N)contextdtypes)indexed_slices)ops) random_seed) tensor_shape) tensor_util) python_io) array_ops)array_ops_stack)control_flow_ops)gen_data_flow_ops)math_ops)resource_variable_ops)*) deprecation)collections_abc) tf_exportcf|Jt|tst|ts|gSt|S)z"Convert dtypes to a list of types.) isinstancelisttuplers S/home/dcms/DCMS/lib/python3.12/site-packages/tensorflow/python/ops/data_flow_ops.py _as_type_listr-s4    VT "j&? 8O <c~|r9t|tjr|rtd|Dr t d|yt|t j r|g}t|ttfstdt|dtd|Dr|g}|Dcgc]}t j|}}|s td|Drt d||s td |Drt d ||Scc}w) z4Convert shapes to a list of tuples of int (or None).c3HK|]}|duxst|tywNrint.0shapes r z!_as_shape_list..@s# HETM 3Zs3 3 H "zAWhen providing partial shapes, a list of shapes must be provided.NzEShapes must be a TensorShape or a list or tuple of TensorShapes, got z instead.c3HK|]}|duxst|tywrr r"s rr%z!_as_shape_list..Ks#EU$ 0*UC0 0Er&c3>K|]}|j ywr)is_fully_definedr"s rr%z!_as_shape_list..Ps .Ss 2%5::  2sz%All shapes must have a defined rank: ) rrSequenceany ValueErrorr TensorShaperr TypeErrortypeallas_shape)shapesrunknown_dim_allowedunknown_rank_allowedr$s r_as_shape_listr78s   v77 8 H HH  M OO ^  001XF FUDM *  F|nI ' (( EfEEXF6< =UL ! !% ( =& =  vhG HH - >s$Dc|yt|ttfs|g}t|t|k7r#t dt|dt|t|S)NzSList of names must have the same length as the list of dtypes, received len(names)=z ,len(dtypes)=)rrrlenr.)namesrs r _as_name_listr;Ysj ]  ED%= ) GEZ3v; 77:5zlC$$'K=2 33 erctj|}tj|}|j%|j|j|jk7rtjSt |j |j Dcgc]\}}|||k(r|nd}}}tj|Scc}}w)z>The greatest lower bound (ordered by specificity) TensorShape.N)r r/ndims unknown_shapezipas_list)s1s2d1d2ds r _shape_commonrFes#"#"XX)RXX-A  % % ''"**, 5 2r NrRxbT1!  ! !! $$ sCqueue.QueueBase)rG io.QueueBase QueueBase)v1rHrIceZdZdZdZedZedZedZ edZ edZ edZ d Z d Zdd Zdd ZdZddZddZddZddZddZddZy )rIaBase class for queue implementations. A queue is a TensorFlow data structure that stores tensors across multiple steps, and exposes operations that enqueue and dequeue tensors. Each queue element is a tuple of one or more tensors, where each tuple component has a static dtype, and may have a static shape. The queue implementations support versions of enqueue and dequeue that handle single elements, versions that support enqueuing and dequeuing a batch of elements at once. See `tf.queue.FIFOQueue` and `tf.queue.RandomShuffleQueue` for concrete implementations of this class, and instructions on how to create them. cb||_|ct|t|k7r#tdt|dt||Dcgc]}tj|c}|_n1|jDcgc]}tj c}|_|Bt|t|k7r#tdt|dt|||_nd|_||_t|tjretjjr$tjj|_nd|_tj |d|_y|jj$j&j)dd|_ycc}wcc}w) a$Constructs a queue object from a queue reference. The two optional lists, `shapes` and `names`, must be of the same length as `dtypes` if provided. The values at a given index `i` indicate the shape and name to use for the corresponding queue component in `dtypes`. Args: dtypes: A list of types. The length of dtypes must equal the number of tensors in each element. shapes: Constraints on the shapes of tensors in an element: A list of shape tuples or None. This list is the same length as dtypes. If the shape of any tensors in the element are constrained, all must be; shapes can be None if the shapes should not be constrained. names: Optional list of names. If provided, the `enqueue()` and `dequeue()` methods will use dictionaries with these names as keys. Must be None or a list or tuple of the same length as `dtypes`. queue_ref: The queue reference, i.e. the output of the queue op. Raises: ValueError: If one of the arguments is invalid. NzGQueue shapes must have the same length as dtypes, received len(shapes)=z, len(dtypes)=zDQueue names must have the same length as dtypes,received len(names)=z,len Empty/)_dtypesr9r.r r/_shapesr>_names _queue_refrr EagerTensorr scope_name_namerEagerResourceDeleter_resource_deleteropnamesplit)selfrr4r: queue_refs_s r__init__zQueueBase.__init__sl,DL  VF #114V >((+F }67 7.s:aAy):zA list of queues expectedrc3<K|]}|jk(ywrr)r#qrs rr%z&QueueBase.from_list..s6av!6Nz-Queues do not have matching component dtypes.c3<K|]}|jk(ywr)r:)r#rfr:s rr%z&QueueBase.from_list..s4Au4rgz,Queues do not have matching component names.)rr4r:r])rrr2r0rr:r4r? functoolsreducerFr stackr]r gatherrI) indexqueuesrf queue_shapesr^reduced_shapesrc queue_refsselected_queuerr:s @@r from_listzQueueBase.from_lists& Z5 :6: : 1 22 AY  F 66!":6 6 E FF 1IOOE 44 4 D EE&,-AHH-L-474F/0 *N!&&V'D 'DEJ%%j%8N   "".(EsD,7!D1-D6c|jS)zThe underlying queue reference.)rSr\s rr]zQueueBase.queue_refs ??rctjr |jS|jjj S)z!The name of the underlying queue.)rexecuting_eagerlyrVrSrYrZrvs rrZzQueueBase.names0  " ZZ ??   " ""rc|jS)z9The list of dtypes for each component of a queue element.rPrvs rrzQueueBase.dtypes <<rc|jS)z9The list of shapes for each component of a queue element.rQrvs rr4zQueueBase.shapesr{rc|jS)z8The list of names for each component of a queue element.rRrvs rr:zQueueBase.names ;;rc t|tr|js tdt |jt t |j t k7r;tdt |j dt |j|jDcgc]}|| }}n0|jr tdt|ttfs|g}g}tt||jD]2\}\}}|jtj||d|z4|Scc}w)aHValidate and convert `vals` to a list of `Tensor`s. The `vals` argument can be a Tensor, a list or tuple of tensors, or a dictionary with tensor values. If it is a dictionary, the queue must have been constructed with a `names` attribute and the dictionary keys must match the queue names. If the queue was constructed with a `names` attribute, `vals` must be a dictionary. Args: vals: A tensor, a list or tuple of tensors, or a dictionary.. Returns: A list of `Tensor` objects. Raises: ValueError: If `vals` is invalid. z-Queue must have names to enqueue a dictionary)keyzHKeys in dictionary to enqueue do not match names of Queue. Dictionary: z,Queue: z3You must enqueue a dictionary in a Queue with names component_%ddtyperZ)rdictrRr.sortedstrkeysrr enumerater?rPappendrconvert_to_tensor)r\valsktensorsivalrs r_check_enqueue_dtypeszQueueBase._check_enqueue_dtypess!($ [[HII   % )E E99? 9L8MN##)$++#6"79: : ${{ +!d1g +d + NOO tUm ,vG$St||%<=L>> q = tf.queue.FIFOQueue(capacity=3, dtypes=tf.int32) >>> q.enqueue(1) >>> q.enqueue(2) >>> q.size() >>> q = tf.queue.FIFOQueue(2, tf.int32, shapes=tf.TensorShape(4)) >>> q.enqueue(tf.constant([1, 2, 3, 4], dtype=tf.int32)) >>> q.size() Args: vals: A tensor, a list or tuple of tensors, or a dictionary containing the values to enqueue. name: A name for the operation (optional). Returns: The operation that enqueues a new tuple of tensors to the queue. z %s_enqueuerZN)r name_scoperVrrr?rQ get_shapeassert_is_compatible_withrSrrPresourcerqueue_enqueue_v2 queue_enqueue)r\rrZscoperr$s renqueuezQueueBase.enqueue:sD lTZZ7((. 0/38  ' ' -dD$,,/9*#u 11%89   '"2"2 2 11 OOT///!.. OOT////sBD!DD ctj|d|jz|j|5}|j |}t j |djjdd}t j|}t||jD]\}}t j |jjdd}t j|}|j|}|jddj|tj|j ||cdddS#1swYyxYw)aEnqueues zero or more elements to this queue. This operation slices each component tensor along the 0th dimension to make multiple queue elements. All of the tensors in `vals` must have the same size in the 0th dimension. If the queue is full when this operation executes, it will block until all of the elements have been enqueued. At runtime, this operation may raise an error if the queue is `tf.QueueBase.close` before or during its execution. If the queue is closed before this operation runs, `tf.errors.CancelledError` will be raised. If this operation is blocked, and either (i) the queue is closed by a close operation with `cancel_pending_enqueues=True`, or (ii) the session is `tf.Session.close`, `tf.errors.CancelledError` will be raised. >>> q = tf.queue.FIFOQueue(capacity=10, dtypes=tf.int32) >>> q.enqueue_many(tf.constant([1, 2, 3, 4, 5], dtype=tf.int32)) >>> q.size() Args: vals: A tensor, a list or tuple of tensors, or a dictionary from which the queue elements are taken. name: A name for the operation (optional). Returns: The operation that enqueues a batch of tuples of tensors to the queue. z%s_EnqueueManyrrhNr)rrrVrrr dimension_valuerwith_rank_at_least Dimensionr?rQ merge_withrrqueue_enqueue_many_v2rS)r\rrZr batch_dimrr$ val_batch_dims r enqueue_manyzQueueBase.enqueue_manyls4@ .;((. 0-38  ' ' -d .. q'    0 0 3A 68i((3iD$,,/=*#u$44 MMO . .q 1! 46 $..}= ((7  55e< = 4 4 //4e-%---s DEE"c|jr,t|jDcic] \}}||| c}}St|dk(r|dS|Scc}}w)aReturn the value to return from a dequeue op. If the queue has names, return a dictionary with the names as keys. Otherwise return either a single tensor or a list of tensors depending on the length of `tensors`. Args: tensors: List of tensors from the dequeue op. Returns: A single tensor, a list of tensors, or a dictionary of tensors. rhr)rRrr9)r\rrns r_dequeue_return_valuezQueueBase._dequeue_return_valuesT {{)2$++(> ?1am ?? W  QZ n @sAc|d|jz}|jjtjk(r-t j |j|j|}n,t j|j|j|}tjsL|dj}t|j|jD]\}}|j||j|S)aDequeues one element from this queue. If the queue is empty when this operation executes, it will block until there is an element to dequeue. At runtime, this operation may raise an error if the queue is `tf.QueueBase.close` before or during its execution. If the queue is closed, the queue is empty, and there are no pending enqueue operations that can fulfill this request, `tf.errors.OutOfRangeError` will be raised. If the session is `tf.Session.close`, `tf.errors.CancelledError` will be raised. >>> q = tf.queue.FIFOQueue(capacity=2, dtypes=tf.int32) >>> q.enqueue(1) >>> q.enqueue(2) >>> q.dequeue() >>> q.dequeue() Args: name: A name for the operation (optional). Returns: The tuple of tensors that was dequeued. z %s_Dequeuerr)rVrSrrPrrqueue_dequeue_v2 queue_dequeuerrxrYr?rrQ set_shaper)r\rZretrYoutputr$s rdequeuezQueueBase.dequeues8 | DJJ &d  0 00  . . //4<>> q = tf.queue.FIFOQueue(10, tf.int32, shapes=tf.TensorShape(2)) >>> q.enqueue(tf.constant([1, 2], dtype=tf.int32, shape=(2))) >>> q.enqueue(tf.constant([3, 4], dtype=tf.int32, shape=(2))) >>> q.enqueue(tf.constant([5, 6], dtype=tf.int32, shape=(2))) >>> q.enqueue(tf.constant([7, 8], dtype=tf.int32, shape=(2))) >>> q.dequeue_many(3) Args: n: A scalar `Tensor` containing the number of elements to dequeue. name: A name for the operation (optional). Returns: The list of concatenated tensors that was dequeued. z%s_DequeueManyrcomponent_typesrZrrh)rVrqueue_dequeue_many_v2rSrPrrxrYr rr constant_valueinputsr?rrQrr/ concatenater)r\rrZrrYrrr$s r dequeue_manyzQueueBase.dequeue_manysH |  *d  1 1 1dll GC  $ $ & q699b((  $ $RYYq\ 24iryy{DLL9F-&%  $ $i[ 1 = =e D FF  % %c **rc|d|jz}tj|j||j|}t j so|dj}t|j|jD]9\}}|jtjdgj|;|j|S)aDequeues and concatenates `n` elements from this queue. **Note** This operation is not supported by all queues. If a queue does not support DequeueUpTo, then a `tf.errors.UnimplementedError` is raised. This operation concatenates queue-element component tensors along the 0th dimension to make a single component tensor. If the queue has not been closed, all of the components in the dequeued tuple will have size `n` in the 0th dimension. If the queue is closed and there are more than `0` but fewer than `n` elements remaining, then instead of raising a `tf.errors.OutOfRangeError` like `tf.QueueBase.dequeue_many`, less than `n` elements are returned immediately. If the queue is closed and there are `0` elements left in the queue, then a `tf.errors.OutOfRangeError` is raised just like in `dequeue_many`. Otherwise the behavior is identical to `dequeue_many`. >>> q = tf.queue.FIFOQueue(10, tf.int32, shapes=tf.TensorShape(2)) >>> q.enqueue(tf.constant([1, 2], dtype=tf.int32, shape=(2))) >>> q.enqueue(tf.constant([3, 4], dtype=tf.int32, shape=(2))) >>> q.close() >>> q.dequeue_up_to(5) Args: n: A scalar `Tensor` containing the number of elements to dequeue. name: A name for the operation (optional). Returns: The tuple of concatenated tensors that was dequeued. Nz%s_DequeueUpTorr)rVrqueue_dequeue_up_to_v2rSrPrrxrYr?rrQrr r/rr)r\rrZrrYrr$s r dequeue_up_tozQueueBase.dequeue_up_tosF |  *d  2 2 1dll GC  $ $ & q699bryy{DLL9N-&%114&9EEeLMN  % %c **rc|d|jz}|jjtjk(r"t j |j||St j|j||S)aCloses this queue. This operation signals that no more elements will be enqueued in the given queue. Subsequent `enqueue` and `enqueue_many` operations will fail. Subsequent `dequeue` and `dequeue_many` operations will continue to succeed if sufficient elements remain in the queue. Subsequently, dequeue and dequeue_many operations that would otherwise block waiting for more elements (if close hadn't been called) will now fail immediately. If `cancel_pending_enqueues` is `True`, all pending requests will also be canceled. >>> q = tf.queue.FIFOQueue(capacity=3, dtypes=tf.int32) >>> q.is_closed() >>> q.close() >>> q.is_closed() Args: cancel_pending_enqueues: (Optional.) A boolean, defaulting to `False` (described above). name: A name for the operation (optional). Returns: The operation that closes the queue. z%s_Closecancel_pending_enqueuesrZ)rVrSrrPrrqueue_close_v2 queue_closer\rrZs rclosezQueueBase.closeNsq: | $** $d  0 00  - - //"9  * * //"9rc|d|jz}|jjtjk(r!t j |j|St j|j|S)aReturns true if queue is closed. This operation returns true if the queue is closed and false if the queue is open. >>> q = tf.queue.FIFOQueue(capacity=3, dtypes=tf.int32) >>> q.is_closed() Args: name: A name for the operation (optional). Returns: True if the queue is closed and false if the queue is open. z %s_Is_Closedr)rVrSrrPrrqueue_is_closed_v2queue_is_closed_r\rZs r is_closedzQueueBase.is_closedxs]  | djj (d  0 00  1 1$// MM  / /d KKrc|d|jz}|jjtjk(r!t j |j|St j|j|S)aCompute the number of elements in this queue. >>> q = tf.queue.FIFOQueue(capacity=10, dtypes=tf.int32) >>> q.enqueue_many(tf.constant([1, 2, 3, 4], dtype=tf.int32)) >>> q.size() Args: name: A name for the operation (optional). Returns: A scalar tensor containing the number of elements in this queue. z%s_Sizer)rVrSrrPrr queue_size_v2 queue_sizers rsizezQueueBase.sizes] |  #d  0 00  , ,T__4 HH  ) )$// EErrFN)__name__ __module__ __qualname____doc__r` staticmethodrtpropertyr]rZrr4r:rrrrrrrrrrrrrrIrIss$0:d&"&"P   # #       )V 0/d3-j.,+\4+l0+d(TL.FrchtjrttjS|Sr)rrxrruid) shared_names r _shared_namers$  swwy> rqueue.RandomShuffleQueue)rio.RandomShuffleQueueRandomShuffleQueuerrc.eZdZdZ dfd ZxZS)rzA queue implementation that dequeues elements in a random order. See `tf.queue.QueueBase` for a description of the methods on this class. c t|}t||}t||}tj|\} } | | d\} } nT|R|Pt | |zj d} ttj| jddddz} tj||||| | t||} tt|C|||| y)a"Create a queue that dequeues elements in a random order. A `RandomShuffleQueue` has bounded capacity; supports multiple concurrent producers and consumers; and provides exactly-once delivery. A `RandomShuffleQueue` holds a list of up to `capacity` elements. Each element is a fixed-length tuple of tensors whose dtypes are described by `dtypes`, and whose shapes are optionally described by the `shapes` argument. If the `shapes` argument is specified, each component of a queue element must have the respective fixed shape. If it is unspecified, different queue elements may have different shapes, but the use of `dequeue_many` is disallowed. The `min_after_dequeue` argument allows the caller to specify a minimum number of elements that will remain in the queue after a `dequeue` or `dequeue_many` operation completes, to ensure a minimum level of mixing of elements. This invariant is maintained by blocking those operations until sufficient elements have been enqueued. The `min_after_dequeue` argument is ignored after the queue has been closed. Args: capacity: An integer. The upper bound on the number of elements that may be stored in this queue. min_after_dequeue: An integer (described above). dtypes: A list of `DType` objects. The length of `dtypes` must equal the number of tensors in each queue element. shapes: (Optional.) A list of fully-defined `TensorShape` objects with the same length as `dtypes`, or `None`. names: (Optional.) A list of string naming the components in the queue with the same length as `dtypes`, or `None`. If specified the dequeue methods return a dictionary with the names as keys. seed: A Python integer. Used to create a random seed. See `tf.compat.v1.set_random_seed` for behavior. shared_name: (Optional.) If non-empty, this queue will be shared under the given name across multiple sessions. name: Optional name for the queue operation. N)rrzutf-8i)rr4capacitymin_after_dequeueseedseed2rrZ)rr7r;rget_seedrencoder!hashlibmd5 hexdigestrrandom_shuffle_queue_v2rsuperrr`)r\rrrr4r:rrrZseed1rstringr] __class__s rr`zRandomShuffleQueue.__init__sf6 "F FF +F % (E''-LE5 }leU +1 E [(009f'++f%//1"15r:ZGe!99+  - I d,VVUINr)NNNNrandom_shuffle_queuerrrrr` __classcell__rs@rrrs'*JOJOrzqueue.FIFOQueue FIFOQueuec,eZdZdZ dfd ZxZS)rzA queue implementation that dequeues elements in first-in first-out order. See `tf.queue.QueueBase` for a description of the methods on this class. c dt|}t||}t||}tj5tj d5t j|||t||}ddddddtt|/|||y#1swY(xYw#1swY,xYw)a/Creates a queue that dequeues elements in a first-in first-out order. A `FIFOQueue` has bounded capacity; supports multiple concurrent producers and consumers; and provides exactly-once delivery. A `FIFOQueue` holds a list of up to `capacity` elements. Each element is a fixed-length tuple of tensors whose dtypes are described by `dtypes`, and whose shapes are optionally described by the `shapes` argument. If the `shapes` argument is specified, each component of a queue element must have the respective fixed shape. If it is unspecified, different queue elements may have different shapes, but the use of `dequeue_many` is disallowed. Args: capacity: An integer. The upper bound on the number of elements that may be stored in this queue. dtypes: A list of `DType` objects. The length of `dtypes` must equal the number of tensors in each queue element. shapes: (Optional.) A list of fully-defined `TensorShape` objects with the same length as `dtypes`, or `None`. names: (Optional.) A list of string naming the components in the queue with the same length as `dtypes`, or `None`. If specified the dequeue methods return a dictionary with the names as keys. shared_name: (Optional.) If non-empty, this queue will be shared under the given name across multiple sessions. name: Optional name for the queue operation. >>> q = tf.queue.FIFOQueue(capacity=10, dtypes=tf.int32) >>> q.size() CPUrr4rrrZN) rr7r;r init_scopedevicer fifo_queue_v2rrrr` r\rrr4r:rrZr]rs rr`zFIFOQueue.__init__ sP6 "F FF +F % (E  3::e,#11 ";/ i )T#FFE9Es#B&$B3B&B# B&&B/NNN fifo_queuerrs@rrrs" 3F3Frc<eZdZdZ dfd ZddZddZxZS)GPUCompatibleFIFOQueueaA queue implementation that dequeues elements in first-in first-out order. GPUCompatibleFIFOQueue is like FIFOQueue, but the queue resource may be placed either on a CPU or on a GPU. It is not cross-device: enqueues and dequeues will be colocated with the queue resource. GPUCompatibleFIFOQueue only supports enqueue and dequeue at the moment, not enqueue_many or dequeue_many. See `tf.queue.QueueBase` for a description of the methods in this class. c t|}t||}t||}tj5t j |||t||}dddtt|+|||y#1swY xYw)aCreates a queue that dequeues elements in a first-in first-out order. A `FIFOQueue` has bounded capacity; supports multiple concurrent producers and consumers; and provides exactly-once delivery. A `FIFOQueue` holds a list of up to `capacity` elements. Each element is a fixed-length tuple of tensors whose dtypes are described by `dtypes`, and whose shapes are optionally described by the `shapes` argument. If the `shapes` argument is specified, each component of a queue element must have the respective fixed shape. If it is unspecified, different queue elements may have different shapes, but the use of `dequeue_many` is disallowed. Args: capacity: An integer. The upper bound on the number of elements that may be stored in this queue. dtypes: A list of `DType` objects. The length of `dtypes` must equal the number of tensors in each queue element. shapes: (Optional.) A list of fully-defined `TensorShape` objects with the same length as `dtypes`, or `None`. names: (Optional.) A list of strings naming the components in the queue with the same length as `dtypes`, or `None`. If specified the dequeue methods return a dictionary with the names as keys. shared_name: (Optional.) If non-empty, this queue will be shared under the given name across multiple sessions. name: Optional name for the queue operation. rN) rr7r;rrrrrrrr`rs rr`zGPUCompatibleFIFOQueue.__init__QsH6 "F FF +F % (E  #11 ";/ i $0y*s $A<<Bctd)z8enqueue_many is not supported on GPUCompatibleFIFOQueue._GPUCompatibleFIFOQueue does not support enqueue_many or dequeue_many, only enqueue and dequeue.NotImplementedError)r\rrZs rrz#GPUCompatibleFIFOQueue.enqueue_many  $ %%rctd)z8dequeue_many is not supported on GPUCompatibleFIFOQueue.rr)r\rrZs rrz#GPUCompatibleFIFOQueue.dequeue_manyr rrr)rrrrr`rrrrs@rrrFs& 0*d% %rrqueue.PaddingFIFOQueue)r io.PaddingFIFOQueuePaddingFIFOQueuer r c*eZdZdZ dfd ZxZS)r a(A FIFOQueue that supports batching variable-sized tensors by padding. A `PaddingFIFOQueue` may contain components with dynamic shape, while also supporting `dequeue_many`. See the constructor for more details. See `tf.queue.QueueBase` for a description of the methods on this class. c8t|}t||d}t||}t|t|k7r$t dt|dt|dt j |||t||}tt|+||||y)aCreates a queue that dequeues elements in a first-in first-out order. A `PaddingFIFOQueue` has bounded capacity; supports multiple concurrent producers and consumers; and provides exactly-once delivery. A `PaddingFIFOQueue` holds a list of up to `capacity` elements. Each element is a fixed-length tuple of tensors whose dtypes are described by `dtypes`, and whose shapes are described by the `shapes` argument. The `shapes` argument must be specified; each component of a queue element must have the respective shape. Shapes of fixed rank but variable size are allowed by setting any shape dimension to None. In this case, the inputs' shape may vary along the given dimension, and `dequeue_many` will pad the given dimension with zeros up to the maximum shape of all elements in the given batch. Args: capacity: An integer. The upper bound on the number of elements that may be stored in this queue. dtypes: A list of `DType` objects. The length of `dtypes` must equal the number of tensors in each queue element. shapes: A list of `TensorShape` objects, with the same length as `dtypes`. Any dimension in the `TensorShape` containing value `None` is dynamic and allows values to be enqueued with variable size in that dimension. names: (Optional.) A list of string naming the components in the queue with the same length as `dtypes`, or `None`. If specified the dequeue methods return a dictionary with the names as keys. shared_name: (Optional.) If non-empty, this queue will be shared under the given name across multiple sessions. name: Optional name for the queue operation. Raises: ValueError: If shapes is not a list of shapes, or the lengths of dtypes and shapes do not match, or if names is specified and the lengths of dtypes and names do not match. T)r5z9Shapes must be provided for all components, but received z dtypes and z shapes.rN) rr7r;r9r.rpadding_fifo_queue_v2rrr r`rs rr`zPaddingFIFOQueue.__init__sZ6 "F FF EF % (E 6{c&k! ''*6{m<f+h0 11"77 -  I D*665)Lr)NNpadding_fifo_queuerrs@rr r s ( ;M;Mrqueue.PriorityQueue)rio.PriorityQueue PriorityQueuerrc,eZdZdZ dfd ZxZS)rzA queue implementation that dequeues elements in prioritized order. See `tf.queue.QueueBase` for a description of the methods on this class. ct|}t||}tj|||t ||}t j g|z}|rdg|zn|} tt|'|| ||y)aCreates a queue that dequeues elements in a first-in first-out order. A `PriorityQueue` has bounded capacity; supports multiple concurrent producers and consumers; and provides exactly-once delivery. A `PriorityQueue` holds a list of up to `capacity` elements. Each element is a fixed-length tuple of tensors whose dtypes are described by `types`, and whose shapes are optionally described by the `shapes` argument. If the `shapes` argument is specified, each component of a queue element must have the respective fixed shape. If it is unspecified, different queue elements may have different shapes, but the use of `dequeue_many` is disallowed. Enqueues and Dequeues to the `PriorityQueue` must include an additional tuple entry at the beginning: the `priority`. The priority must be an int64 scalar (for `enqueue`) or an int64 vector (for `enqueue_many`). Args: capacity: An integer. The upper bound on the number of elements that may be stored in this queue. types: A list of `DType` objects. The length of `types` must equal the number of tensors in each queue element, except the first priority element. The first tensor in each element is the priority, which must be type int64. shapes: (Optional.) A list of fully-defined `TensorShape` objects, with the same length as `types`, or `None`. names: (Optional.) A list of strings naming the components in the queue with the same length as `dtypes`, or `None`. If specified, the dequeue methods return a dictionary with the names as keys. shared_name: (Optional.) If non-empty, this queue will be shared under the given name across multiple sessions. name: Optional name for the queue operation. rrN) rr7rpriority_queue_v2rrPint64rrr`) r\rtypesr4r:rrZr]priority_dtypespriority_shapesrs rr`zPriorityQueue.__init__syT % E FE *F!33 -  I}}o-O'-rdVm6O -'%(13r)NNNpriority_queuerrs@rrrs $ 8383rcfeZdZdZd dZedZedZd dZ d dZ ddZ d d Z d d Z y)BarrierzARepresents a key-value map that persists across graph executions.Nct||_|t||j}|Dcgc]}tj|c}|_t |j D])\}}|jdk(std|d|n1|jDcgc]}tjc}|_tj|j|j |||_ tjr$tjj|_y|jj"j$j'dd|_ycc}wcc}w)a Creates a barrier that persists across different graph executions. A barrier represents a key-value map, where each key is a string, and each value is a tuple of tensors. At runtime, the barrier contains 'complete' and 'incomplete' elements. A complete element has defined tensors for all components of its value tuple, and may be accessed using take_many. An incomplete element has some undefined components in its value tuple, and may be updated using insert_many. The barrier call `take_many` outputs values in a particular order. First, it only outputs completed values. Second, the order in which completed values are returned matches the order in which their very first component was inserted into the barrier. So, for example, for this sequence of insertions and removals: barrier = Barrier((tf.string, tf.int32), shapes=((), ())) barrier.insert_many(0, keys=["k1", "k2"], values=["a", "b"]).run() barrier.insert_many(1, keys=["k1"], values=[1]).run() barrier.insert_many(0, keys=["k3"], values=["c"]).run() barrier.insert_many(1, keys=["k3"], values=[3]).run() barrier.insert_many(1, keys=["k2"], values=[2]).run() (indices, keys, values) = barrier.take_many(2) (indices_val, keys_val, values0_val, values1_val) = session.run([indices, keys, values[0], values[1]]) The output will be (up to permutation of "k1" and "k2"): indices_val == (-2**63, -2**63) keys_val == ("k1", "k2") values0_val == ("a", "b") values1_val == (1, 2) Note the key "k2" was inserted into the barrier before "k3". Even though "k3" was completed first, both are complete by the time take_many is called. As a result, "k2" is prioritized and "k1" and "k2" are returned first. "k3" remains in the barrier until the next execution of `take_many`. Since "k1" and "k2" had their first insertions into the barrier together, their indices are the same (-2**63). The index of "k3" will be -2**63 + 1, because it was the next new inserted key. Args: types: A single dtype or a tuple of dtypes, corresponding to the dtypes of the tensor elements that comprise a value in this barrier. shapes: Optional. Constraints on the shapes of tensors in the values: a single tensor shape tuple; a tuple of tensor shape tuples for each barrier-element tuple component; or None if the shape should not be constrained. shared_name: Optional. If non-empty, this barrier will be shared under the given name across multiple sessions. name: Optional name for the barrier op. Raises: ValueError: If one of the `shapes` indicate no elements. Nrz5Empty tensors are not supported, but received shape 'z ' at index )rr4rrZrNrO)r_typesr7r r/rQr num_elementsr.r>rbarrier _barrier_refrrxrUrVrYrZr[) r\rr4rrZr^rr$r_s rr`zBarrier.__init__'s3t &DK fdkk2f;ABal..q1Bdl -<(!U    1 $%%*G;qc;< << =AKKHql002Hdl)11 ||  D   "??$//dj$$'',,2237;dj!C Is E&!E+c|jS)z%Get the underlying barrier reference.)r#rvs r barrier_refzBarrier.barrier_refws   rctjr |jS|jjj S)z#The name of the underlying barrier.)rrxrVr#rYrZrvs rrZz Barrier.name|s2  " ZZ      $ $$rcl|d|jz}tj|j||||S)aYFor each key, assigns the respective value to the specified component. This operation updates each element at component_index. Args: component_index: The component of the value that is being assigned. keys: A vector of keys, with length n. values: An any-dimensional tensor of values, which are associated with the respective keys. The first dimension must have length n. name: Optional name for the op. Returns: The operation that performs the insertion. Raises: InvalidArgumentsError: If inserting keys and values without elements. z%s_BarrierInsertManyr)rVrbarrier_insert_manyr#)r\component_indexrrrZs r insert_manyzBarrier.insert_manys=" | #djj 0d  0 0 4t EErc|d|jz}tj|j||j|||}t j s |dj}|rd}n5tjtj|jd}|jdjtj|g|jdjtj|gt!|jdd|j"D]9\}} |jtj|gj%| ;|S)aTakes the given number of completed elements from this barrier. This operation concatenates completed-element component tensors along the 0th dimension to make a single component tensor. If barrier has no completed elements, this operation will block until there are 'num_elements' elements to take. TODO(b/25743580): the semantics of `allow_small_batch` are experimental and may be extended to other cases in the future. TODO(ebrevdo): If a take_many(allow_small_batch=True) is blocking already when the barrier is closed, it will block for ever. Fix this by using asynchronous operations. Args: num_elements: The number of elements to take. allow_small_batch: If the barrier is closed, don't block if there are less completed elements than requested, but instead return all available completed elements. timeout: This specifies the number of milliseconds to block before returning with DEADLINE_EXCEEDED. (This option is not supported yet.) name: A name for the operation (optional). Returns: A tuple of (index, key, value_list). "index" is a int64 tensor of length num_elements containing the index of the insert_many call for which the very first component of the given element was inserted into the Barrier, starting with the value -2**63. Note, this value is different from the index of the insert_many call for which the element was completed. "key" is a string tensor of length num_elements containing the keys. "value_list" is a tuple of tensors, each one with size num_elements in the 0th dimension for each component in the barrier's values. Nz%s_BarrierTakeManyrrrh)rVrbarrier_take_manyr#r rrxrYr rr rroutputsrr/r?rQr) r\r!allow_small_batchtimeoutrZrrYrrr$s r take_manyzBarrier.take_manys.T | !DJJ .d  - -     C  $ $ & q699b   **  & &ryy| 46 jjml66 {CDjjml66 {CDrzz!"~t||<F-&%  $ $i[ 1 = =e D FF Jrch|d|jz}tj|j||S)azCloses this barrier. This operation signals that no more new key values will be inserted in the given barrier. Subsequent InsertMany operations with new keys will fail. InsertMany operations that just complement already existing keys with other components, will continue to succeed. Subsequent TakeMany operations will continue to succeed if sufficient elements remain in the barrier. Subsequent TakeMany operations that would block will fail immediately. If `cancel_pending_enqueues` is `True`, all pending requests to the underlying queue will also be canceled, and completing of already started values is also not acceptable anymore. Args: cancel_pending_enqueues: (Optional.) A boolean, defaulting to `False` (described above). name: Optional name for the op. Returns: The operation that closes the barrier. z%s_BarrierCloser)rVr barrier_closer#rs rrz Barrier.closes9, |  +d  * *  7  rcf|d|jz}tj|j|S)zCompute the number of complete elements in the given barrier. Args: name: A name for the operation (optional). Returns: A single-element tensor containing the number of complete elements in the given barrier. z%s_BarrierReadySizer)rVrbarrier_ready_sizer#rs r ready_sizezBarrier.ready_sizes0 | "TZZ /d  / /0A0A MMrcf|d|jz}tj|j|S)zCompute the number of incomplete elements in the given barrier. Args: name: A name for the operation (optional). Returns: A single-element tensor containing the number of incomplete elements in the given barrier. z%s_BarrierIncompleteSizer)rVrbarrier_incomplete_sizer#rs rincomplete_sizezBarrier.incomplete_size s5 | '$** 4d  4 4  &&r)NNr"r)FNNr) rrrrr`rr%rZr*r1rr6r9rrrrr$s\IN<`   % % E0#( CJ: N &rrConditionalAccumulatorBasecVeZdZdZdZedZedZedZd dZ d dZ y) r:8A conditional accumulator for aggregating gradients. Up-to-date gradients (i.e., time step at which gradient was computed is equal to the accumulator's time step) are added to the accumulator. Extraction of the average gradient is blocked until the required number of gradients has been accumulated. cj||_|tj||_ntj|_||_t jr$t j j|_ y|j jjjdd|_ y)zCreates a new ConditionalAccumulator. Args: dtype: Datatype of the accumulated gradients. shape: Shape of the accumulated gradients. accumulator_ref: A handle to the conditional accumulator, created by sub- classes NrNrO) _dtyper r/_shaper>_accumulator_refrrxrUrVrYrZr[)r\rr$accumulator_refs rr`z#ConditionalAccumulatorBase.__init__$sDK  ,,U3dk ..0dk+D  "??$//dj((++0066s;B?djrc|jS)z%The underlying accumulator reference.)r@rvs rrAz*ConditionalAccumulatorBase.accumulator_ref8s   rc|jS)z'The name of the underlying accumulator.rVrvs rrZzConditionalAccumulatorBase.name= ::rc|jS)z>The datatype of the gradients accumulated by this accumulator.)r>rvs rrz ConditionalAccumulatorBase.dtypeBrrNcf|d|jz}tj|j|SzNumber of gradients that have currently been aggregated in accumulator. Args: name: Optional name for the operation. Returns: Number of accumulated gradients currently in accumulator. z%s_NumAccumulatedr)rVr$resource_accumulator_num_accumulatedr@rs rnum_accumulatedz*ConditionalAccumulatorBase.num_accumulatedGs5 | 4:: -d  A A D **rctj|jtjt j |tj|Sa|Sets the global time step of the accumulator. The operation logs a warning if we attempt to set to a time step that is lower than the accumulator's own time step. Args: new_global_step: Value of new time step. Can be a variable or a constant name: Optional name for the operation. Returns: Operation that sets the accumulator's time step. r) r$resource_accumulator_set_global_stepr@rcastrrrPrr\new_global_steprZs rset_global_stepz*ConditionalAccumulatorBase.set_global_stepVs@  A A  c++Orrr?rrNrPrr#resource_accumulator_apply_gradientr@r\gradr]rZs r apply_gradz!ConditionalAccumulator.apply_gradsv"  t{{ 3DNN..t{{;s44Z@'--PJ  @ @ *t$ PPrctj|j||j|}|j |j |S)aAttempts to extract the average gradient from the accumulator. The operation blocks until sufficient number of gradients have been successfully applied to the accumulator. Once successful, the following actions are also triggered: - Counter of accumulated gradients is reset to 0. - Aggregated gradient is reset to 0 tensor. - Accumulator's internal time step is incremented by 1. Args: num_required: Number of gradients that needs to have been aggregated name: Optional name for the operation Returns: A tensor holding the value of the average gradient. Raises: InvalidArgumentError: If num_required < 1 r)r"resource_accumulator_take_gradientr@r>rr?)r\ num_requiredrZouts r take_gradz ConditionalAccumulator.take_grads=,  > > |4;;T KCMM$++ Jr)NNconditional_accumulatorMEANrNr)rrrrr`rbrgrrs@rrRrRis'-$ P8P0rz#sparse.SparseConditionalAccumulatorSparseConditionalAccumulatorcbeZdZdZ d fd Zd dZ d dZd dZd dZd dZ d dZ xZ S) rkaA conditional accumulator for aggregating sparse gradients. Sparse gradients are represented by `IndexedSlices`. Up-to-date gradients (i.e., time step at which gradient was computed is equal to the accumulator's time step) are added to the accumulator. Extraction of the average gradient is blocked until the required number of gradients has been accumulated. Args: dtype: Datatype of the accumulated gradients. shape: Shape of the accumulated gradients. shared_name: Optional. If non-empty, this accumulator will be shared under the given name across multiple sessions. name: Optional name for the accumulator. reduction_type: Reduction type to use when taking the gradient. cdtj|||||}tt||||y)NrU)rsparse_conditional_accumulatorrrkr`r[s rr`z%SparseConditionalAccumulator.__init__s> (FF % 'O  &6ue7FHrcj|j|j|j|j||S)aAttempts to apply a gradient to the accumulator. The attempt is silently dropped if the gradient is stale, i.e., `local_step` is less than the accumulator's global time step. Args: grad: The gradient `IndexedSlices` to be applied. local_step: Time step at which the gradient was computed. name: Optional name for the operation. Returns: The operation that (conditionally) applies a gradient to the accumulator. Raises: InvalidArgumentError: If grad is of the wrong shape ) grad_indices grad_values grad_shaper]rZ)rbindicesr dense_shaper`s rapply_indexed_slices_gradz6SparseConditionalAccumulator.apply_indexed_slices_grads7" ??\\KK##   rc Ptjtj|tj }t j|j|tj|tj |tj|gn|tj |du|S)aAttempts to apply a sparse gradient to the accumulator. The attempt is silently dropped if the gradient is stale, i.e., `local_step` is less than the accumulator's global time step. A sparse gradient is represented by its indices, values and possibly empty or None shape. Indices must be a vector representing the locations of non-zero entries in the tensor. Values are the non-zero slices of the gradient, and must have the same first dimension as indices, i.e., the nnz represented by indices and values must be consistent. Shape, if not empty or None, must be consistent with the accumulator's shape (if also provided). Example: A tensor [[0, 0], [0, 1], [2, 3]] can be represented indices: [1,2] values: [[0,1],[2,3]] shape: [3, 2] Args: grad_indices: Indices of the sparse gradient to be applied. grad_values: Values of the sparse gradient to be applied. grad_shape: Shape of the sparse gradient to be applied. local_step: Time step at which the gradient was computed. name: Optional name for the operation. Returns: The operation that (conditionally) applies a gradient to the accumulator. Raises: InvalidArgumentError: If grad is of the wrong shape N)r]gradient_indicesgradient_valuesgradient_shapehas_known_shaperZ) rrNrrrPrr!sparse_accumulator_apply_gradientr@)r\rprqrrr]rZs rrbz'SparseConditionalAccumulator.apply_gradsJs44Z@'--PJ  > > !|W]]C#}}$B*gmmE#4/  rc\tj|j||j|S)aAttempts to extract the average gradient from the accumulator. The operation blocks until sufficient number of gradients have been successfully applied to the accumulator. Once successful, the following actions are also triggered: - Counter of accumulated gradients is reset to 0. - Aggregated gradient is reset to 0 tensor. - Accumulator's internal time step is incremented by 1. Args: num_required: Number of gradients that needs to have been aggregated name: Optional name for the operation Returns: A tuple of indices, values, and shape representing the average gradient. Raises: InvalidArgumentError: If `num_required` < 1 r)r sparse_accumulator_take_gradientr@r>)r\rerZs rrgz&SparseConditionalAccumulator.take_grad1s,*  = = |4;;T KKrctj|j||j|}t j |j |j|jS)aAttempts to extract the average gradient from the accumulator. The operation blocks until sufficient number of gradients have been successfully applied to the accumulator. Once successful, the following actions are also triggered: - Counter of accumulated gradients is reset to 0. - Aggregated gradient is reset to 0 tensor. - Accumulator's internal time step is incremented by 1. Args: num_required: Number of gradients that needs to have been aggregated name: Optional name for the operation Returns: An `IndexedSlices` holding the value of the average gradient. Raises: InvalidArgumentError: If `num_required` < 1 r)rsrrt) rr}r@r>r IndexedSlicesrsrr$)r\rerZ return_vals rtake_indexed_slices_gradz5SparseConditionalAccumulator.take_indexed_slices_gradIsV*#CC |4;;TKJ  ' '""  $$ &&rcf|d|jz}tj|j|SrH)rVraccumulator_num_accumulatedr@rs rrJz,SparseConditionalAccumulator.num_accumulatedfs5 | 4:: -d  8 8 D **rctj|jtjt j |tj|SrL) raccumulator_set_global_stepr@rrNrrrPrrOs rrQz,SparseConditionalAccumulator.set_global_stepus@  8 8  c++OrR _capacity _memory_limitrrno_op _coloc_op) r\rr4r:rr memory_limitr^r_s rr`zBaseStagingArea.__init__sD   ! - -dnn.E.E F j K %dj ;K=I JJDL  VF #OPP;ABal..q1Bdl<@LLIql002Idl  Us6{ "NOOdkdkDN%D  DJJ. /0'--/dn00CI00sE1E6E;;Fc|jS)zThe name of the staging area.rDrvs rrZzBaseStagingArea.namerErc|jS)z@The list of dtypes for each component of a staging area element.rzrvs rrzBaseStagingArea.dtypesr{rc|jS)z@The list of shapes for each component of a staging area element.r}rvs rr4zBaseStagingArea.shapesr{rc|jS)z?The list of names for each component of a staging area element.rrvs rr:zBaseStagingArea.namesrrc|jS)z4The maximum number of elements of this staging area.)rrvs rrzBaseStagingArea.capacitys >>rc|jS)z1The maximum number of bytes of this staging area.)rrvs rrzBaseStagingArea.memory_limits   rc t|tr|js tdt |j j |js;tdt|j dt|jtt|jDcgc]\}}||vr||||fc}}\}}}nz|jr td| tdt|t|k7r#tdt|dt|t|ttfs|g}dg}t|t|jks-td t|d t|jg}t||D]\}}|j||j|} }|j|k7r-td t!|jd t!||j#j%| |j't)j*||d |z||fScc}}w)aValidate and convert `vals` to a list of `Tensor`s. The `vals` argument can be a Tensor, a list or tuple of tensors, or a dictionary with tensor values. If `vals` is a list, then the appropriate indices associated with the values must be provided. If it is a dictionary, the staging area must have been constructed with a `names` attribute and the dictionary keys must match the staging area names. `indices` will be inferred from the dictionary keys. If the staging area was constructed with a `names` attribute, `vals` must be a dictionary. Checks that the dtype and shape of each value matches that of the staging area. Args: vals: A tensor, a list or tuple of tensors, or a dictionary. Returns: A (tensors, indices) tuple where `tensors` is a list of `Tensor` objects and `indices` is a list of indices associated with the tensors. Raises: ValueError: If `vals` or `indices` is invalid. z5Staging areas must have names to enqueue a dictionaryzJKeys in dictionary to put do not match names of staging area. Dictionary: zQueue: z:You must enqueue a dictionary in a staging area with namesz9Indices must be supplied when inserting a list of tensorszNumber of indices z doesn't match number of values rzUnexpected number of inputs z vs z+Datatypes do not match. Received val.dtype z and dtype rr)rrrRr.setrissubsetrr?rr9rrrPrQrrrrrrr) r\rrsrrr_rrrr$s r_check_put_dtypesz!BaseStagingArea._check_put_dtypess`8$ [[ CE E   & &t{{ 399? 9L8M"6$++#6"79: : +4T[[+A-#'1a"#t)!%QA-.dGQ &' ' &' ' WT "-c'l^<--0YK9: :tUm ,v# t9DLL) ) 5c$i[dll+,. //GdG$ LQ\\!_dll1oUe e //2399~.>?""%e*/0 0 mmo//6 nn  5~7I JL L G Q-s2I ct|ttfs|g}tjj }||j j k7r"|Dcgc]}tj|}}|Scc}w)zjEncode inter-device transfers if the current device is not the same as the Staging Area's device. ) rrrrrrrr identity)r\rcurr_device_scopets r_create_device_transfersz(BaseStagingArea._create_device_transfers!sh gt} - g(..077DNN1110781##A&8g8 N9sA5c |j|}t||D]#\}}|j|j|%|jr-t||Dcic]\}}|j||c}}S|Scc}}w)aReturn the value to return from a get op. If the staging area has names, return a dictionary with the names as keys. Otherwise return either a single tensor or a list of tensors depending on the length of `tensors`. Args: tensors: List of tensors from the get op. indices: Indices of associated names and shapes Returns: A single tensor, a list of tensors, or a dictionary of tensors. )rr?rrQrR)r\rrsrrrs r_get_return_valuez!BaseStagingArea._get_return_value0s ++G4G'*(  t||A'( {{-0,A BDAqdkk!na BB NCsA?cxt|ttfr|St|tr|j S|gSrrrs rrzBaseStagingArea._scope_valsLrrNNNrrr)rrrr _identifier threadingLock_lockr`rrZrr4r:rrrrrrrrrrrs%+ ).. % $0L            N` 8rrc\eZdZdZ d fd Zd dZdZd dZd dZd dZ d dZ xZ S) StagingAreaaClass for staging inputs. No ordering guarantees. A `StagingArea` is a TensorFlow data structure that stores tensors across multiple steps, and exposes operations that can put and get tensors. Each `StagingArea` element is a tuple of one or more tensors, where each tuple component has a static dtype, and may have a static shape. The capacity of a `StagingArea` may be bounded or unbounded. It supports multiple concurrent producers and consumers; and provides exactly-once delivery. Each element of a `StagingArea` is a fixed-length tuple of tensors whose dtypes are described by `dtypes`, and whose shapes are optionally described by the `shapes` argument. If the `shapes` argument is specified, each component of a staging area element must have the respective fixed shape. If it is unspecified, different elements may have different shapes, It can be configured with a capacity in which case put(values) will block until space becomes available. Similarly, it can be configured with a memory limit which will block put(values) until space is available. This is mostly useful for limiting the number of tensors on devices such as GPUs. All get() and peek() commands block if the requested data is not present in the Staging Area. c6tt| ||||||y)aConstructs a staging area object. The two optional lists, `shapes` and `names`, must be of the same length as `dtypes` if provided. The values at a given index `i` indicate the shape and name to use for the corresponding queue component in `dtypes`. The device scope at the time of object creation determines where the storage for the `StagingArea` will reside. Calls to `put` will incur a copy to this memory space, if necessary. Tensors returned by `get` will be placed according to the device scope when `get` is called. Args: dtypes: A list of types. The length of dtypes must equal the number of tensors in each element. shapes: (Optional.) Constraints on the shapes of tensors in an element. A list of shape tuples or None. This list is the same length as dtypes. If the shape of any tensors in the element are constrained, all must be; shapes can be None if the shapes should not be constrained. names: (Optional.) If provided, the `get()` and `put()` methods will use dictionaries with these names as keys. Must be None or a list or tuple of the same length as `dtypes`. shared_name: (Optional.) A name to be used for the shared object. By passing the same name to two different python objects they will share the underlying staging area. Must be a string. capacity: (Optional.) Maximum number of elements. An integer. If zero, the Staging Area is unbounded memory_limit: (Optional.) Maximum number of bytes of all tensors in the Staging Area. An integer. If zero, the Staging Area is unbounded Raises: ValueError: If one of the arguments is invalid. N)rrr`)r\rr4r:rrrrs rr`zStagingArea.__init__s"R +t%ffe[&. >rc tj|d|jz|j|5}t |t t tfs|g}t tt|}|j||\}}tj|j5tj||j||j|j }dddcdddS#1swYxYw#1swYyxYw)a4Create an op that places a value into the staging area. This operation will block if the `StagingArea` has reached its capacity. Args: values: A single tensor, a list or tuple of tensors, or a dictionary with tensor values. The number of elements must match the length of the list provided to the dtypes argument when creating the StagingArea. name: A name for the operation (optional). Returns: The created op. Raises: ValueError: If the number or type of inputs don't match the staging area. %s_put)rrrZrrN)rrrVrrrrrranger9r colocate_withrrstagerr)r\rrZrrsrr_rYs rputzStagingArea.puts$ h3((0 25: ud 3 4U3v;'(g&&vw7gdA   T^^ ,-  $ $ ^^++ --%--s$A0C<$9C0 C<0C9 5C<<Dctj|j5|}dddtt t |j }|j|S#1swYBxYwr)rrrrrr9rPr)r\get_fnrZrrss r__internal_getzStagingArea.__internal_gets]  4>> * Hc5T\\*+,G  ! !#w // s A))A2cXdjzfd}j|S)aFGets one element from this staging area. If the staging area is empty when this operation executes, it will block until there is an element to dequeue. Note that unlike others ops that can block, like the queue Dequeue operations, this can stop other work from happening. To avoid this, the intended use is for this to be called only when there will be an element already available. One method for doing this in a training loop would be to run a `put()` call during a warmup session.run call, and then call both `get()` and `put()` in each subsequent step. The placement of the returned tensor will be determined by the current device scope when this function is called. Args: name: A name for the operation (optional). Returns: The tuple of tensors that was gotten. %s_getctjjjjj SNrrrZrr)runstagerPrVrr)rZr\srz!StagingArea.get..s2"**$,, $ !^^!%!3!35rrV_StagingArea__internal_get)r\rZfns`` rgetzStagingArea.gets4, |  "d 5B   r4 ((rc\djzfd}j|S)aPeeks at an element in the staging area. If the staging area is too small to contain the element at the specified index, it will block until enough elements are inserted to complete the operation. The placement of the returned tensor will be determined by the current device scope when this function is called. Args: index: The index of the tensor within the staging area to look up. name: A name for the operation (optional). Returns: The tuple of tensors that was gotten. z%s_peekctjjjjj Sr)r stage_peekrPrVrr)rnrZr\srrz"StagingArea.peek..s4"--e<X$L0 )D)<)&)rrcxeZdZdZ d fd Zd dZddZd dZddZd dZ d dZ dd Z dd Z dd Z xZS)MapStagingAreaa A `MapStagingArea` is a TensorFlow data structure that stores tensors across multiple steps, and exposes operations that can put and get tensors. Each `MapStagingArea` element is a (key, value) pair. Only int64 keys are supported, other types should be hashed to produce a key. Values are a tuple of one or more tensors. Each tuple component has a static dtype, and may have a static shape. The capacity of a `MapStagingArea` may be bounded or unbounded. It supports multiple concurrent producers and consumers; and provides exactly-once delivery. Each value tuple of a `MapStagingArea` is a fixed-length tuple of tensors whose dtypes are described by `dtypes`, and whose shapes are optionally described by the `shapes` argument. If the `shapes` argument is specified, each component of a staging area element must have the respective fixed shape. If it is unspecified, different elements may have different shapes, It behaves like an associative container with support for: - put(key, values) - peek(key) like dict.get(key) - get(key) like dict.pop(key) - get(key=None) like dict.popitem() - size() - clear() If ordered a tree structure ordered by key will be used and get(key=None) will remove (key, value) pairs in increasing key order. Otherwise a hashtable It can be configured with a capacity in which case put(key, values) will block until space becomes available. Similarly, it can be configured with a memory limit which will block put(key, values) until space is available. This is mostly useful for limiting the number of tensors on devices such as GPUs. All get() and peek() commands block if the requested (key, value) pair is not present in the staging area. Partial puts are supported and will be placed in an incomplete map until such time as all values associated with the key have been inserted. Once completed, this (key, value) pair will be inserted into the map. Data in the incomplete map counts towards the memory limit, but not towards capacity limit. Partial gets from the map are also supported. This removes the partially requested tensors from the entry, but the entry is only removed from the map once all tensors associated with it are removed. ctt| ||||||||_|rtj |_tj|_tj|_ tj|_ tj|_tj|_tj"|_ytj&|_tj(|_tj*|_ tj,|_ tj.|_tj0|_tj2|_y)a-Args: dtypes: A list of types. The length of dtypes must equal the number of tensors in each element. capacity: (Optional.) Maximum number of elements. An integer. If zero, the Staging Area is unbounded memory_limit: (Optional.) Maximum number of bytes of all tensors in the Staging Area (excluding keys). An integer. If zero, the Staging Area is unbounded ordered: (Optional.) If True the underlying data structure is a tree ordered on key. Otherwise assume a hashtable. shapes: (Optional.) Constraints on the shapes of tensors in an element. A list of shape tuples or None. This list is the same length as dtypes. If the shape of any tensors in the element are constrained, all must be; shapes can be None if the shapes should not be constrained. names: (Optional.) If provided, the `get()` and `put()` methods will use dictionaries with these names as keys. Must be None or a list or tuple of the same length as `dtypes`. shared_name: (Optional.) A name to be used for the shared object. By passing the same name to two different python objects they will share the underlying staging area. Must be a string. Raises: ValueError: If one of the arguments is invalid. N)rrr`_orderedrordered_map_stage_put_fnordered_map_unstage_pop_fnordered_map_unstage_no_key _popitem_fnordered_map_peek_peek_fnordered_map_size_size_fnordered_map_incomplete_size_incomplete_size_fnordered_map_clear _clear_fn map_stage map_unstagemap_unstage_no_keymap_peekmap_sizemap_incomplete_size map_clear) r\rr4r:rorderedrrrs rr`zMapStagingArea.__init__{sF .$( )1<ADM&88dl&::dl*EEd'88dm'88dm!2!N!Nd(::dn&00dl&22dl*==d'00dm'00dm!2!F!Fd(22dnrc tj|d|jz|j|5}|j ||\}}tj |j 5|j||||j|j||j|j}ddddddS#1swYxYw#1swYSxYw)aCreate an op that stores the (key, vals) pair in the staging area. Incomplete puts are possible, preferably using a dictionary for vals as the appropriate dtypes and shapes can be inferred from the value names dictionary key values. If vals is a list or tuple, indices must also be specified so that the op knows at which element position to perform the insert. This operation will block if the capacity or memory limit of this container is reached. Args: key: Key associated with the data vals: Tensor (or a dict/tuple of Tensors) to place into the staging area. indices: (Optional) if vals is a tuple/list, this is required. name: A name for the operation (optional) Returns: The created op Raises: ValueError: If the number or type of inputs don't match the staging area. rrN) rrrVrrrrrrPrr)r\rrrsrZrrYs rrzMapStagingArea.puts6 h3((. 0-38,,T7;mdG   T^^ , - \\   << ^^++- - - I - - - Is$5C )AB=+C =C C  Ccp|'ttt|j}t |t tfst dt|t|dk(r tdtd|DrE|jtd|d |Dcgc]}|jj|}}n"td|Drnt d |d |Dcgc]}|j|}}||fScc}w#t$rtd|jwxYwcc}w) NzInvalid indices type rz Empty indicesc3<K|]}t|tywr)rrr#rs rr%z9MapStagingArea._get_indices_and_dtypes..s /!:a  /rdzString indices provided z3, but this Staging Area was not created with names.z&Named index not in Staging Area names c3<K|]}t|tywrr rs rr%z9MapStagingArea._get_indices_and_dtypes..s 1AZ3  1rdzMixed types in indices z. May only be str or int) rrr9rPrrr0r1r.r2rRrn)r\rsrrrs r_get_indices_and_dtypesz&MapStagingArea._get_indices_and_dtypessLU3t||,-.g gt} - -d7m_= >> 7|q  '' /w //  3G9=IIJ J>189A4;;$$Q'99 1 1 1 /y9// 00(/ /!dll1o /F / F?: >//3{{m=> >>0s$D""D D-D3 D"D0c D|d|jz}|j|\}}tj|j5|j ||j||||j |j}ddd|j|S#1swYxYw)a Peeks at staging area data associated with the key. If the key is not in the staging area, it will block until the associated (key, value) is inserted. Args: key: Key associated with the required data indices: Partial list of tensors to retrieve (optional). A list of integer or string indices. String indices are only valid if the Staging Area has names associated with it. name: A name for the operation (optional) Returns: The created op Nz%s_poprrsrrZrr) rVrrrrrrrrr\rrsrZrresults rrzMapStagingArea.peeks$ |  "d227;OGV  4>> *+}} jj>>))+f+  ! !&' 22++s 7BBcT||j||S|j|||S)a~If the key is provided, the associated (key, value) is returned from the staging area. If the key is not in the staging area, this method will block until the associated (key, value) is inserted. If no key is provided and the staging area is ordered, the (key, value) with the smallest key will be returned. Otherwise, a random (key, value) will be returned. If the staging area is empty when this operation executes, it will block until there is an element to dequeue. Args: key: Key associated with the required data (Optional) indices: Partial list of tensors to retrieve (optional). A list of integer or string indices. String indices are only valid if the Staging Area has names associated with it. name: A name for the operation (optional) Returns: The created op )rsrZ)_popitem_pop)r\rrsrZs rrzMapStagingArea.get" s1. { ]]7] 66 YYsG$Y 77rc H|d|jz}|j|\}}tj|j5|j ||j||||j |j}ddd||j|fS#1swYxYw)aERemove and return the associated (key, value) is returned from the staging area. If the key is not in the staging area, this method will block until the associated (key, value) is inserted. Args: key: Key associated with the required data indices: Partial list of tensors to retrieve (optional). A list of integer or string indices. String indices are only valid if the Staging Area has names associated with it. name: A name for the operation (optional) Returns: The created op Nrr) rVrrrrrrrrrs rrzMapStagingArea._pop> s  |  "d227;OGV  4>> *+|| jj>>))+f+ &&vw7 77++s 7BB!c x|d|jz}|j|\}}tj|j5|j |j||||j |j\}}ddd|jd}|j|}||fS#1swY3xYw)aIf the staging area is ordered, the (key, value) with the smallest key will be returned. Otherwise, a random (key, value) will be returned. If the staging area is empty when this operation executes, it will block until there is an element to dequeue. Args: key: Key associated with the required data indices: Partial list of tensors to retrieve (optional). A list of integer or string indices. String indices are only valid if the Staging Area has names associated with it. name: A name for the operation (optional) Returns: The created op Nz %s_get_nokeyrr) rVrrrrrrrrr)r\rsrZrrrs rrzMapStagingArea._popitem_ s$ | djj (d227;OGV  4>> *+$$jj>>)) %+kc6+  ' ' ,Q /C  # #FG 4F ;++s 9B00B9c|d|jz}|j|j||j|j|jS)rrrrZrrr)rVrrPrrrs rrzMapStagingArea.size sN |  #d ==JJ ||''  ))rc|d|jz}|j|j||j|j|jS)zReturns the number of incomplete elements in the staging area. Args: name: A name for the operation (optional) Returns: The created op z%s_incomplete_sizer)rVrrPrrrs rr9zMapStagingArea.incomplete_size sP | !DJJ .d  # #JJ ||'' $ ))rc|d|jz}|j|j||j|j|jS)rrr)rVrrPrrrs rrzMapStagingArea.clear sN | $** $d >>JJ ||''  ))r)NNNFrr)NNr)NNN)rrrrr`rrrrrrrr9rrrs@rrr?sT9z83t*X>CCD#/d$rc tjjtj|j}t j |j |j|j|j|j|j||j}|j|Stj|j5t!|jDcgc]}g}}t#j$||jd}t'|D]<\}}||jz}||j)t#j*|g>|cdddScc}w#1swYyxYw)zAdds a node that yields a group of records every time it is executed. If RecordInput `batches` parameter is not None, it yields a list of record batches with the specified `batch_size`. )rfile_buffer_sizefile_parallelismfile_shuffle_shift_ratiorfile_random_seedr rZNr)r TFRecordOptionsget_compression_type_stringrr record_inputrrrrrrrVrrrrr r[rrreshape)r\r recordsr_ batch_listrnprotobuf batch_indexs r get_yield_opzRecordInput.get_yield_op s< !00LL!!$"8"89;,,''****!%!2!2##) ZZG }} n >>$** %"' "67Qb7 7//'4+;+;Q?(1 JOE8 -+ [ ! ( ():):8R)H I J 7s E+% E&.A.E+&E++E4)rhrhrhrrNNN)rrrrr`rrrrrr s,  $-0^rr)FF)2rrjrrtensorflow.python.eagerrtensorflow.python.frameworkrrPrrrr r tensorflow.python.lib.ior tensorflow.python.opsr r rrrr'tensorflow.python.ops.gen_data_flow_opstensorflow.python.utilrtensorflow.python.util.compatr tensorflow.python.util.tf_exportrrr7r;rFdeprecated_endpointsrIrrrrr rrr:rRrkrrrrrrrr$s+96+343.+123*76.96 (-(-B  % >@!!!>;"?@lFlFA@lF\  78"!!235QOQO5 8 QOh "3[!AB!!!+.:F :F/C:F@G%YG%T LN"!!#8:L"MNEMyEMONEMP JL!!!#5"GH?3I?3IL?3Jr&r&j +,-LL.L^ '()W7W*Wt -/MNP#=PDRRj_)/_)D~)_~)B VVr