AVhdZddlZddlmZddlmZddlmZddl m Z ddl m Z ddl m Z dd l mZdd l mZdd l mZdd l mZdd l mZddl mZddl mZddlmZddlmZddlmZddlmZddlmZddlmZddlm Z ddlm!Z!ddlm"Z"ddlm#Z#ddlm$Z$ddlm%Z%ddlm&Z&ddl'm(Z(ddl'm)Z)ddl'm*Z*dd l'm+Z+ dAd!Z,ejZd"ejZd#d$ej\fd%Z/d&Z0d'Z1d(Z2d)Z3d*Z4d+Z5d,Z6d-Z7d.Z8d/Z9d0Z:ejvd1gd2Z<Gd3d4ejzZ>d5Z?d6Z@d7ZAd8ZBd9ZCd:ZDd;ZEd<ZFd=ZGd>ZHd?ZId$ej\fd@ZJy)Bawhile_v2 and gradient. This is a version of while_loop that emits a single While op, as well as the gradient function for While ops produced by while_loop. This will eventually replace the current tf.while_loop implementation once it reaches feature and performance parity. N)attr_value_pb2)pywrap_tf_session) backprop_util)auto_control_deps_utils) constant_op)dtypes func_graph)indexed_slices)opstensor) tensor_shape) tensor_spec) tensor_util) array_ops)control_flow_ops)control_flow_util)control_flow_util_v2)default_gradient)gen_functional_ops)gen_resource_variable_ops)gradients_util)handle_data_util)list_ops)math_ops)tensor_array_ops) while_v2_indexed_slices_rewriter)compat)nest)object_identity)variable_utilsc z &'()tj)tj)d't )} t tj tjd|&tj tj|(n$tj tj(tj dtj(d} |sd}tj|5} tjd5tj| d} tj| d} dddt}t!j"d |j$ndd }||gt'zt(j*j-|t(j*j-|gt'(z}tj.j0}'(fd }t3j4 |gi|tj6| tj.j8 | &&'()fd}t3j4 |gi|tj:| tj.j8 | }tj|j<Dcgc] }| c}d}|j>z|z|j@jC|jD|j@jC|jF&jI5t &j>}&j>|j>d|k(sJtK&|j>|d|zdddt tj)d}d}tM|j@|||z| tj||| zdt |j@}|rtjNrtQ|}|D]}tSjT|j$|jV}jY|&jI5&j[|ddd|jI5tSj\||}|j@jY|dddtjd}t_&|t |ta||tjbt'&jdjft'|jdjfz5|j@D cgc]} | jV}!} ti|||z}"| |!|"<tk|&||!|| |}#dddtj.jlstod#D}#ddd#|z}$|s"|$D cgc]} tqjr| }$} tu('|$}#|r|#Stj|#d}%t |%dk(r|%dS|#S#1swYxYwcc}w#1swYxYw#1swYxYw#1swYjxYwcc} w#1swYxYw#1swYxYwcc} w)z3Like tf.while_loop, except emits a single While op.Texpand_compositesNc|jSN)shape)specs N/home/dcms/DCMS/lib/python3.12/site-packages/tensorflow/python/ops/while_v2.pyzwhile_loop..as 4::whilecondbodyr loop_counterdtypenamec t|}tj|rA|jj|jjrt j |}|Stj||k|S)z@Extra `cond` wrapper that can handle the extra counter loop_var.) _pack_sequence_asr is_tf_typer(dimsr squeeze_v2r logical_and)r0maximum_iterations_argargspredr.flat_orig_loop_varsloop_vars_signaturemaximum_iterationss r* wrapped_condz while_loop..wrapped_cond~s} 02Et LNd   & ::?? "djjoo##D)  # ## 1 149 9r,) collections) signaturer add_control_dependenciescBttj dddtj|djD]%}t j j |'t |}tj|s|g} tj| ddt|}|dz|gt|zS#t$rFtj|}tj }tj||ddYlwxYw)a)Loop body augmented with counter update. Args: loop_counter: Loop counter which needs to be incremented in the body. maximum_iterations_arg: Maximum iterations of the loop. *args: List of args Returns: A list of tensors the same length as args. NTr$F) check_typesr%)_copy_handle_datar flattenexternal_capturesr get_default_graphcapturer5 is_nestedassert_same_structure ValueErrorr"convert_variables_to_tensors_tensor_array_to_flowlist) r0r:r;toutputsvars1vars2r/ cond_graphr= loop_varsr>orig_loop_varss r* wrapped_bodyz while_loop..wrapped_bodys Yqr]dK TTBD+++! ''*+ 02Et LNg ^^G $) ; ""7N59 ;&g.gQ 6 7$w- GG;;;GD;;NK ""5%U59 ; ;sCA DDrE) element_dtype element_shapemax_num_elements output_shapesparallel_iterationsr3num_original_outputsc3FK|]}tj|ywr')ridentity).0rSs r* zwhile_loop../s=i((+=!rG);r"rPr rIlenrQ map_structurer ,internal_convert_to_tensor_or_indexed_slicesr_shape_invariant_to_type_specr name_scopeutilunique_fn_name"_build_maximum_iterations_loop_varrconstantr2rRr TensorSpec from_tensorrK_add_control_dependenciesfunc_graph_modulefunc_graph_from_py_funcWhileCondFuncGraph _collectionsWhileBodyFuncGraphdeferred_external_capturesrJrTextendinternal_capturesdeferred_internal_captures as_default _duplicate_body_captures_in_cond_check_shapes_compatoutput_all_intermediates_get_intermediatesrempty_tensor_listr(appendrLtensor_list_push_back_check_num_inputs_outputs!_check_inputs_outputs_types_matchcontrol_dependenciesfunction_capturescontrolslice_build_while_opbuilding_functiontupler stop_gradientr5)*r.r/rXshape_invariantsr`r?r3return_same_structure back_proplen_orig_loop_varsflat_shape_invariantsscope cond_name body_namemaximum_iterations_loop_varr0func_graph_signaturerCr@rZ body_graphcrxnum_cond_capturesnum_flattened_outputsfirst_loop_var_indexraintermediate_tensorsintermediate_tensor tensor_listappended_tensor_listflattened_loop_varsrSr_orig_loop_vars_rangerToutput_loop_varsflattened_outputsrWr=r>rYs*``` ` @@@@r* while_loopr:s99)D). ^tL>* $I.)  AA)!,,66#%,,66 C,, ll&$?A  D ~~dH>u  5%%eV4i%%eV4i5#E#''  )*///3  L :;d9oMI    + +L 9    + +,G H J ! " #446PP9&#::  &** 3#8#8#:#G#GI!9;J-H-H^#::  &** 3#8#8#:#G#GI!9;J"& ;;<<"  J0003MMj::;jCCD    &j::;****+=,=>?@?& j223D3EF $%& &  ^?C!EF/0D011 2 Y34H1523FJ L Mz112T2240 ;!5: 00-33-33/1  %  " " $ *   [ ) * " " $ :"*!?!?."0     # #$8 9  : ::$,,yDIj*!"568%j2EF ! ! Z ) ) 1 12T  ( ( 0 062 2 35)3(:(:;1qww;m;"#7#7:O#OQ,Am()   %135g5  " 4 4=W==gQH>T12F2334 H>ZNsZ,-Y"E:Z,Y/ A5Z,AY4C*Z,>ZZ,(2ZBZ,/Z >Z(Z 98Z,Z8"Y, ' Z,4Y> 9Z,Z Z,Z Z,Z  Z) %Z,,Z5StatelessWhileWhileopc D |jdj}t|dd}t|dd}t|j}|jd}|j d} |j d}t|j|z } t |d ||jd ||jd ||jd |D cgc]\} } } } t| | | | c} } } } d g| zz}t|d d Gt|jt|jz }|jD] }||k\s d ||<t t |j|j|D cgc] \}}} | ||| fc} }}\}}}t|||||tj|j||\}}|jr|xjd z c_|xjd z c_|j }|j|d }|j#dtj$||j#dtj$|t|j&t|jt|zk7rDt)d t|j&d t|jdt|d|j+d|j&|j-d|j.|j1||j3|Dcgc]}|j4c}|Dcgc]}|j6c}t9||j|d |j;dt=j>t|jtA|||}||z}tCjD||||j}d}tj|j dj}tGjH|||itjJ|}tM||t|tO||||jDcgc]}|j6c}|d|jzt|j}|Dcgc]}tQjR|}}tU|||S#t|j}YxYwcc} } } } wcc} }}wcc}wcc}wcc}wcc}w)z2The gradient of a While op produced by while_loop.rr. _cond_graphr/ _body_graphrGr`_num_original_outputsNskip_input_indices _rewrittenzjInputs and outputs constructed for the forward op of a While gradient don't match with 'output_types' at z,'inputs' at length z, and 'new_inputs' at length z-. This doesn't make sense, please file a bug.Tr_ic ||kSr')counterunused_maximum_iterations_argforward_loop_iters unused_argss r* grad_condz_WhileGrad..grad_conds ' ''r,r z%s_gradr^)+rTr _get_graphrginputsget_attrzip_preprocess_gradgetattrrzr_create_grad_funcrlunique_grad_fn_namer3while_op_needs_rewrite extra_inputs_set_func_attrcreate_new_tf_function output_typesAssertionError_set_type_list_attr_set_shape_list_attrr__add_while_inputs _add_outputsr2r(rH _set_attrr AttrValue_resolve_grad_capturesrrewrite_grad_indexed_slicesrsrtrurrrrc_get_structured_grad_output) rgradswhile_oprWrorig_num_paramsr?r`ranum_intermediatesgradbody_outwhile_in while_outcaptures_start_indexryxysxsnon_none_gradsbody_grad_graphr; new_inputs new_outputsrScaptured_inputsrXrgrad_cond_namecond_grad_graphrTs r* _WhileGradrBsZZ]  ((FM:*(FM:* **+/yy| $9:1#,,-DE(**+.BB25 %% &   22 3 //// 0   00 1 23   -$(ItXx;  f   !% R%t,8 J  Z%A%A!BB  " " " "a#*++UC4!I!I,1a7;7G#$Q!IJ"b.,"nj* z/5GI/4++ OO|#OOO|#O!--J$$_%56K FD$?$? $KL FD$?$? $KL : " "#s8??';c*o'MM  : (( ) **>  ! ""? _ J  L MM    j&=&=> !!/:3K3KL z* K8q1778,78q1778:k8#3#3O4D#EF ,#--H4D4D0EFH+:+35/_$)/JJ _i:)(++BKK,?,D,DE.%==iB((8:/O_c)nM &5&=&=>QWW>- x}} $667 9'-4 4qY   " 4' 4 $We_ EEC1x//0 (!ID98>? 5s0*S"S? 5T 2T T 5T7T"S<cX jDcgc]}|js|}}jDcgc]}|js|} }|s| rtj ntj  fd} t j | |Scc}wcc}w)z.Builds the functional StatelessWhile/While op.c tj|tjtj \}}tj|tj |tj |t|g|jdtjtj_ tj_ |_|_|S)N)r_r`r3rr)rlget_op_and_outputsrrHrTmaybe_set_lowering_attr*maybe_propagate_compile_time_consts_in_xla#_set_read_only_resource_inputs_attrrrrr rK outer_graphrr) rrtensorsrrWr3raop_fnr_r`s r*_make_opz!_build_while_op.._make_ops// ##J/ ##J/#/  1Hgj(('2  *33H=':z2JK .%//2FGI !224J 224J%H%H Nr,)get_operations _is_statefulr_whilestateless_whilerl"run_as_function_for_tape_gradients) rXrWrr_r`r3rarcond_stateful_opsbody_stateful_opsrrs `````` @r*rrs,,. "//b,,. "//b,  % %E  . .E.  0 09 EEEsB"B"B'B'cg}td|jD}|jD]}|jdk(r|jdk(r#|jD]b}||j dus|j tjk7s3t|?|j|vsR|j|d|S)z?Returns all tensors in `func_graph` that should be accumulated.c3FK|]\}}|j|fywr'ref)rdkvs r*rez%_get_intermediates..sG41a1557A,GrfIdentity MutexLockr) dictcapturesrtyperTrr2rresource_get_accumulatorrr)r intermediatesreverse_capturesros r*rrs0-G:3F3FGG  % % '  b ww* ww+ ZZ  :$$Q' ' ''V__ $ 1  % %%') )Q    r,ct|sy|jtjtjfvr#t j |r| t||St|tjrtj|S|S)aReturns the initial gradient to be used for a given output tensor. Args: grad: the original gradient Tensor passed to the gradient function. body_graph_output: the corresponding Tensor in the body graph. while_op_input: the corresponding Tensor input of the While op. while_op_output: the corresponding Tensor output of the While op. Returns: A Tensor or None. N) _is_trainabler2rrvariantrsupports_default_grad _zeros_like isinstancer IndexedSlicesr convert_to_tensor)rbody_graph_outputwhile_op_inputwhile_op_outputs r*rr st* ( ) @@,,^< ~ 77 n223   && +r,c|jtjk(r=tjt j |tj|Stj|S)zBLike array_ops.zeros_like() but also accepts resource var handles.r2) r2rrrzerosrvariable_shaperget_zeros_dtype zeros_like)op_input op_outputs r*r r ;sT__' ??!00;..x8 ::   i ((r,ctj|sy|jjdk(r^|jdk(rO|j t jk(sJ|jjd}tj|Sy)z.Returns whether the given tensor is trainable.FTensorListPopBackrr[T) r IsTrainablerr value_indexr2rrr)r element_types r*rrGsq  " "6 *  YY^^**v/A/AQ/F <<6>> )) )99%%o6L  $ $\ 22 r,ct||d}|c|jdDcgc]}tj|}}|j|j}t j |||}||_|Scc}w)zReturns `FuncGraph` for the given function attribute. Args: while_op: The While Operation. func_attr_name: string attr_graph_name: cached forward graph name Returns: `FuncGraph` Nr_)rrr TensorShaper3rlget_func_graphr)rfunc_attr_nameattr_graph_namer s input_shapes func_names r*rrYsx$7*.6->->-O()   #L!!.166I$$X|YGJ*  sA<ctt|k(sJ|jd}tjd|jd} t j j} t j j} | ||gt|z} tj|fd| it||||| | } | jD]\}}tj|| jvr#| jtj|}n$t!dt#|dt#|d| jj%|| j&j%|| | fS) aBuilds and returns the gradient FuncGraph of `func_graph` and its args. The returned grad_func_graph must be called with the returned args + grad_func_graph.captures. Args: ys: A `Tensor` or list of tensors to be differentiated. xs: A `Tensor` or list of tensors to be used for differentiation. grads: The incoming grads for `ys`. cond_graph: FuncGraph for the forward cond function. body_graph: FuncGraph for the forward body function. name: Name of the returned gradient function. while_op: The forward While op. maximum_iterations: Tensor. The maximum number of iterations. Returns: 2-tuple of (grad_func_graph, args). r grad_counterr1c t|Sr')_grad_fn)r;rrrs r*r+z#_create_grad_func..sHRT:6r,r zTensor z which captures zG is in list of internal_captures but not in internal_capture_to_output.)rgrTrror2r!ObjectIdentitySetrrRrsrt_WhileBodyGradFuncGraphrr tensor_idinternal_capture_to_outputrOstrrstructured_outputs)rrrrWrr3rr? total_itersrbody_graph_inputsbody_graph_outputsr;grad_func_graphexternal_captureinternal_capture new_outputs`` ` r*rrps( RCJ    #+  {  ~ 7'&77 8I8IJ&889K9KL %{ 3d5k A$&== 6 B(z:);X):.s .qQd] .srrGrE)r_GradientsHelperall) rrr;r r: grad_outsrr?r0s r*r)r)sv& H'--"g"$) .I . .. . G'AwQ+ A+); 7) CCr,cg}|jD]y}|j|k(rWt|jjD]\}}||us |j|}n|j|jk(sJ|j |{|S)aReturns the tensors to pass as captured inputs to `body_grad_graph`. `body_grad_graph` may have external references to: 1. Its outer graph containing the input gradients. These are left as-is. 2. Accumulators captured from the forward-pass graph. These should have been added as `while_op` outputs after the gradient graph was built. We replace these with the corresponding output of `while_op`, i.e. a tensor in `body_graph.outer_graph`. In the case of nested control flow or functions, the gradient logic handling `body_grad_graph.outer_graph` will make sure the tensor from `body_graph.outer_graph` is also correctly captured. Args: body_graph: FuncGraph. The forward-pass body function. body_grad_graph: FuncGraph. The body gradients function. while_op: The forward-pass While Operation calling `body_graph`. Returns: A list of input tensors to be passed as the captured inputs to `body_grad_graph`. )rJgraph enumeraterTrr)rrrnew_capture_inputsrSroutputs r*rrs*  , ,!a ww* 1)!V Q;q!! WW .. .. .a ! r,c g}d}d}|D]}||jd|j|}|dz }t|tjr<|jtj||||dz||dz|dz }t|t j sJ|j|||dz }|S)a3Returns the values that should be returned from the while grad function. Args: outputs: the raw Tensor outputs of the grad While op. grads: the input gradients to the gradient function. body_grad_graph: _WhileBodyGradFuncGraph. Returns: A list of gradient values. May include Nones. r8NrGrE)valuesindices dense_shape)rr/r r r  tensor_libTensor)rTrrresult outputs_idxstructured_outputs_idxr>rFs r*rrs &+ ay mmD  / /0F GFa&.667 mmN00%+/*kAo.01Qk  1 1 22 2 mmGK()Qk%( -r,ctjtjsJfd}j D]}|j dk7rd}t jjD]\}}||jdusnP||jd}|ht jjD]\}}||us ||k(r|ccSy)aReturns TensorList if any containing accumulated values of tensor. We try to find a pattern of the form: input_tl tensor \ / (TensorListPushBack) | output_tl which satisfies the following conditions: 1. input_tl must be in tensor.graph.inputs. 2. output_tl or Identity(output_tl) must be in tensor.graph.outputs. 3. tensor.graph.input_index(input_tl) == tensor.graph.output_index(output_t). output_tl or Identity(output_tl) (whichever is in tensor.graph.outputs) is returned if such a pattern is found else None is returned. Args: tensor: The Tensor to be accumulated. Returns: A variant tensor in the same graph as `tensor` or None if no accumulator is found. cjjD] }||us|cS|jdjdk(r7t fdjjDrjdSy)zEReturns t or Identity(t) whichever exists in graph outputs else None.rrc3@K|]}jd|uyw)rN)rT)rdrS identity_ops r*rezB_get_accumulator..get_func_graph_output..Ss! FAK   "a ' FsN)rCrT consumersrany)rSrFrSrs @r*get_func_graph_outputz/_get_accumulator..get_func_graph_outputKst,,&& 1++-"KJ& F1E1E FF   ## r,TensorListPushBackrN) r rCrs FuncGraphrTrrDrrT)rrVconsumeraccum_input_idxinprFaccum_output_idxouts` r*rr.s6 FLL"3"="= >> > ""$h}},,O )&,,*=*= > " "  "8#3#3A#6 7F ~!*6<<+?+?!@#  . .- '2 r,OptimizedReductionOpsCacheKey)op_typerr input_typesr3attrsop_defcompute_deviceceZdZdZfdZedZ d fd Z d dZ d dZ fdZ dZ xZ S) r+a FuncGraph for the gradient function of the body of a While op. Contains the logic for capturing the tensors from the body of the forward While op which is as follows: 1. If the tensor is of resource type (these are not accumulated): a. Ensure that the tensor is a loop invariant, i.e., it exists in both loop inputs and outputs at the same index. b. Lookup the corresponding resource tensor in the forward outer graph and try to capture that. 2. If the tensor is not of resource type: a. Create an accumulator for that tensor and output it from the forward pass. Note this also requires adding it as an input to the forward pass. b. Capture the accumulator from the forward pass in this FuncGraph. This will later be resolved to the correct output of the forward While op. c. Pop a value from the captured placeholder and use it as the captured value for the forward pass tensor. This only allows capturing tensors in the forward graph. A ValueError is raised if an attempt is made to capture a tensor not in the forward graph. To manually capture a tensor that is not in the forward graph, call `capture` with `allowlisted=True`. Note: The `captures` dict does not contain the forward tensor since it is not directly captured. It contains the accumulator corresponding to this forward tensor. Attributes: while_op_needs_rewrite: True if any non-resource intermediates were captured, meaning the forward While op needs to be rewritten to output the corresponding accumulators. extra_inputs: list of EmptyTensorList tensors to be used as initial input to the new accumulators in the forward graph. It may also contain external captures of the custom gradient function. internal_capture_to_output: dict from a tensor_id(captured placeholder) to the corresponding tensor that needs to be added to the list of outputs. For instance, when capturing an accumulator TensorList this contains the TensorList obtained after popping a tensor from the list. Other entries in this dict are expected, though not enforced, to be identities. This dict is needed because these output tensors need to be added to FuncGraph.outputs "after" the tensors returned from the gradient function. ctt| |g|_i|_||_||_||_||_i|_ yr') superr+__init__rr-_forward_graph_forward_cond_graph_maximum_iterations_forward_while_op_indirect_captures) selfr3forward_cond_graphforward_body_graphr?forward_while_opr1r2 __class__s r*rhz _WhileBodyGradFuncGraph.__init__sO !41$7D&(D#,D1D1D-D!Dr,c|jSr')r)rns r*rz._WhileBodyGradFuncGraph.while_op_needs_rewrites   r,c nhd} || vrtjs}tfd|Dritd|DrWtjj s8tj j sj||||||||Stt+||||||||S)N>RankSizeShapeTensorListLengthTensorListElementShapec3NK|]}|jjuywr')rCri)rdinputrns r*rez>_WhileBodyGradFuncGraph._create_op_internal..s C5EKK4.. . Cs"%c36K|]}t|duywr')r)rdr{s r*rez>_WhileBodyGradFuncGraph._create_op_internal..s @ U #t + @srrar3rbrcrd) rlrr@util_v1GraphOrParentsInXlaContextri-graph_wrapped_for_higher_order_tape_gradients_move_op_to_forward_graphrgr+_create_op_internal) rnr`rrrar3rbrcrdoptimized_reduction_opsrrs ` r*rz+_WhileBodyGradFuncGraph._create_op_internals6 **  ) ) + CF CC @ @@  . .t/B/B C  > >    !  + +  !',)) ($ C % D ''r,c t|jdsi|j_|j||||||||} |jjj | } | | S|jj 5t j|}|jj||||||||} | |jj| <| cdddS#1swYyxYw)N_optimized_reduction_ops_cacher}) hasattrrir&_get_optimized_reduction_ops_cache_keygetr|r name_from_scope_namer) rnr`rrrar3rbrcrd cache_key cached_oprMs r*rz1_WhileBodyGradFuncGraph._move_op_to_forward_graphs 4&&(H I;=d8;;dE6I##BBFFI   ' ' )  % %d +d""66  !'7)fGMd88C )s AC  C)c Lttd|}| t|}| t|}|Og} t|jD]&\} } | j | | j f(t| }||j }t ||||||||S)Nc"|jSr'r)rSs r*r+zP_WhileBodyGradFuncGraph._get_optimized_reduction_ops_cache_key..6s r,)rmapsorteditemsrSerializeToStringr_) rnr`rrrar3rbrcrdhashable_attrs attr_name attr_values r*rz>_WhileBodyGradFuncGraph._get_optimized_reduction_ops_cache_key+s3(&1 2F V}f+&k n#)%++-#8K )Zy**F*F*HIJKN#e '')f (&&+)-ufn NNr,c |jjtj|}||S|j|j ur}t ||jjv}tt|/||}|sD||jtj|<||jtj|<|S|jjdk(r3|jjd}|jjdk(r3|jjtj|}||St!||j j|j j"rZtt|/||}||jtj|<||jtj|<|St%j&|rXt%j(t+j,||j.}||jtj|<|S|j.t0j2k(r|j5|St7|}|A|j j8j;5t=j>5tAjB|j.|jD|jFtI|}dddddd|jJjM|j j;5tAjN||}ddd|j j"jM||jPj;5|jPjS|dddtt|/||}tAjT||j.\} }||jtj|<| |jtj|<|S#1swYGxYw#1swYLxYw#1swYxYw#1swYxYw)z:Implements the capturing described in the class docstring.Nrrr)r[r\r]r3)r[)+rmrr r,rCriidrby_val_internalrgr+_capture_helperr-rrr_is_loop_invariantrTr is_constantrorconstant_valuer2rr_resource_capture_helperrrr|rlclear_control_inputsrrr(rk_build_accumulator_namerrrrjrLtensor_list_pop_back) rnrr3captured_tensoralready_captured real_value accumulatorrcaptured_accumulatornew_tensor_listrrs r*rz'_WhileBodyGradFuncGraph._capture_helperJs--11#--2GHO"  ||4...Ft'='='M'MM5tL $o !0 '' ) 9H f 56  ))..J &yy"f ))..J & --11#--2GHO"  &$"5"5"<"<--5575"44:D$A . %%cmm '7FdcmmF34 v&''  $ $V ,FLLBj7AdcmmF34 ||v&  * *6 22 #6*K    * * 5 5 74  & & ( 4 22"LL"LL#77*62 4+ 44 {+    ) ) +J44[&I J !!((5  # # . . 06   ((56 !!8$OT(0'D'DFLL(:$O_6EDCMM&12!0 ##CMM% E 4 444JJ66s=#Q 8AP>9Q >QQ%>Q Q  QQ"%Q.c|jtjk(sJ|jjDcgc]}|j }}|jj Dcic]}|j |j}}tj|j |||jj}|jj|}|jjj|}|jtjk(sJ|jtjk(sJ|tj|jj|j |||jjk7rtd||j||jt!j"|<|jt!j"|Scc}wcc}w)aReturns the captured resource tensor. Resource-type tensors are not accumulated. If a resource tensor exists in the loop body it must either be a loop input or an output of a nested While op inside the loop body which had captured the external resource. Args: tensor: the external resource Tensor to be captured. Returns: Tensor in this graph. z)Resource tensors must be loop invariants )r2rrrirr3rnode_defrlresource_input_index _functionsrrTrrLrmr r,) rnrrSforward_graph_input_namesrforward_graph_name_to_opdefindexinput_placeholdertensor_in_outer_graphs r*rz0_WhileBodyGradFuncGraph._resource_capture_helpers <<6?? ** *151D1D1K1K LA L L'+':':'I'I'K#M!##M#M  % % .# && (E ++2259 //66==eD  " "foo 55 5 & &&// 99 9  )) ##E*//1J# &&((  56K5L M OO6:\\6DCMM&12  " "3==#8 995!M#Ms G*)G/)NNNNNT) __name__ __module__ __qualname____doc__rhpropertyrrrrrr __classcell__)rrs@r*r+r+s|(T!*     7'z   ,d   N>m^):r,r+c t|||D]O\}}}tj|j|r(t d|j d|d|jdy)NzInput tensor `z` enters the loop with shape z, but has shape z after one iteration. To allow the shape to vary across iterations, use the `shape_invariants` argument of tf.while_loop to specify a less-specific shape.)rr_ShapeLessThanOrEqualr(rOr3)flat_output_tensorsrflat_input_tensorsrSr(input_ts r*r~r~sr !46K!35=q%  1 1!''5 A  7<<.(EeWM77)$< < ===r,ct|j|k(sJdt|j|fzt|jdk(sJdt|jzt|j|k(sJdt|j|fzt|j|k(sJdt|j|fzy)zEChecks the number of inputs/outputs of `cond_graph` and `body_graph`.z(cond_graph takes %d inputs; Expected: %drGz&cond_graph has %d outputs; Expected: 1z(body_graph takes %d inputs; Expected: %dz'body_graph has %d outputs; Expected: %dN)rgrrT)rWrnum_flattened_loop_varss r*rrs Z   #: :N0C 8I8I4J4K4MMN : Z   A %J.Z5G5G1HHJ % Z   #: :N0C 8I8I4J4K4MMN : Z   $; ;M/3z7I7I3J3J3LLM ;r,c  t|j|j|D]_\}}}|j|jk7s!t d|j d|jd|jd|j d y)Nz Loop var z enters the loop with type z but has type z after 1 iteration. z type should remain constant.)rrrTr2 TypeErrorr3)rrr\r^loop_vars r*rrs 1 1:3E3E 35%c3 yyCII  hmm_$? {K))$8H$ $ %%%r,c>|j|jdzS)N___redundant_placeholder) unique_namer3)rWs r*$_build_cond_placeholders_name_prefixr s    2L L MMr,c |Dcgc]}|jj}}|jj5}t j ||t jt|}dddDcgc],}tjj|j|.}}g}|D] } |j| jd"t||} |Dcgc] }t!|} }t| | D]*\} } |j"j%| | d| dd,|j&j)|ycc}w#1swYxYwcc}wcc}w)zCreates placeholders for body captures in cond_graph. This is needed to match signatures of cond and body graphs. Args: cond_graph: cond branch graph body_graph_captures: Tensors which were captured when building the `body_graph`. NrrGF)keyexternalinternal is_by_ref)r2as_datatype_enum_c_graphrc_apiTF_CreatePlaceholdersras_strrr Operation _from_c_opoperrrTrr_function_capturesadd_or_replacerry)rWbody_graph_capturesrStypesc_graph placeholdersphplaceholder_opsrrtupleskeysrrs r*r}r}sg.A A177 # # A% A IG.. ::FGILI ?K8:cmmrww 3/ ' "b NN2::a=!" "G ,&, -A"Q% -$ -$da!!00 11 1  7#= BII .sE4E>1E"(E'EcXt||D]\}}tj||yr')rrcopy_handle_data) src_tensors tgt_tensorssrc_ttgt_ts r*rHrH9s,+{34leU%%eU34r,cd}ttj|d|Dcgc]}|| }}tj||dScc}w)zGLike `nest.pack_sequence_as` but also replaces flows with TensorArrays.cft|tjrtj||S|Sr')r r TensorArraybuild_ta_with_new_flow)flowtas r*flow_to_tensor_arrayz/_pack_sequence_as..flow_to_tensor_arrayAs4AK  ( (B*  3 3B =5/35r,Tr$)rr rIpack_sequence_as)r>r=rXrzrs r*r5r5>sd5 4<< TB&( A   24G15 77 s A c8d}tj||dS)NcRt|tjr |jS|Sr')r rrr)maybe_tas r*fz _tensor_array_to_flow..fPs"(,889 ]] Or,Tr$)r rh)rXrs r*rQrQNs   AyD AAr,cV|d}tj|tjdS)NrXr?r1)r r rint32)r?s r*rnrnXs1   3G IIr,cXdj|jjddS)Nz{}/accumulator:_)formatr3replacer s r*rrbs$   - 5 5c3 ??r,cXtfd|Dxrtfd|DS)Nc3&K|]}|u ywr'rrdrSrs r*rez%_is_loop_invariant..hs *afk *c3&K|]}|u ywr'rrs r*rez%_is_loop_invariant..is +afk +r)rU)rrrTs` r*rrgs) *6 * * , +7 + +-r,cttt|j}|D],}|sn(t j |}|j |}.tj|tjt|y)zSets the list of resource inputs which are read-only. This is used by AutomaticControlDependencies. Args: op: While Operation. branch_graphs: List of branch FuncGraphs. N) setrangergracd*get_read_only_resource_input_indices_graph intersectionr set_int_list_attrREAD_ONLY_RESOURCE_INPUTS_ATTRr)r branch_graphsread_only_indices branch_graphbranch_read_only_indicess r*rrls|%BII/0#Ql  "MM )667OP QC>>013r,)N NNTT)KrrAtensorflow.core.frameworkrtensorflow.python.clientrrtensorflow.python.eagerrtensorflow.python.frameworkrrrrr rsr r rrKrrrtensorflow.python.opsrrrr~rrlrrrrrrrrrtensorflow.python.utilrr r!r"rRegisterGradientrrrrrr rrrr)rrr namedtupler_rwr+r~rrrr}rHr5rQrnrrrrr,r*rs4?1F3.G6+<433+2>>24;02**2B)'21!%#%"&%)EP&'gqF3==qF(qFh%FP(V+` )$.CL$DN&R%PBJ!7 6 6# & !b:d55b:J = M%N($V4 7 BI@ - 3CMM3r,