BVh%PdZddlZddlZddlmZddlmZddlmZddlm Z ddlm Z ddlm Z dd l m Z dd lmZdd lmZdd lmZdd lmZddlmZddlmZddlmZddlmZiZdZeed<dZ e ed<ejBZ"iZ#dZ$dZ%ejLZ'e jPe%dde'Z)e jPe%dde'Z*dZ+ d#dZ,ejZe,edg Gd!d"Z.y)$z5Utilities for forward-mode automatic differentiation.N)function_cache) pywrap_tfe)backprop) backprop_util)execute)forwardprop_util)tracing_compilation)ops) tensor_shape) array_ops)control_flow_ops)UnconnectedGradients) tf_logging)nest) tf_exportcX~~~|Dcgc]}tj|c}Scc}wNr identity attr_tupleinputsoutputstangentsts S/home/dcms/DCMS/lib/python3.12/site-packages/tensorflow/python/eager/forwardprop.py _identity_jvpr*s*&')1 2A)  Q  22 2'IdentitycX~~~|Dcgc]}tj|c}Scc}wrrrs r_read_variable_jvpr!5s*&')1 2A)  Q  22 2rReadVariableOpc t5tj|ddzt|<dddtj|d}| |||||S|sgSt j 5g}g}g}t |D]Q\} } tj| s|j| |j| |j|| Stj5} tj5} | j|tj||||dddg} g}g}t |D]c\}}tj|s| jtj |d|j||j|e| j|  j#||| t$j&}ddd j# |}dgt)|z}t+|D] \}}|||< |cdddS#1swYxYw#1swYxYw#1swYjxYw#1swYyxYw)aComputes a Jacobian-vector product for an op. Note that this function would be wasteful if executed eagerly. It runs the backward gradient function and throws away the result just to record its operations on a GradientTape. These unused ops are pruned away when this function is traced. Args: op_name: A string, the type of operation being executed. attr_tuple: Attributes of the operation. inputs: A flat list of input Tensors to the operation. outputs: A flat list of output Tensors from the operation. tangents: A flat list of Tensors, same shape as `inputs`. Returns: A flat list of tangents corresponding to `outputs`. rNunused_forwardprop_aid)name)unconnected_gradients)output_gradients)_TRACE_COUNT_CONSISTENCY_LOCK _TRACE_COUNTget_SPECIAL_CASESrpush_forwardprop_state enumerater IsTrainableappendr GradientTapewatchrrecord_gradientr ones_likegradientrZEROlenzip)op_namerrrr special_casetrainable_inputstrainable_indicesnontrivial_tangents input_indextensortranspose_tape backfunc_tapeforwardprop_aidstrainable_outputsnontrivial_output_indices output_indexoutputgradsnontrivial_output_tangentsoutput_tangentsindextangents r _jvp_helperrLEs{$%=),,Wa81>  I..0%(0: V  " "6 *'  -""8K#89 :    ;N  "Fm,-WEF"$"+G"49 ,  $ $V ,  ! !!!&/GH J  " "6 * # * *< 8 9+,$$    4 9 9 %;e!;*"0!8!8 2E"9"Gfs7|+O78:'w&oe' K%%==2FF;;%%sV!H&+-I A I %I:*H3$4IA>IAI &H03H= 8II I  Ic h|rt||D]b\}}|jjdg|jzr0tdj |dg|jz|jt j tjt|||||St|||||S)aComputes a batch of Jacobian-vector product for an op. Args: op_name: A string, the type of operation being executed. attr_tuple: Attributes of the operation. inputs: A flat list of input Tensors to the operation. outputs: A flat list of output Tensors from the operation. tangents: A flat list of Tensors, compatible with shape `[None] + input_shape`. use_batch: A bool, True to vetorize over batch of tangents of shape `[None] + input_shape`. Returns: A flat list of tangents compatible with `outputs` or `[None] + output_shape`. Raises: ValueError: if tangent shapes are not compatible with input shapes. NzDTangent {} was expected to be of shape {} but is instead of shape {}) r8shapeis_compatible_with ValueErrorformatr vectorized_map functoolspartialrL)r9rrrr use_batchprimalrKs r_jvp_helper_wrapperrWs*vx0M ]] - -tfv||.C D99?$tfv||&;W]]:LM MM  * *+w FGL  Wj&'8 DD_jvp_relaxed_shapesT)r&reduce_retracingr_jvp_exact_shapesF ctj|dtkrt}nt}t j ||||||f|S)z-Determine which forwardprop function to call.r)tracing_options)r*r+_TRACE_COUNT_LIMIT_jvp_exact_config_jvp_relaxed_configr call_function)r9rrrrrUconfigs r _jvp_dispatchrdsJgq!$66 F F  * * FGXyA rXzautodiff.ForwardAccumulator)v1cneZdZdZdZdZdZdZdZdZ e jfdZ e fd ZxZS) ForwardAccumulatoraComputes Jacobian-vector products ("JVP"s) using forward-mode autodiff. Compare to `tf.GradientTape` which computes vector-Jacobian products ("VJP"s) using reverse-mode autodiff (backprop). Reverse mode is more attractive when computing gradients of a scalar-valued function with respect to many inputs (e.g. a neural network with many parameters and a scalar loss). Forward mode works best on functions with many outputs and few inputs. Since it does not hold on to intermediate activations, it is much more memory efficient than backprop where it is applicable. Consider a simple linear regression: >>> x = tf.constant([[2.0, 3.0], [1.0, 4.0]]) >>> targets = tf.constant([[1.], [-1.]]) >>> dense = tf.keras.layers.Dense(1) >>> dense.build([None, 2]) >>> with tf.autodiff.ForwardAccumulator( ... primals=dense.kernel, ... tangents=tf.constant([[1.], [0.]])) as acc: ... loss = tf.reduce_sum((dense(x) - targets) ** 2.) >>> acc.jvp(loss) The example has two variables containing parameters, `dense.kernel` (2 parameters) and `dense.bias` (1 parameter). Considering the training data `x` as a constant, this means the Jacobian matrix for the function mapping from parameters to loss has one row and three columns. With forwardprop, we specify a length-three vector in advance which multiplies the Jacobian. The `primals` constructor argument is the parameter (a `tf.Tensor` or `tf.Variable`) we're specifying a vector for, and the `tangents` argument is the "vector" in Jacobian-vector product. If our goal is to compute the entire Jacobian matrix, forwardprop computes one column at a time while backprop computes one row at a time. Since the Jacobian in the linear regression example has only one row, backprop requires fewer invocations: >>> x = tf.constant([[2.0, 3.0], [1.0, 4.0]]) >>> targets = tf.constant([[1.], [-1.]]) >>> dense = tf.keras.layers.Dense(1) >>> dense.build([None, 2]) >>> loss_fn = lambda: tf.reduce_sum((dense(x) - targets) ** 2.) >>> kernel_fprop = [] >>> with tf.autodiff.ForwardAccumulator( ... dense.kernel, tf.constant([[1.], [0.]])) as acc: ... kernel_fprop.append(acc.jvp(loss_fn())) >>> with tf.autodiff.ForwardAccumulator( ... dense.kernel, tf.constant([[0.], [1.]])) as acc: ... kernel_fprop.append(acc.jvp(loss_fn())) >>> with tf.autodiff.ForwardAccumulator(dense.bias, tf.constant([1.])) as acc: ... bias_fprop = acc.jvp(loss_fn()) >>> with tf.GradientTape() as tape: ... loss = loss_fn() >>> kernel_grad, bias_grad = tape.gradient(loss, (dense.kernel, dense.bias)) >>> np.testing.assert_allclose( ... kernel_grad, tf.stack(kernel_fprop)[:, tf.newaxis]) >>> np.testing.assert_allclose(bias_grad, bias_fprop[tf.newaxis]) Implicit in the `tape.gradient` call is a length-one vector which left-multiplies the Jacobian, a vector-Jacobian product. `ForwardAccumulator` maintains JVPs corresponding primal tensors it is watching, derived from the original `primals` specified in the constructor. As soon as a primal tensor is deleted, `ForwardAccumulator` deletes the corresponding JVP. `acc.jvp(x)` retrieves `acc`'s JVP corresponding to the primal tensor `x`. It does not perform any computation. `acc.jvp` calls can be repeated as long as `acc` is accessible, whether the context manager is active or not. New JVPs are only computed while the context manager is active. Note that `ForwardAccumulator`s are always applied in the order their context managers were entered, so inner accumulators will not see JVP computation from outer accumulators. Take higher-order JVPs from outer accumulators: >>> primal = tf.constant(1.1) >>> with tf.autodiff.ForwardAccumulator(primal, tf.constant(1.)) as outer: ... with tf.autodiff.ForwardAccumulator(primal, tf.constant(1.)) as inner: ... primal_out = primal ** tf.constant(3.5) >>> inner_jvp = inner.jvp(primal_out) >>> inner_jvp # 3.5 * 1.1 ** 2.5 >>> outer.jvp(inner_jvp) # 3.5 * 2.5 * 1.1 ** 1.5 Reversing the collection in the last line to instead retrieve `inner.jvp(outer.jvp(primal_out))` will not work. Strict nesting also applies to combinations of `ForwardAccumulator` and `tf.GradientTape`. More deeply nested `GradientTape` objects will ignore the products of outer `ForwardAccumulator` objects. This allows (for example) memory-efficient forward-over-backward computation of Hessian-vector products, where the inner `GradientTape` would otherwise hold on to all intermediate JVPs: >>> v = tf.Variable([1., 2.]) >>> with tf.autodiff.ForwardAccumulator( ... v, ... # The "vector" in Hessian-vector product. ... tf.constant([1., 0.])) as acc: ... with tf.GradientTape() as tape: ... y = tf.reduce_sum(v ** 3.) ... backward = tape.gradient(y, v) >>> backward # gradient from backprop >>> acc.jvp(backward) # forward-over-backward Hessian-vector product ctjd|_d|_t }t j |D]4}t||vr td|jt|6|j||y)aSpecify tensors to watch and their Jacobian-vector products. Mathematically, `tangents` is a vector right-multiplying the Jacobian matrix (a Jacobian-vector product) for the function computed while this accumulator is active. Since JVPs are computed in forward mode as the computation happens, this vector must be supplied in advance. Listing a single tensor multiple times in `primals` raises an exception. Excluding a tensor from `primals` is equivalent to watching it with a tangent tensor of zeros. Args: primals: A tensor or nested structure of tensors to watch. tangents: A tensor or nested structure of tensors, with the same nesting structure as `primals`, with each element being a vector with the same size as the corresponding primal element. Raises: ValueError: If the same tensor or variable is specified multiple times in `primals`. FTensor {} was specified as a primal multiple times. This may indicate an error. If it was intended, please sum the corresponding tangents.N) rTFE_Py_ForwardAccumulatorNew _accumulator _recordingsetrflattenidrPadd_watch)selfprimalsr primal_idsrVs r__init__zForwardAccumulator.__init__Ts},#??FDDOJ,,w'! Fz ! &' 'nnRZ ! KK"rXc&|j|Sr)_push_accumulatorrrs r __enter__zForwardAccumulator.__enter__vs KrXc>|jr|jyyr)rl_pop_accumulator)rrtypvalue tracebacks r__exit__zForwardAccumulator.__exit__zs  rXc~|jr tdtj|jd|_y)Nz!Accumulator is already recording.T)rlrPrTFE_Py_ForwardAccumulatorSetAddrkrxs rrwz$ForwardAccumulator._push_accumulator~s0  : ;;..t/@/@ADOrXc~|js tdtj|jd|_y)NzAccumulator is not recording.F)rlrPr"TFE_Py_ForwardAccumulatorSetRemoverkrxs rr{z#ForwardAccumulator._pop_accumulators0 ?? 6 7711$2C2CDDOrXc>fd}tj|||y)aaEnsures that `primals` are being traced by this accumulator. Mathematically, `tangents` is a vector right-multiplying the Jacobian matrix (a Jacobian-vector product) for the function computed while this accumulator is active. Since JVPs are computed in forward mode as the computation happens, this vector must be supplied in advance. Watching a single tensor multiple times sums each of its `tangents`. Any un-watched tensor has zeros for its tangent vector. Args: primals: A Tensor or list of Tensors. tangents: A Tensor or list of Tensors matching `primals`. cl|jjs0tjtjdd|jt j ||j}t|drt j |j}tjj||y)NzJThe dtype of the watched primal must be floating (e.g. tf.float32), got %r)dtypehandle) r is_floatinglogging log_first_nWARNr convert_to_tensorhasattrrrTFE_Py_ForwardAccumulatorWatchrk)rVrKrrs rrqz)ForwardAccumulator._watch.._watchs \\ % % LL123V\\ C%%gV\\Bg  "&&v}}5//0A0A6079rXN)r map_structure)rrrsrrqs` rrqzForwardAccumulator._watchs 9 vw1rXctj tdfd}tj||S)aFetches the Jacobian-vector product computed for `primals`. Note that this method performs no computation, and simply looks up a JVP that was already computed (unlike backprop using a `tf.GradientTape`, where the computation happens on the call to `tape.gradient`). Args: primals: A watched Tensor or structure of Tensors to fetch the JVPs for. unconnected_gradients: A value which can either hold 'none' or 'zero' and alters the value which will be returned if no JVP was computed for `primals`. The possible values and effects are detailed in 'tf.UnconnectedGradients' and it defaults to 'none'. Returns: Tensors with the same shapes and dtypes as `primals`, or None if no JVP is available. z,Called jvp() without first tracing anything.ct|dr tj|j}n|}t j j |}|(tjk(rtj|}|S)Nr) rr rrrTFE_Py_ForwardAccumulatorJVPrkrr6r zeros_like)r?unwrapped_tensorresultrrr's r _fetch_jvpz*ForwardAccumulator.jvp.._fetch_jvpsm  "00?!66t7H7H7GIf 15I5N5NN%%f- mrX)rrkrPrr)rrrsr'rs` ` rjvpzForwardAccumulator.jvpsC$11FG  E FF    j' 22rXctt| |||}d|_t j d|_t}ttj|tj|D]s\}}|jjtjdg|jzt||vr t!d|j#t|u|j%|||S)aFactory constructor to test accumulator on batches of tangents. Args: primals: A tensor or nested structure of tensors to watch. tangents: A tensor or nested structure of tensors, with the same nesting structure as `primals`, with each element being a vector with compatible shape `[None] + primal.shape` of the corresponding primal element. Returns: A batch accumulator object. FTNri)superrg__new__rlrrjrkrmr8rrnrNassert_is_compatible_withr TensorShaperorPrprq)clsrsraccrtrVrK __class__s r_batch_accumulatorz%ForwardAccumulator._batch_accumulators "C 0gx HCCN!>>tDCJt||G4dll86LM! mm--  " "D6 *V\\ 9; Fz ! &' 'nnRZ !JJw! JrX)__name__ __module__ __qualname____doc__ruryrrwr{rqrNONEr classmethodr __classcell__)rs@rrgrgsOkZ #D  2>0D/H/H!3FrXrg)F)/rrS threading%tensorflow.core.function.polymorphismrtensorflow.pythonrtensorflow.python.eagerrrrr,tensorflow.python.eager.polymorphic_functionr tensorflow.python.frameworkr r tensorflow.python.opsr "tensorflow.python.ops.parallel_forr +tensorflow.python.ops.unconnected_gradientsrtensorflow.python.platformrrtensorflow.python.utilr tensorflow.python.util.tf_exportrr,rr!Lockr)r*rLrW FunctionCache_jvp_function_cacheTracingOptionsrar`r_rdTFE_Py_RegisterJVPFunctionrgrXrrs*<@(,1+4L+4+?L<'6 3+z3$6 !/  0 EP E\3n2248)88 & 7'66 & " && %%m4 (R0AA1ArX