Vh *ddlZddlZddlmZmZmZddlmZddlZddlmZddl m Z m Z m Z gdZ eejejejfZdZe dded ed ed eed ejf d Ze ddededejd eed df dZe ddeded ed ed eed ejf dZede ddeded ed ed eed ejf dZe ddeded eed dfdZy)N)castOptionalUnion) deprecated)Tensor)_device_has_foreach_support"_group_tensors_by_device_and_dtype_has_foreach_support)clip_grad_norm_clip_grad_normclip_grad_value_c>fd}tj||S)z This wrapper is needed to avoid a circular import when using @torch.no_grad on the exposed functions clip_grad_norm_ and clip_grad_value_ themselves. chtj5|i|cdddS#1swYyxYwN)torchno_grad)argskwargsfuncs H/home/dcms/DCMS/lib/python3.12/site-packages/torch/nn/utils/clip_grad.py_no_grad_wrapperz"_no_grad.._no_grad_wrapper$s. ]]_ )(( ) ) )s(1) functoolsupdate_wrapper)rrs` r_no_gradrs! )-t4 tensors norm_typeerror_if_nonfiniteforeachreturnc t|tjr|g}n t|}t |}t |dk(rtj dS|dj}t|g}g}|jD]\\}}\\} }| t| |s |r1t|r&|jtj| |O|rtd|jd|j| D cgc]"} tj j#| |$c} tj j#tj$|D cgc]} | j'|c} |} |rAtj(| j+| j-rtd|d| Scc} wcc} w)aCompute the norm of an iterable of tensors. The norm is computed over the norms of the individual tensors, as if the norms of the individual tensors were concatenated into a single vector. Args: tensors (Iterable[Tensor] or Tensor): an iterable of Tensors or a single Tensor that will be normalized norm_type (float): type of the used p-norm. Can be ``'inf'`` for infinity norm. error_if_nonfinite (bool): if True, an error is thrown if the total norm of :attr:`tensors` is ``nan``, ``inf``, or ``-inf``. Default: ``False`` foreach (bool): use the faster foreach-based implementation. If ``None``, use the foreach implementation for CUDA and CPU native tensors and silently fall back to the slow implementation for other device types. Default: ``None`` Returns: Total norm of the tensors (viewed as a single vector). rg:foreach=True was passed, but can't use the foreach API on  tensorszThe total norm of order z for gradients from `parameters` is non-finite, so it cannot be clipped. To disable this error and scale the gradients by the non-finite norm anyway, set `error_if_nonfinite=False`) isinstancerrlistfloatlentensordevicer itemsr rextend _foreach_norm RuntimeErrortypelinalg vector_normstackto logical_orisnanisinf) rrrr first_devicegrouped_tensorsnormsr)_device_tensorsgnorm total_norms r_get_total_normr>,s8'5<<()w-i I 7|q||C  1:$$L +    E.=.C.C.E * *&~ O 4^V L 3F; LL,,^YG H LV[[MYab  LLAOPA))!Y7P  )) u=tTWW\*=> Je..z/?/?/A:CSCSCUV&yk2- -  Q>s >'G G parametersmax_normr=cht|tjr|g}|Dcgc]}|j|j}}t |}t |dk(ryt |g}||dzz }tj|d}|jD]\\} } \\} } | t| | s |r1t| r&tj| |j| O|rtd| jd|j| } | D]} | j| ycc}w)aScale the gradients of an iterable of parameters given a pre-calculated total norm and desired max norm. The gradients will be scaled by the following calculation .. math:: grad = grad * \frac{max\_norm}{total\_norm + 1e-6} Gradients are modified in-place. This function is equivalent to :func:`torch.nn.utils.clip_grad_norm_` with a pre-calculated total norm. Args: parameters (Iterable[Tensor] or Tensor): an iterable of Tensors or a single Tensor that will have gradients normalized max_norm (float): max norm of the gradients total_norm (Tensor): total norm of the gradients to use for clipping foreach (bool): use the faster foreach-based implementation. If ``None``, use the foreach implementation for CUDA and CPU native tensors and silently fall back to the slow implementation for other device types. Default: ``None`` Returns: None Nrgư>g?)maxr"r#)r$rrgradr&r'r clampr*r r _foreach_mul_r2r-r.mul_)r?r@r=rpgrads grouped_grads clip_coefclip_coef_clampedr)r9 device_gradsclip_coef_clamped_devicer;s r_clip_grads_with_norm_rNss<@*ell+ \ ' >166+=QVV >E >XH 5zQ +   J-.I I37,9,?,?,A 1( (n|a O 4\6 J 3F;    .?.B.B6.J K LV[[MYab (9';';F'C $! 1/0 1 1 ?s D/D/ct|tjr|g}n t|}|Dcgc]}|j|j}}t ||||}t |||||Scc}w)aLClip the gradient norm of an iterable of parameters. The norm is computed over the norms of the individual gradients of all parameters, as if the norms of the individual gradients were concatenated into a single vector. Gradients are modified in-place. This function is equivalent to :func:`torch.nn.utils.get_total_norm` followed by :func:`torch.nn.utils.clip_grads_with_norm_` with the ``total_norm`` returned by ``get_total_norm``. Args: parameters (Iterable[Tensor] or Tensor): an iterable of Tensors or a single Tensor that will have gradients normalized max_norm (float): max norm of the gradients norm_type (float): type of the used p-norm. Can be ``'inf'`` for infinity norm. error_if_nonfinite (bool): if True, an error is thrown if the total norm of the gradients from :attr:`parameters` is ``nan``, ``inf``, or ``-inf``. Default: False (will switch to True in the future) foreach (bool): use the faster foreach-based implementation. If ``None``, use the foreach implementation for CUDA and CPU native tensors and silently fall back to the slow implementation for other device types. Default: ``None`` Returns: Total norm of the parameter gradients (viewed as a single vector). )r$rrr%rCr>rN)r?r@rrrrGrHr=s rr r spD*ell+ \ *% ' >166+=QVV >E >  3EwOJ:xWE  ?s A.A.z_`torch.nn.utils.clip_grad_norm` is now deprecated in favor of `torch.nn.utils.clip_grad_norm_`.)categoryc t|||||S)zClip the gradient norm of an iterable of parameters. .. warning:: This method is now deprecated in favor of :func:`torch.nn.utils.clip_grad_norm_`. )r )r?r@rrrs rr r s$ :x166+=QVV >E >6w?M%2%8%8%: K! \guq O$T$v,%>vN5f=  % %d4<&?* M  % %d4<&? L LV[[MYab  KVT"))zkz)J K K ?s E E )g@FNr)rtypingrrrtyping_extensionsrrrtorch.utils._foreach_utilsrr r __all__Iterable_tensor_or_tensorsrr&boolr>rNr FutureWarningr r rrras# (((   LL OOELL!#  $" C CCCd^ C  \\ C CL # <1"<1<1 <1d^ <1  <1 <1~ $" )"))) ) d^ )  \\ ) )X 4 $" Y" Y Y Y Y d^ Y  \\ Y  Y  #(K"(K(Kd^(K (K (Kr