Vhq<ddlZddlZddlmZddlmZdddddZy)N)ExpandedWeight)_pytreesumT) batch_sizeloss_reduction batch_firstc2fdfddvrtdttjjs!tdt j 1tts!tdt j dkrtdjD]*}t|d s|jtd |d tjjfd }|S) a; Return a forward function for a module, populating grad_sample with per sample gradients on backward invocation. Args: module: The ``nn.Module`` to get per sample gradients with respect to. All trainable parameters will compute per sample gradients, located in a ``grad_sample`` field when ``backward`` is invoked batch_size: The batch size of the input. If None is passed, all tensor arguments in args and kwargs must have the same batch size, which is the size of the first dimension. Otherwise, it must be passed manually. Default: None loss_reduction: Indicates if the loss reduction (for aggregating the gradients) is a sum or a mean operation. If "mean", per sample gradients will be scaled by the batch size to offset the crossbatch interaction from running mean across a batch. Must be "mean" or "sum". Default: "sum" batch_first: Indicates if the batch dimension is the first dimension. If True, the batch dimension is the first dimension. If False, it's the second dimension. Default: True. Examples:: >>> # xdoctest: +SKIP >>> model = nn.Linear(4, 3) >>> batched_input = torch.randn(5, 4) # batch size of 5 >>> res = call_for_per_sample_grads(model)(batched_input).sum() >>> res.backward() >>> assert model.weight.shape == (3, 4) >>> assert model.weight.grad_sample.shape == (5, 3, 4) >>> assert model.weight.grad is None >>> assert model.bias.shape == (3,) >>> assert model.bias.grad_sample.shape == (5, 3) >>> assert model.bias.grad is None An example using "mean" loss reduction. The grad_sample fields will be scaled by batch_size from what they would be if we ran the same code with loss_reduction="sum". This is because the mean at the end will scale all grad_outputs by 1 / batch_size from cross batch interaction. >>> model = nn.Linear(4, 3) >>> batched_input = torch.randn(5, 4) # batch size of 5 >>> res = call_for_per_sample_grads(model, 5, loss_reduction="mean")(batched_input).mean() >>> res.backward() Note:: Does not work with any `nn.RNN`, including `nn.GRU` or `nn.LSTM`. Please use custom rewrites that wrap an `nn.Linear` module. See Opacus for an example c:|jr t||S|SN) requires_gradr) og_tensorrrs O/home/dcms/DCMS/lib/python3.12/site-packages/torch/nn/utils/_per_sample_grad.pymaybe_build_expanded_weightz>call_for_per_sample_grads..maybe_build_expanded_weight<s!  " "!)ZH H c tj|i|}d}|D]X}t|tjsr|j dn|j d}|||k7rt d|d|d|}Z| t d|S)NrzDWhen computing batch size, found at least one input with batch size z and one with batch size zV. Please specify it explicitly using the batch size kwarg in call_for_per_sample_gradszUnable to find a tensor in the passed args and kwargs. They may not be pytree-able and so ExpandedWeights cannot compute the batch size from the inputs. Please specify it explicitly)pytreearg_tree_leaves isinstancetorchTensorshape RuntimeError)argskwargsargs_and_kwargsrargarg_batch_sizers rcompute_batch_sizez5call_for_per_sample_grads..compute_batch_sizeBs 00$A&A " (Cc5<<0-8SYYq\ciilN%**F"Z!l";N;KLYY (J (     r)rmeanz8Expected loss_reduction argument to be sum or mean, got z%Module passed must be nn.Module, got z2Batch size passed must be None or an integer, got rz!Batch size must be positive, got grad_samplezCurrent Expanded Weights accumulates the gradients, which will be incorrect for multiple calls without clearing gradients. Please clear out the grad_sample parameter of zC or post an issue to pytorch/pytorch to prioritize correct behaviorc }||i|} jDcic]\}}|||}}}tjj |||Scc}}wr )named_parametersrfuncfunctional_call) rrwrapper_batch_sizenamevalueparamsrrrmodules rwrapperz*call_for_per_sample_grads..wrapperps{'  %!3T!DV!D "(!8!8!: u -e5GH H  zz))&&$GG  sA)rrrnnModuletype__name__int parametershasattrr! functoolswrapsforward)r*rrrweightr+rrs```` @@rcall_for_per_sample_gradsr7 s8b ._,F~FV W   fehhoo .3DL4I4I3J K    *Z"=@jAQAZAZ@[ \  *q.>zlKLL##% 6= )f.@.@.LccibjkRR __V^^$ H% H Nr)r3r6torch.nn.utils._expanded_weights.expanded_weights_implr torch.utilsrrr7rrr;s" Q) qr