Vh^ UddlmZddlmZmZddlZgZeee d<ejjjZ ejjjZe d deej j"deej$j&deeefded df d Zd ej$j.d eej j"fd Zy))Iterable)Any no_type_checkN__all__optimizer_classparamsoptimizer_kwargs register_hookreturnctjjdtdtjj ddffd }|D] }|| y)a Upon ``backward()``, the optimizer specified for each parameter will fire after the gradient has been accumulated into the parameter. Note - gradients for these parameters will be set to None after ``backward()``. This means that any other optimizer not specified via `_apply_optimizer_in_backward` over this parameter will be a no-op. Args: optimizer_class: (Type[torch.optim.Optimizer]): Optimizer to apply to parameter params: (Iterator[nn.Parameter]): parameters to apply optimizer state to optimizer_kwargs: (Dict[str, Any]): kwargs to pass to optimizer constructor register_hook: (bool): whether to register a hook that runs the optimizer after gradient for this parameter is accumulated. This is the default way that optimizer in backward is implemented, but specific use cases (such as DDP) may wish to override this to implement custom behavior. (Default = True) Example:: params_generator = model.parameters() param_1 = next(params_generator) remainder_params = list(params_generator) apply_optimizer_in_backward(torch.optim.SGD, [param_1], {"lr": 0.02}) apply_optimizer_in_backward(torch.optim.Adam, remainder_params, {"lr": 0.04}) model(...).sum().backward() # after backward, parameters will already # have their registered optimizer(s) applied. z3torch.distributed.optim.apply_optimizer_in_backwardparamr Nctvr2jjjddt<gfi}t dsg_g_g_j j|j jjjsydfd }tj|}tvr gt<tj|y)Nr_in_backward_optimizerscVjD]}|jd_y)N)rstepgrad)_unusedoptr s c/home/dcms/DCMS/lib/python3.12/site-packages/torch/distributed/optim/apply_optimizer_in_backward.pyoptimizer_hookzc_apply_optimizer_in_backward.._apply_optimizer_in_backward_to_param..optimizer_hookVs*44   EJ)r N) param_to_acc_grad_mapview_asgrad_fnnext_functionshasattrr_optimizer_classes_optimizer_kwargsappendr param_to_optim_hook_handle_map)r optimizerrhandlerr r s` r%_apply_optimizer_in_backward_to_paramzK_apply_optimizer_in_backward.._apply_optimizer_in_backward_to_param8s - -+0==+?+G+G+V+V,, !% ($UG@/?@ u78,.E )(*E $&(E # %%,,Y7   ''8 &&'78   'u-;;NK 6 646 *5 1&u-44VCDE Mr)T)collections.abcrtypingrrr$rliststr__annotations__utilsweakWeakTensorKeyDictionaryr rtypeoptim Optimizerr'r(dictboolr)Moduler0rrr@s$% c "'!1!1!I!I!K ((@@B  P5%++//0P5 UXX'' (P538nP5 P5  P5P5fDAVAV