VhnddlmZddlmZmZddlmZddlmZddl m Z GddejZ y))Mapping)AnyUnionN)Tensor) ShardedTensorczeZdZdeeeeefffdZd de fdZ ddZ de ee ffdZd eee ffd Zd e fd Zy)ShardedOptimizer named_paramsc^g}|jD]J}t|tr'|jd|j D:|j |L||_||g|i||_|jj|_|jj|_ y)a ShardedOptimizer collects all tensors and local shard tensors of ShardedTensor, then use these tensors as ``params`` for optimizers Args: named_params (Dict[str, Union[Tensor, ShardedTensor]]) : a Dict of parameters, where key is the parameter key, value is either Tensor or ShardedTensor parameter. optimizer_class (torch.optim.Optimizer): the Optimizer to use locally, i.e. torch.optim.SGD, torch.optim.Adagrad, etc. *optimizer_args: the arguments to initialize the optimizer. **optimizer_kwargs: the key-word arguments to initialize the optimizer. c34K|]}|jywN)tensor).0 local_shards Z/home/dcms/DCMS/lib/python3.12/site-packages/torch/distributed/_shard/sharded_optim/api.py z,ShardedOptimizer.__init__..#s+6K&&sN) values isinstancerextend local_shardsappendr _optim param_groupsstate)selfr optimizer_classoptimizer_argsoptimizer_kwargstensorsvalues r__init__zShardedOptimizer.__init__ s*!#!((* &E%/:?:L:L:Nu%  &)%gSSBRS  KK44[[&& set_to_nonec:|jj|y)aResets the gradients of all optimized :class:`torch.Tensor` s. Args: set_to_none (bool): instead of setting to zero, set the grads to None. This will in general have lower memory footprint, and can modestly improve performance. However, it changes certain behaviors. For example: 1. When the user tries to access a gradient and perform manual ops on it, a None attribute or a Tensor full of 0s will behave differently. 2. If the user requests ``zero_grad(set_to_none=True)`` followed by a backward pass, ``.grad``\ s are guaranteed to be None for params that did not receive a gradient. 3. ``torch.optim`` optimizers have a different behavior if the gradient is 0 or None (in one case it does the step with a gradient of 0 and in the other it skips the step altogether). N)r zero_grad)rr#s rr%zShardedOptimizer.zero_grad.s k*r"Nc:|jj|y)afPerforms a single optimization step (parameter update). Args: closure (Callable): A closure that reevaluates the model and returns the loss. Optional for most optimizers. .. note:: Unless otherwise specified, this function should not modify the ``.grad`` field of the parameters. N)rstep)rclosures rr'zShardedOptimizer.step?s !r"returnctd)z Returned state and param_groups will contain parameter keys instead of parameter indices like torch.optim.Optimizer. This allows for advanced functionality like optimizer re-sharding to be implemented. z0ShardedOptimizer state_dict not implemented yet!NotImplementedError)rs r state_dictzShardedOptimizer.state_dictLs""TUUr"r-ctd)zLoads the ShardedOptimizer state. Args: state_dict (dict): ShardedOptimizer state. Should be an object returned from a call to :meth:`state_dict`. z5ShardedOptimizer load_state_dict not implemented yet!r+)rr-s rload_state_dictz ShardedOptimizer.load_state_dictUs" C  r" param_groupctd)zAdd a new param groupz5ShardedOptimizer add_param_group not implemented yet!r+)rr0s radd_param_groupz ShardedOptimizer.add_param_groupas" C  r")Tr )__name__ __module__ __qualname__rstrrrrr!boolr%r'dictrr-r/r2r"rr r sl!'c5)>#??@!'F+T+" "VDcNV  '#s(*;   3 r"r ) collections.abcrtypingrr torch.optimoptimtorchr'torch.distributed._shard.sharded_tensorr Optimizerr r9r"rrAs&#A\ u\ r"