Vh· ddlZddlZddlZddlZddlZddlZddlmZmZm Z ddlm Z ddl m Z m Z ddlmZmZmZmZddlZddlmZddlmcmcmZddlmZddlmZmZddl m!Z!ddl"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(dd l)m*Z*dd l+m,Z,m-Z-m.Z.m/Z/m0Z0m1Z1m2Z2m3Z3m4Z4m5Z5m6Z6m7Z7m8Z8dd l9m:Z:m;Z;mZ>m?Z?m@Z@mAZAmBZBmCZCdd lDmEZEdd lFmGZGmHZHmIZImJZJmKZKmLZLmMZMmNZNmOZOmPZPmQZQmRZRmSZSmTZTddlUmVZVddlWmXZXddlYmZZZm[Z[ddl\m]Z]m^Z^m_Z_m`Z`maZambZbmcZcmdZdddlemfZfddlgmhZhmiZimjZjmkZkmlZlddlmmnZnmoZoddgZpdZqGdde ZrGddeje#Ztdeejdevdejdejdejf d Zyd!ej8jdezej8je{ffd"Z|d!ej8jdeze{ej8jffd#Z}y)$N) GeneratorIterableIterator)contextmanager)autoEnum)AnyCallableOptionalUnion)_CHECKPOINT_WRAPPED_MODULEActivationWrapper)LOW_PRECISION_HOOKS) _FSDPState_get_param_to_fqns FSDP_PREFIXFSDP_WRAPPED_MODULEHandleTrainingState TrainingState)_annotate_modules_for_dynamo) _check_orig_params_flattened_init_buffer_state_init_core_state_init_device_handle_init_extension_init_ignored_module_states_init_param_handle_from_module_init_prefetching_state_init_process_group_state_init_runtime_state_init_state_dict_stateHYBRID_SHARDING_STRATEGIESProcessGroupType) _get_fsdp_root_states _is_fsdp_root _lazy_init _post_forward_post_forward_reshard _pre_forward_pre_forward_unshard_root_pre_forward_unshard_wait_for_computation_stream) _auto_wrap)BackwardPrefetch CPUOffloadFullOptimStateDictConfigFullStateDictConfigLocalOptimStateDictConfigLocalStateDictConfigMixedPrecisionOptimStateDictConfigShardedOptimStateDictConfigShardedStateDictConfigShardingStrategyStateDictConfigStateDictSettings StateDictType) DeviceMesh) _p_assert) FlatParameterFlatParamHandle)_flatten_optim_state_dict'_get_param_id_to_param_from_optim_input_get_param_key_to_param'_get_param_to_param_id_from_optim_input_get_param_to_param_key_optim_state_dict_rekey_sharded_optim_state_dict_set_optim_use_dtensor)_register_all_state_dict_hooks)_deregister_orig_params_register_flat_param_register_orig_params_unshard_params_unshard_params_for_summon) CustomPolicyModuleWrapPolicyFullyShardedDataParallelOptimStateKeyType _flat_paramc,eZdZdZeZeZy)rSz6Represents the type of key in an optimizer state-dict.N)__name__ __module__ __qualname____doc__r PARAM_NAMEPARAM_IDb/home/dcms/DCMS/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.pyrSrSns@JvHr]c#eZdZdZddddej ddddddddddfdejdede e de e d e e e eefd e ed e ed e eej jd e e ejgdfde e eej(fdedededede e eej j,e eej jfde ef fd ZedejfdZedefdZede efdZdedeffd Z dedeffd Z!defdZ"e# dfdejdede$dfd Z%d!e ejgdfddffd" Z&defd#Z'defd$Z(dgd%Z)e# dhdejd&e*d'e e+d(e e,de-f d)Z.e#dejde-fd*Z/e#e0jb dhdejd&e*d'e e+d(e e,de2f d+Z3d,ed-edefd.Z4e#e0jb didejd/ed0ed1ed2ed3ede2fd4Z5e0jbd5Z6fd6Z7de8e9eejtfffd7 Z;de8e9eej j,fffd8 Ze1de2fd;Z?ej djdZBe#d?d@dAefdBZCe#defdCZDe#d?d@dDedEedAefdFZEe# dkd?dGdHej jdIejjdJeHeefdKe e e$eHeefeej j,fd1edLedMe eIjdedNedeHeeffdOZKe# dldJeHeefdHej jdKe e e$eHeefeej j,fdIe ejjdLed1edPedMe eIjdeHeeffdQZLe# dmdHej jdIejjdKe e e$eHeefeej j,fd1edMe eIjdeHeeff dRZMe# dndHej jdIejjdMe eIjdeHeeffdSZNe# dhdTeHeefdHej jdKe e e$eHeefeej j,fdIe ejjdeHeeff dUZOe#dVeHeefdHej jdIejjdeHeeffdWZPe# dodTe eHeefdHej jdKe e e$eHeefeej j,fdIe ejjdMe edeHeeff dXZQe# dhdJeHeefdYeRdHej jdKe e e$eHeefeej j,fdIe ejjdeHeeff dZZSe# dhdHej jdIejjdJe eHeefdMe eIjdeHeeff d[ZTe# dpdHej jdIejjdJeHeefdPed\edMe eIjdeHeeffd]ZUd9eVd^eWfd_ZXdfd`efdaZYdbZZe0jbdce=dde[fdeZ\xZ]S)qrRaqAA wrapper for sharding module parameters across data parallel workers. This is inspired by `Xu et al.`_ as well as the ZeRO Stage 3 from DeepSpeed_. FullyShardedDataParallel is commonly shortened to FSDP. .. _`Xu et al.`: https://arxiv.org/abs/2004.13336 .. _DeepSpeed: https://www.deepspeed.ai/ To understand FSDP internals, refer to the :ref:`fsdp_notes`. Example:: >>> # xdoctest: +SKIP("undefined variables") >>> import torch >>> from torch.distributed.fsdp import FullyShardedDataParallel as FSDP >>> torch.cuda.set_device(device_id) >>> sharded_module = FSDP(my_module) >>> optim = torch.optim.Adam(sharded_module.parameters(), lr=0.0001) >>> x = sharded_module(x, y=3, z=torch.Tensor([1])) >>> loss = x.sum() >>> loss.backward() >>> optim.step() Using FSDP involves wrapping your module and then initializing your optimizer after. This is required since FSDP changes the parameter variables. When setting up FSDP, you need to consider the destination CUDA device. If the device has an ID (``dev_id``), you have three options: * Place the module on that device * Set the device using ``torch.cuda.set_device(dev_id)`` * Pass ``dev_id`` into the ``device_id`` constructor argument. This ensures that the FSDP instance's compute device is the destination device. For option 1 and 3, the FSDP initialization always occurs on GPU. For option 2, the FSDP initialization happens on module's current device, which may be a CPU. If you're using the ``sync_module_states=True`` flag, you need to ensure that the module is on a GPU or use the ``device_id`` argument to specify a CUDA device that FSDP will move the module to in the FSDP constructor. This is necessary because ``sync_module_states=True`` requires GPU communication. FSDP also takes care of moving input tensors to the forward method to the GPU compute device, so you don't need to manually move them from CPU. For ``use_orig_params=True``, ``ShardingStrategy.SHARD_GRAD_OP`` exposes the unsharded parameters, not the sharded parameters after forward, unlike ``ShardingStrategy.FULL_SHARD``. If you want to inspect the gradients, you can use the ``summon_full_params`` method with ``with_grads=True``. With ``limit_all_gathers=True``, you may see a gap in the FSDP pre-forward where the CPU thread is not issuing any kernels. This is intentional and shows the rate limiter in effect. Synchronizing the CPU thread in that way prevents over-allocating memory for subsequent all-gathers, and it should not actually delay GPU kernel execution. FSDP replaces managed modules' parameters with ``torch.Tensor`` views during forward and backward computation for autograd-related reasons. If your module's forward relies on saved references to the parameters instead of reacquiring the references each iteration, then it will not see FSDP's newly created views, and autograd will not work correctly. Finally, when using ``sharding_strategy=ShardingStrategy.HYBRID_SHARD`` with the sharding process group being intra-node and the replication process group being inter-node, setting ``NCCL_CROSS_NIC=1`` can help improve the all-reduce times over the replication process group for some cluster setups. **Limitations** There are several limitations to be aware of when using FSDP: * FSDP currently does not support gradient accumulation outside ``no_sync()`` when using CPU offloading. This is because FSDP uses the newly-reduced gradient instead of accumulating with any existing gradient, which can lead to incorrect results. * FSDP does not support running the forward pass of a submodule that is contained in an FSDP instance. This is because the submodule's parameters will be sharded, but the submodule itself is not an FSDP instance, so its forward pass will not all-gather the full parameters appropriately. * FSDP does not work with double backwards due to the way it registers backward hooks. * FSDP has some constraints when freezing parameters. For ``use_orig_params=False``, each FSDP instance must manage parameters that are all frozen or all non-frozen. For ``use_orig_params=True``, FSDP supports mixing frozen and non-frozen parameters, but it's recommended to avoid doing so to prevent higher than expected gradient memory usage. * As of PyTorch 1.12, FSDP offers limited support for shared parameters. If enhanced shared parameter support is needed for your use case, please post in `this issue `__. * You should avoid modifying the parameters between forward and backward without using the ``summon_full_params`` context, as the modifications may not persist. Args: module (nn.Module): This is the module to be wrapped with FSDP. process_group (Optional[Union[ProcessGroup, Tuple[ProcessGroup, ProcessGroup]]]): This is the process group over which the model is sharded and thus the one used for FSDP's all-gather and reduce-scatter collective communications. If ``None``, then FSDP uses the default process group. For hybrid sharding strategies such as ``ShardingStrategy.HYBRID_SHARD``, users can pass in a tuple of process groups, representing the groups over which to shard and replicate, respectively. If ``None``, then FSDP constructs process groups for the user to shard intra-node and replicate inter-node. (Default: ``None``) sharding_strategy (Optional[ShardingStrategy]): This configures the sharding strategy, which may trade off memory saving and communication overhead. See :class:`ShardingStrategy` for details. (Default: ``FULL_SHARD``) cpu_offload (Optional[CPUOffload]): This configures CPU offloading. If this is set to ``None``, then no CPU offloading happens. See :class:`CPUOffload` for details. (Default: ``None``) auto_wrap_policy (Optional[Union[Callable[[nn.Module, bool, int], bool], ModuleWrapPolicy, CustomPolicy]]): This specifies a policy to apply FSDP to submodules of ``module``, which is needed for communication and computation overlap and thus affects performance. If ``None``, then FSDP only applies to ``module``, and users should manually apply FSDP to parent modules themselves (proceeding bottom-up). For convenience, this accepts ``ModuleWrapPolicy`` directly, which allows users to specify the module classes to wrap (e.g. the transformer block). Otherwise, this should be a callable that takes in three arguments ``module: nn.Module``, ``recurse: bool``, and ``nonwrapped_numel: int`` and should return a ``bool`` specifying whether the passed-in ``module`` should have FSDP applied if ``recurse=False`` or if the traversal should continue into the module's subtree if ``recurse=True``. Users may add additional arguments to the callable. The ``size_based_auto_wrap_policy`` in ``torch.distributed.fsdp.wrap.py`` gives an example callable that applies FSDP to a module if the parameters in its subtree exceed 100M numel. We recommend printing the model after applying FSDP and adjusting as needed. Example:: >>> def custom_auto_wrap_policy( >>> module: nn.Module, >>> recurse: bool, >>> nonwrapped_numel: int, >>> # Additional custom arguments >>> min_num_params: int = int(1e8), >>> ) -> bool: >>> return nonwrapped_numel >= min_num_params >>> # Configure a custom `min_num_params` >>> my_auto_wrap_policy = functools.partial(custom_auto_wrap_policy, min_num_params=int(1e5)) backward_prefetch (Optional[BackwardPrefetch]): This configures explicit backward prefetching of all-gathers. If ``None``, then FSDP does not backward prefetch, and there is no communication and computation overlap in the backward pass. See :class:`BackwardPrefetch` for details. (Default: ``BACKWARD_PRE``) mixed_precision (Optional[MixedPrecision]): This configures native mixed precision for FSDP. If this is set to ``None``, then no mixed precision is used. Otherwise, parameter, buffer, and gradient reduction dtypes can be set. See :class:`MixedPrecision` for details. (Default: ``None``) ignored_modules (Optional[Iterable[torch.nn.Module]]): Modules whose own parameters and child modules' parameters and buffers are ignored by this instance. None of the modules directly in ``ignored_modules`` should be :class:`FullyShardedDataParallel` instances, and any child modules that are already-constructed :class:`FullyShardedDataParallel` instances will not be ignored if they are nested under this instance. This argument may be used to avoid sharding specific parameters at module granularity when using an ``auto_wrap_policy`` or if parameters' sharding is not managed by FSDP. (Default: ``None``) param_init_fn (Optional[Callable[[nn.Module], None]]): A ``Callable[torch.nn.Module] -> None`` that specifies how modules that are currently on the meta device should be initialized onto an actual device. As of v1.12, FSDP detects modules with parameters or buffers on meta device via ``is_meta`` and either applies ``param_init_fn`` if specified or calls ``nn.Module.reset_parameters()`` otherwise. For both cases, the implementation should *only* initialize the parameters/buffers of the module, not those of its submodules. This is to avoid re-initialization. In addition, FSDP also supports deferred initialization via torchdistX's (https://github.com/pytorch/torchdistX) ``deferred_init()`` API, where the deferred modules are initialized by calling ``param_init_fn`` if specified or torchdistX's default ``materialize_module()`` otherwise. If ``param_init_fn`` is specified, then it is applied to all meta-device modules, meaning that it should probably case on the module type. FSDP calls the initialization function before parameter flattening and sharding. Example:: >>> # xdoctest: +SKIP("undefined variables") >>> module = MyModule(device="meta") >>> def my_init_fn(module: nn.Module): >>> # E.g. initialize depending on the module type >>> ... >>> fsdp_model = FSDP(module, param_init_fn=my_init_fn, auto_wrap_policy=size_based_auto_wrap_policy) >>> print(next(fsdp_model.parameters()).device) # current CUDA device >>> # With torchdistX >>> module = deferred_init.deferred_init(MyModule, device="cuda") >>> # Will initialize via deferred_init.materialize_module(). >>> fsdp_model = FSDP(module, auto_wrap_policy=size_based_auto_wrap_policy) device_id (Optional[Union[int, torch.device]]): An ``int`` or ``torch.device`` giving the CUDA device on which FSDP initialization takes place, including the module initialization if needed and the parameter sharding. This should be specified to improve initialization speed if ``module`` is on CPU. If the default CUDA device was set (e.g. via ``torch.cuda.set_device``), then the user may pass ``torch.cuda.current_device`` to this. (Default: ``None``) sync_module_states (bool): If ``True``, then each FSDP module will broadcast module parameters and buffers from rank 0 to ensure that they are replicated across ranks (adding communication overhead to this constructor). This can help load ``state_dict`` checkpoints via ``load_state_dict`` in a memory efficient way. See :class:`FullStateDictConfig` for an example of this. (Default: ``False``) forward_prefetch (bool): If ``True``, then FSDP *explicitly* prefetches the next forward-pass all-gather before the current forward computation. This is only useful for CPU-bound workloads, in which case issuing the next all-gather earlier may improve overlap. This should only be used for static-graph models since the prefetching follows the first iteration's execution order. (Default: ``False``) limit_all_gathers (bool): If ``True``, then FSDP explicitly synchronizes the CPU thread to ensure GPU memory usage from only *two* consecutive FSDP instances (the current instance running computation and the next instance whose all-gather is prefetched). If ``False``, then FSDP allows the CPU thread to issue all-gathers without any extra synchronization. (Default: ``True``) We often refer to this feature as the "rate limiter". This flag should only be set to ``False`` for specific CPU-bound workloads with low memory pressure in which case the CPU thread can aggressively issue all kernels without concern for the GPU memory usage. use_orig_params (bool): Setting this to ``True`` has FSDP use ``module`` 's original parameters. FSDP exposes those original parameters to the user via :meth:`nn.Module.named_parameters` instead of FSDP's internal :class:`FlatParameter` s. This means that the optimizer step runs on the original parameters, enabling per-original-parameter hyperparameters. FSDP preserves the original parameter variables and manipulates their data between unsharded and sharded forms, where they are always views into the underlying unsharded or sharded :class:`FlatParameter`, respectively. With the current algorithm, the sharded form is always 1D, losing the original tensor structure. An original parameter may have all, some, or none of its data present for a given rank. In the none case, its data will be like a size-0 empty tensor. Users should not author programs relying on what data is present for a given original parameter in its sharded form. ``True`` is required to use ``torch.compile()``. Setting this to ``False`` exposes FSDP's internal :class:`FlatParameter` s to the user via :meth:`nn.Module.named_parameters`. (Default: ``False``) ignored_states (Optional[Iterable[torch.nn.Parameter]], Optional[Iterable[torch.nn.Module]]): Ignored parameters or modules that will not be managed by this FSDP instance, meaning that the parameters are not sharded and their gradients are not reduced across ranks. This argument unifies with the existing ``ignored_modules`` argument, and we may deprecate ``ignored_modules`` soon. For backward compatibility, we keep both ``ignored_states`` and `ignored_modules``, but FSDP only allows one of them to be specified as not ``None``. device_mesh (Optional[DeviceMesh]): DeviceMesh can be used as an altenative to process_group. When device_mesh is passed, FSDP will use the underlying process groups for all-gather and reduce-scatter collective communications. Therefore, these two args need to be mutually exclusive. For hybrid sharding strategies such as ``ShardingStrategy.HYBRID_SHARD``, users can pass in a 2D DeviceMesh instead of a tuple of process groups. For 2D FSDP + TP, users are required to pass in device_mesh instead of process_group. For more DeviceMesh info, please visit: https://pytorch.org/tutorials/recipes/distributed_device_mesh.html NFTmodule process_groupsharding_strategy cpu_offloadauto_wrap_policybackward_prefetchmixed_precisionignored_modules param_init_fn device_idsync_module_statesforward_prefetchlimit_all_gathersuse_orig_paramsignored_states device_meshctjjdt|t |t jt jfrtjd|dt||||t|||j| t||j|||_t#||||||g|||||| | | | | ||j|d }|t$vr||j&|j(f|d<t+|||j|j|t,d}d}t/||||| |||t1|t3||| t5||t7||t9||| | | ||_|s"t=||jt?||tA|tC|d|_"y)Nztorch.distributed.fsdpzRFSDP will not all-gather parameters for containers that do not implement forward:  stacklevel) rarbrcrerfrhrirjrkrlrmrnrorar?)#torch_C_log_api_usage_oncesuper__init__ isinstancenn ModuleList ModuleDictwarningswarnrr_ignored_paramsr_ignored_modules _device_meshrr"ra_inter_node_pgr.rRrr rrrr_fsdp_wrapped_modulerrLr!rJ _zero_scalar)selfr`rarbrcrdrerfrgrhrirjrkrlrmrnro root_kwargsbackward_prefetch_limitforward_prefetch_limit __class__s r^rxz!FullyShardedDataParallel.__init__s. $$%=>  fr}}bmm< = MM**03  $D&/>RD&$*>*> J %VT-B-BOT(!         '!.%6*%6#2!.&&8$4%6#2"&"6"6*K!$>>;CV150B0BDDWDW/X O,  %%$$(  #$!"       # "  D!&79IJ4(k*&       %+! (t/C/C D t , t$&t, r]returnct|jtrt|jtS|jS)zReturn the wrapped module.)ryrrgetattrr rs r^r`zFullyShardedDataParallel.modules5 d//1B C4446PQ Q(((r]c:t|dxr|jduS)z:Returns whether this FSDP instance manages any parameters._handleN)hasattrrrs r^ _has_paramsz$FullyShardedDataParallel._has_paramsstY'DDLL,DDr]cJ|jr|jjSdSN)r flat_paramrs r^rTz$FullyShardedDataParallel._flat_params*.,,t||&&@D@r]namecn t||S#t$rt|j|cYSwxYw)z1Forward missing attributes to the wrapped module.)rw __getattr__AttributeErrorrr)rrrs r^rz$FullyShardedDataParallel.__getattr__s: <7&t, , <444d; ; None): function to be applied to each submodule Returns: Module: self NTF) writeback rank0_onlyoffload_to_cpu with_grads) _is_root _assert_staterIDLErOrwapplyrr_reset_lazy_init)rr uninitializedretr`rs r^rzFullyShardedDataParallel.apply?s -  =--. (      $'-#C $ T]])::4@ *'') * # $ $s BBc2|jjduS)a(Return whether the user explicitly enabled buffer mixed precision. NOTE: Unlike parameters and gradient reduction, buffer mixed precision is applied at the FSDP instance level, not the ``FlatParameter`` level, which may be different for the composable code path. N)rf buffer_dtypers r^$_mixed_precision_enabled_for_buffersz=FullyShardedDataParallel._mixed_precision_enabled_for_buffersgs##00<>> # xdoctest: +SKIP("undefined variables") >>> model = DDP(FSDP(...)) >>> FSDP.set_state_dict_type( >>> model, >>> StateDictType.SHARDED_STATE_DICT, >>> state_dict_config = ShardedStateDictConfig(offload_to_cpu=True), >>> optim_state_dict_config = OptimStateDictConfig(offload_to_cpu=True), >>> ) >>> param_state_dict = model.state_dict() >>> optim_state_dict = FSDP.optim_state_dict(model, optim) Args: module (torch.nn.Module): Root module. state_dict_type (StateDictType): the desired ``state_dict_type`` to set. state_dict_config (Optional[StateDictConfig]): the configuration for the target ``state_dict_type``. optim_state_dict_config (Optional[OptimStateDictConfig]): the configuration for the optimizer state dict. Returns: A StateDictSettings that include the previous state_dict type and configuration for the module. aFSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .Nz#Expected state_dict_config of type but got z)Expected optim_state_dict_config of type z6All FSDP modules should have the same state_dict_type.z>All FSDP modules must have the same type of state_dict_config.zDAll FSDP modules must have the same type of optim_state_dict_config.)r}r~ FutureWarningr<FULL_STATE_DICTr2LOCAL_STATE_DICTr4SHARDED_STATE_DICTr8r1r3r7type RuntimeErrorrr_state_dict_type_state_dict_configry_optim_state_dict_configr;) r`rrr_state_dict_type_to_config _optim_state_dict_type_to_configstate_dict_config_typeoptim_state_dict_config_typeprev_state_dict_typeprev_state_dict_configprev_optim_state_dict_config submodules r^set_state_dict_typez,FullyShardedDataParallel.set_state_dict_typexs<`   c    ) )+>  * *,@  , ,.D& "  ) )+C  * *,E  , ,.I, ("t|j|j|j }||k(sJd|d|dt |||S)aGet the state_dict_type and the corresponding configurations for the FSDP modules rooted at ``module``. The target module does not have to be an FSDP module. Returns: A ``StateDictSettings`` containing the state_dict_type and state_dict / optim_state_dict configs that are currently set. Raises: ``AssertionError`` if the ``StateDictSettings`` for different FSDP submodules differ. N)rrrz>vF FI"*&7$-$>$>&/&B&B,5,N,N'# 'y2EF%6..0066&" +.@@-.e4G3HK@'y2DE% F&#"r]c#Ktj||||}dtj||j|j|jyw)aSet the ``state_dict_type`` of all the descendant FSDP modules of the target module. This context manager has the same functions as :meth:`set_state_dict_type`. Read the document of :meth:`set_state_dict_type` for the detail. Example:: >>> # xdoctest: +SKIP("undefined variables") >>> model = DDP(FSDP(...)) >>> with FSDP.state_dict_type( >>> model, >>> StateDictType.SHARDED_STATE_DICT, >>> ): >>> checkpoint = model.state_dict() Args: module (torch.nn.Module): Root module. state_dict_type (StateDictType): the desired ``state_dict_type`` to set. state_dict_config (Optional[StateDictConfig]): the model ``state_dict`` configuration for the target ``state_dict_type``. optim_state_dict_config (Optional[OptimStateDictConfig]): the optimizer ``state_dict`` configuration for the target ``state_dict_type``. N)rRrrrr)r`rrrprev_state_dict_settingss r^rz(FullyShardedDataParallel.state_dict_typesV>$<#O#O    # $   44  $ 4 4 $ 6 6 $ < <  sAAargskwargsc |j}tjjj d5t ||||\}}d}t ||t|j||\}}|rQt|jj|jk(d|jd|jj|j|i|}t||t|||cdddS#1swYyxYw)zjRun the forward pass for the wrapped module, inserting FSDP-specific pre- and post-forward sharding logic.z FullyShardedDataParallel.forwardNz5Expected `FlatParameter` to be on the compute device r)rrtautogradprofilerrecord_functionr+r)r*rr>rdevicecompute_devicer'r()rrrhandleunusedoutputs r^forwardz FullyShardedDataParallel.forwardBs ^^ $ $ 4 4 .  -T4vFLD&F'$)) LD&%%,,0C0CCK**+9V5F5F5M5M4NP /T..??F f3T66)   s B-C--C6recurserrrrc#^Kt||||||5ddddy#1swYyxYww)a1Expose full params for FSDP instances with this context manager. Can be useful *after* forward/backward for a model to get the params for additional processing or checking. It can take a non-FSDP module and will summon full params for all contained FSDP modules as well as their children, depending on the ``recurse`` argument. .. note:: This can be used on inner FSDPs. .. note:: This can *not* be used within a forward or backward pass. Nor can forward and backward be started from within this context. .. note:: Parameters will revert to their local shards after the context manager exits, storage behavior is the same as forward. .. note:: The full parameters can be modified, but only the portion corresponding to the local param shard will persist after the context manager exits (unless ``writeback=False``, in which case changes will be discarded). In the case where FSDP does not shard the parameters, currently only when ``world_size == 1``, or ``NO_SHARD`` config, the modification is persisted regardless of ``writeback``. .. note:: This method works on modules which are not FSDP themselves but may contain multiple independent FSDP units. In that case, the given arguments will apply to all contained FSDP units. .. warning:: Note that ``rank0_only=True`` in conjunction with ``writeback=True`` is not currently supported and will raise an error. This is because model parameter shapes would be different across ranks within the context, and writing to them can lead to inconsistency across ranks when the context is exited. .. warning:: Note that ``offload_to_cpu`` and ``rank0_only=False`` will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with ``rank0_only=True``. Args: recurse (bool, Optional): recursively summon all params for nested FSDP instances (default: True). writeback (bool, Optional): if ``False``, modifications to params are discarded after the context manager exits; disabling this can be slightly more efficient (default: True) rank0_only (bool, Optional): if ``True``, full parameters are materialized on only global rank 0. This means that within the context, only rank 0 will have full parameters and the other ranks will have sharded parameters. Note that setting ``rank0_only=True`` with ``writeback=True`` is not supported, as model parameter shapes will be different across ranks within the context, and writing to them can lead to inconsistency across ranks when the context is exited. offload_to_cpu (bool, Optional): If ``True``, full parameters are offloaded to CPU. Note that this offloading currently only occurs if the parameter is sharded (which is only not the case for world_size = 1 or ``NO_SHARD`` config). It is recommended to use ``offload_to_cpu`` with ``rank0_only=True`` to avoid redundant copies of model parameters being offloaded to the same CPU memory. with_grads (bool, Optional): If ``True``, gradients are also unsharded with the parameters. Currently, this is only supported when passing ``use_orig_params=True`` to the FSDP constructor and ``offload_to_cpu=False`` to this method. (Default: ``False``) N)rN)r`rrrrrs r^summon_full_paramsz+FullyShardedDataParallel.summon_full_params]s7L GY NJ      s-! -*-c#0Kt|jdtj|D]}t || dtj|D]}t ||y#tj|D]}t ||wxYww)aDeregister the original parameters and expose the :class:`FlatParameter`. If a :class:`FlatParameter` is sharded, then this refreshes the sharded views before exiting. This method should only be called when using the original parameters. zR`_deregister_orig_params_ctx()` should only be called when `_use_orig_params=True`N)r>_use_orig_paramsrrrKrM)r fsdp_modules r^_deregister_orig_params_ctxz4FullyShardedDataParallel._deregister_orig_params_ctxs   ! ! & +;;DA >K #K = > @ .??E @ %k;? @??E @ %k;? @s=BA+'B+(BBc|jr|jntj}|5t ||i|cdddS#1swYyxYw)zgDeregister the original parameters and expose the :class:`FlatParameter` s before calling ``_apply()``.N)rr contextlib nullcontextrw_apply)rrrcontextrs r^rzFullyShardedDataParallel._applysX$$  , , .'')   37>4262 3 3 3s A  Ac/K|jtjk(}t||i|D]#\}}|r|j t d}||f%yw)aReturn an iterator over module buffers, yielding both the name of the buffer and the buffer itself. Intercepts buffer names and removes all occurrences of the FSDP-specific flattened buffer prefix when inside the :meth:`summon_full_params` context manager. N)training_staterSUMMON_FULL_PARAMSrw named_buffersreplacer)rrrshould_clean_name buffer_namebufferrs r^rz&FullyShardedDataParallel.named_buffersse!//=3S3SS#(7#8$#I&#I ( K *11+rB ' '  (AAc/K|jtjk(}t||i|D]#\}}|r|j t d}||f%yw)a)Return an iterator over module parameters, yielding both the name of the parameter and the parameter itself. Intercepts parameter names and removes all occurrences of the FSDP-specific flattened parameter prefix when inside the :meth:`summon_full_params` context manager. rN)rrrrwnamed_parametersrr)rrrr param_nameparamrs r^rz)FullyShardedDataParallel.named_parametersse!//=3S3SS!&!94!J6!J & J (// R@ u% %  &rstatect|tr|g}|j|vr\d|d|j}|jdk(r0t d|t d|t j t|y)z!Assert we are in the given state.zexpected to be in states z but current state is rzAsserting FSDP instance is: zERROR: N)ryrrrankprint traceback print_stack ValueError)rrmsgs r^rz&FullyShardedDataParallel._assert_states e] +GE   e ++E73))*,  yyA~4TF;<uo&%%'S/ ! ,r]c#Kt|||js td|jtj g}|j D]7}t|ts|j||jfd|_ 9 d|D]\}}|jrJd||_ !y#|D]\}}|jrJd||_ !wxYww)aDisable gradient synchronizations across FSDP instances. Within this context, gradients will be accumulated in module variables, which will later be synchronized in the first forward-backward pass after exiting the context. This should only be used on the root FSDP instance and will recursively apply to all children FSDP instances. .. note:: This likely results in higher memory usage because FSDP will accumulate the full model gradients (instead of gradient shards) until the eventual sync. .. note:: When used with CPU offloading, the gradients will not be offloaded to CPU when inside the context manager. Instead, they will only be offloaded right after the eventual sync. zb`no_sync()` on inner FSDP instances is not supported. Please call `no_sync()` on root FSDP module.FNzX`_sync_gradients` was incorrectly set to `True` while in the `no_sync()` context manager) r&rrrrrmodulesryrRappend_sync_gradients)r old_flagsmold_flags r^no_syncz FullyShardedDataParallel.no_syncs$ 4}}t  =--.  *A!56  !Q%6%6!78$)! * - ( - 8,,F,%-!  -y - 8,,F,%-!  -s$A(C$+&C$B;%C$;&C!!C$max_norm norm_typec t|||js td|j&t j d|j |_|jtjtd|jD}|r9tjjj|j||St!|}t!|}t#}t#}g}g}g}|jD]} | j$r|} |} n|} |} | j&rk| j(j*D]Q} | | vs| j-| | j/| | j07|j/| j0S| j(| vs| j-| j(| j/| j(| j(j0|j/| j(j0|jD]Y} | |vxr| |v} | s|j-| |j/| | j0?|j/| j0[t3|||j|j }|r"t3|||j|j nd}|t4j6k(r_|t j8||n|}t;j<|tj>j@jB|jDn8||z}t;j<||jD||||zz }|d|z z}|jFjHr|jK}||d zz }t jL|d }|D]7}|jO|jQ|jR|jT9tW|d k(r1tYjZd |j\d |jT|St_j`tjb|Dcgc]}|jTc}}|jQ|Scc}w)aClip the gradient norm of all parameters. The norm is computed over all parameters' gradients as viewed as a single vector, and the gradients are modified in-place. Args: max_norm (float or int): max norm of the gradients norm_type (float or int): type of the used p-norm. Can be ``'inf'`` for infinity norm. Returns: Total norm of the parameters (viewed as a single vector). If every FSDP instance uses ``NO_SHARD``, meaning that no gradients are sharded across ranks, then you may directly use :func:`torch.nn.utils.clip_grad_norm_`. If at least some FSDP instance uses a sharded strategy (i.e. one other than ``NO_SHARD``), then you should use this method instead of :func:`torch.nn.utils.clip_grad_norm_` since this method handles the fact that gradients are sharded across ranks. The total norm returned will have the "largest" dtype across all parameters/gradients as defined by PyTorch's type promotion semantics. For example, if *all* parameters/gradients use a low precision dtype, then the returned norm's dtype will be that low precision dtype, but if there exists at least one parameter/ gradient using FP32, then the returned norm's dtype will be FP32. .. warning:: This needs to be called on all ranks since it uses collective communications. zC`clip_grad_norm_()` should only be called on the root FSDP instanceNgrc36K|]}|j ywr)uses_sharded_strategy).0rs r^ z;FullyShardedDataParallel.clip_grad_norm_..Zs 17,, , s)opgroup)rg?gư>)maxrz&Called FSDP.clip_grad_norm_() on rank zD with no gradients -- returning the total norm in the default dtype )2r&rrrrttensorrrrrall _all_handlesrzutilsclip_grad_norm_ parametersfloatsetrrr_paramsaddrgrad_get_grad_normmathinfmaximumdist all_reduce distributedReduceOpMAXrarcoffload_paramscpuclampmul_tordtypelenr}r~r functoolsreduce promote_types)rr r  all_no_shardsharded_params_setnonsharded_params_setsharded_paramsnonsharded_paramsgradsr target_set target_listrnot_fsdp_managedlocal_sharded_normlocal_nonsharded_norm total_norm clip_coefclip_coef_clampedr total_norm_dtypes r^rz(FullyShardedDataParallel.clip_grad_norm_,s/H 4}}U     $ % S9L9L MD  =--. ;?;L;L   88>>11!8Y  ?)$  U #$&'' =F++/ , 2 / &&#..665EJ."u-#**51 ::1!LL4 5$$J6NN6#4#45&&v'8'89((--9 V%6%6%;%;<' =(__& -E//VEAV4V  %))%0!((/::)LL, -, It'8'8$:M:M ! !9d.?.?ATAT    )4 02GH'  OOu0099==TEWEW ,Y6J OOJd.@.@ A%03Y>> #i8J    * *#)J T 12 "KK s; ED II'**4;; C D E u:? MM8 DN##$&   $++   $) *DTZZ * }}-.. +sR r?rrrscH| tjdt|dzyy)NzThe `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.r?rrr}r~r) optim_inputrss r^_warn_optim_inputz*FullyShardedDataParallel._warn_optim_inputs)  " MMW%>   #r]c||y|yy)NTFr\)rEoptims r^_is_using_optim_inputz.FullyShardedDataParallel._is_using_optim_inputs  5=  "r]currnewc Vtjd|d|d|dt|dzy)Nz``FullyShardedDataParallel.zD``is being deprecated and is replaced by ``FullyShardedDataParallel.z``. ``FullyShardedDataParallel.z$`` may be removed after PyTorch 2.2.r?rrrD)rJrKrss r^_warn_legacy_optim_state_dictz6FullyShardedDataParallel._warn_legacy_optim_state_dicts> )$0669U;**./S U !A~  r]) _stacklevelmodelrHoptim_state_dictrEfull_state_dictrrNc F |r1tj||dztj||} nd} ||rJtj|dj t fdtj|DsJdt |||||| || | S)aRTransform the state-dict of an optimizer corresponding to a sharded model. This is the internal API that is used by all the optim_state_dict implementations. Given model, optim, the original optim_state_dict, this API removes the FSDP internal information and internal sharding from the optim_state_dict. r?rrFrc3<K|]}|jk(ywrrrrrms r^rzBFullyShardedDataParallel._optim_state_dict_impl..$  q11 1 9Not all FSDP modules have the same _use_orig_params value) rOrHrPrEr shard_staterusing_optim_inputrmrc)rRrFrIrrrrG) rOrHrPrErrQrrcrNrZrms @r^_optim_state_dict_implz/FullyShardedDataParallel._optim_state_dict_impls0  $ 6 6 a 7 !9 N N!  !& &z 992??F     -::5A   G G G !-#!++/+#  r]is_named_optimizerc |r,tj|tj||}nd}||rJtj|dj t fdtj|DsJd|rt j|dkDri}t|| |r|nd||} t| |||||S)a} Convert an optimizer state-dict so that it can be loaded into the optimizer associated with the FSDP model. This is the internal API that is used by all the load optim_state_dict implementations. Given model, optim, and the saved optim_state_dict, this API adds the FSDP internal information and internal sharding to the optim_state_dict. FNrc3<K|]}|jk(ywrrTrUs r^rzJFullyShardedDataParallel._optim_state_dict_to_load_impl..>rVrWrX)rOrmrHrr)rOrHrErZr\) rRrFrIrrrr%get_rankrBrH) rPrOrErHrQrr\rrZ sharded_osdrms @r^_optim_state_dict_to_load_implz7FullyShardedDataParallel._optim_state_dict_to_load_impls.  $ 6 6{ C 8 N N!  !& &z 992??F     -::5A   G G G $--.2! / +.5D!  / #/1   r]c tjdddtj|||j|||ddS)a$ Return the full optimizer state-dict. Consolidates the full optimizer state on rank 0 and returns it as a :class:`dict` following the convention of :meth:`torch.optim.Optimizer.state_dict`, i.e. with keys ``"state"`` and ``"param_groups"``. The flattened parameters in ``FSDP`` modules contained in ``model`` are mapped back to their unflattened parameters. This needs to be called on all ranks since it uses collective communications. However, if ``rank0_only=True``, then the state dict is only populated on rank 0, and all other ranks return an empty :class:`dict`. Unlike ``torch.optim.Optimizer.state_dict()``, this method uses full parameter names as keys instead of parameter IDs. Like in :meth:`torch.optim.Optimizer.state_dict`, the tensors contained in the optimizer state dict are not cloned, so there may be aliasing surprises. For best practices, consider saving the returned optimizer state dict immediately, e.g. using ``torch.save()``. Args: model (torch.nn.Module): Root module (which may or may not be a :class:`FullyShardedDataParallel` instance) whose parameters were passed into the optimizer ``optim``. optim (torch.optim.Optimizer): Optimizer for ``model`` 's parameters. optim_input (Optional[Union[List[Dict[str, Any]], Iterable[torch.nn.Parameter]]]): Input passed into the optimizer ``optim`` representing either a :class:`list` of parameter groups or an iterable of parameters; if ``None``, then this method assumes the input was ``model.parameters()``. This argument is deprecated, and there is no need to pass it in anymore. (Default: ``None``) rank0_only (bool): If ``True``, saves the populated :class:`dict` only on rank 0; if ``False``, saves it on all ranks. (Default: ``True``) group (dist.ProcessGroup): Model's process group or ``None`` if using the default process group. (Default: ``None``) Returns: Dict[str, Any]: A :class:`dict` containing the optimizer state for ``model`` 's original unflattened parameters and including keys "state" and "param_groups" following the convention of :meth:`torch.optim.Optimizer.state_dict`. If ``rank0_only=True``, then nonzero ranks return an empty :class:`dict`. full_optim_state_dictrPrqrrT)rOrHrPrErrrQrNrRrMr[ state_dict)rOrHrErrs r^rcz.FullyShardedDataParallel.full_optim_state_dictVs\x !>> #  ? (>>"--/#! ?  r]c tjdddtj|||jddd|dS)a`Return the optimizer state-dict in its sharded form. The API is similar to :meth:`full_optim_state_dict` but this API chunks all non-zero-dimension states to :class:`ShardedTensor` to save memory. This API should only be used when the model ``state_dict`` is derived with the context manager ``with state_dict_type(SHARDED_STATE_DICT):``. For the detailed usage, refer to :meth:`full_optim_state_dict`. .. warning:: The returned state dict contains ``ShardedTensor`` and cannot be directly used by the regular ``optim.load_state_dict``. sharded_optim_state_dictrPrqrrNF)rOrHrPrErrQrrNrd)rOrHrs r^rgz1FullyShardedDataParallel.sharded_optim_state_dicts[$ !>> &  ? (>>"--/!?  r]rcchtjdddtj||||ddS)a Shard a full optimizer state-dict. Remaps the state in ``full_optim_state_dict`` to flattened parameters instead of unflattened parameters and restricts to only this rank's part of the optimizer state. The first argument should be the return value of :meth:`full_optim_state_dict`. Example:: >>> # xdoctest: +SKIP("undefined variables") >>> from torch.distributed.fsdp import FullyShardedDataParallel as FSDP >>> model, optim = ... >>> full_osd = FSDP.full_optim_state_dict(model, optim) >>> torch.save(full_osd, PATH) >>> # Define new model with possibly different world size >>> new_model, new_optim = ... >>> full_osd = torch.load(PATH) >>> sharded_osd = FSDP.shard_full_optim_state_dict(full_osd, new_model) >>> new_optim.load_state_dict(sharded_osd) .. note:: Both :meth:`shard_full_optim_state_dict` and :meth:`scatter_full_optim_state_dict` may be used to get the sharded optimizer state dict to load. Assuming that the full optimizer state dict resides in CPU memory, the former requires each rank to have the full dict in CPU memory, where each rank individually shards the dict without any communication, while the latter requires only rank 0 to have the full dict in CPU memory, where rank 0 moves each shard to GPU memory (for NCCL) and communicates it to ranks appropriately. Hence, the former has higher aggregate CPU memory cost, while the latter has higher communication cost. Args: full_optim_state_dict (Dict[str, Any]): Optimizer state dict corresponding to the unflattened parameters and holding the full non-sharded optimizer state. model (torch.nn.Module): Root module (which may or may not be a :class:`FullyShardedDataParallel` instance) whose parameters correspond to the optimizer state in ``full_optim_state_dict``. optim_input (Optional[Union[List[Dict[str, Any]], Iterable[torch.nn.Parameter]]]): Input passed into the optimizer representing either a :class:`list` of parameter groups or an iterable of parameters; if ``None``, then this method assumes the input was ``model.parameters()``. This argument is deprecated, and there is no need to pass it in anymore. (Default: ``None``) optim (Optional[torch.optim.Optimizer]): Optimizer that will load the state dict returned by this method. This is the preferred argument to use over ``optim_input``. (Default: ``None``) Returns: Dict[str, Any]: The full optimizer state dict now remapped to flattened parameters instead of unflattened parameters and restricted to only include this rank's part of the optimizer state. shard_full_optim_state_dictoptim_state_dict_to_loadrqrrTFrPrOrErHrQr\rRrMra)rcrOrErHs r^riz4FullyShardedDataParallel.shard_full_optim_state_dictsNB !>> ) & ? (FF2# $ G  r]rgchtjdddtj||d|ddS)a@Flatten a sharded optimizer state-dict. The API is similar to :meth:`shard_full_optim_state_dict`. The only difference is that the input ``sharded_optim_state_dict`` should be returned from :meth:`sharded_optim_state_dict`. Therefore, there will be all-gather calls on each rank to gather ``ShardedTensor`` s. Args: sharded_optim_state_dict (Dict[str, Any]): Optimizer state dict corresponding to the unflattened parameters and holding the sharded optimizer state. model (torch.nn.Module): Refer to :meth:`shard_full_optim_state_dict`. optim (torch.optim.Optimizer): Optimizer for ``model`` 's parameters. Returns: Refer to :meth:`shard_full_optim_state_dict`. flatten_sharded_optim_state_dictrjrqrrNFrkrl)rgrOrHs r^rnz9FullyShardedDataParallel.flatten_sharded_optim_state_dictsM2 !>> . & ? (FF5!$ G  r]c ltjdddtj||||ddd|S)a Scatter the full optimizer state dict from rank 0 to all other ranks. Returns the sharded optimizer state dict on each rank. The return value is the same as :meth:`shard_full_optim_state_dict`, and on rank 0, the first argument should be the return value of :meth:`full_optim_state_dict`. Example:: >>> # xdoctest: +SKIP("undefined variables") >>> from torch.distributed.fsdp import FullyShardedDataParallel as FSDP >>> model, optim = ... >>> full_osd = FSDP.full_optim_state_dict(model, optim) # only non-empty on rank 0 >>> # Define new model with possibly different world size >>> new_model, new_optim, new_group = ... >>> sharded_osd = FSDP.scatter_full_optim_state_dict(full_osd, new_model, group=new_group) >>> new_optim.load_state_dict(sharded_osd) .. note:: Both :meth:`shard_full_optim_state_dict` and :meth:`scatter_full_optim_state_dict` may be used to get the sharded optimizer state dict to load. Assuming that the full optimizer state dict resides in CPU memory, the former requires each rank to have the full dict in CPU memory, where each rank individually shards the dict without any communication, while the latter requires only rank 0 to have the full dict in CPU memory, where rank 0 moves each shard to GPU memory (for NCCL) and communicates it to ranks appropriately. Hence, the former has higher aggregate CPU memory cost, while the latter has higher communication cost. Args: full_optim_state_dict (Optional[Dict[str, Any]]): Optimizer state dict corresponding to the unflattened parameters and holding the full non-sharded optimizer state if on rank 0; the argument is ignored on nonzero ranks. model (torch.nn.Module): Root module (which may or may not be a :class:`FullyShardedDataParallel` instance) whose parameters correspond to the optimizer state in ``full_optim_state_dict``. optim_input (Optional[Union[List[Dict[str, Any]], Iterable[torch.nn.Parameter]]]): Input passed into the optimizer representing either a :class:`list` of parameter groups or an iterable of parameters; if ``None``, then this method assumes the input was ``model.parameters()``. This argument is deprecated, and there is no need to pass it in anymore. (Default: ``None``) optim (Optional[torch.optim.Optimizer]): Optimizer that will load the state dict returned by this method. This is the preferred argument to use over ``optim_input``. (Default: ``None``) group (dist.ProcessGroup): Model's process group or ``None`` if using the default process group. (Default: ``None``) Returns: Dict[str, Any]: The full optimizer state dict now remapped to flattened parameters instead of unflattened parameters and restricted to only include this rank's part of the optimizer state. scatter_full_optim_state_dictrjrqrrTFrPrOrErHrQrr\rrl)rcrOrErHrs r^rpz6FullyShardedDataParallel.scatter_full_optim_state_dict:sTH !>> + & ? (FF2# $G  r]optim_state_key_typectj|tj||}|tjtj fvsJ|}|dDcgc]}t |tu}}|dDcgc]}t |tu} }t|r t|rt| r,t| s!d|dj} t| |tjk(r t|s|tj k(r t| r|Si} |tjk(r|r t||n t|} t|} | j!Dcgc]}| | }}|dj#Dcic] \}}||| c}}| d<t%j&|d| d<| dD]$}t)|dDcgc]}|| c}|d<&| S|tj k(rt+|}|r t-||n t/|}|j#Dcic]\}}||vr|||}}}|dj#Dcic] \}}||| c}}| d<t%j&|d| d<| dD]$}t)|dDcgc]}|| c}|d<&| S| Scc}wcc}wcc}wcc}}wcc}wcc}}wcc}}wcc}w)a?Re-keys the optimizer state dict ``optim_state_dict`` to use the key type ``optim_state_key_type``. This can be used to achieve compatibility between optimizer state dicts from models with FSDP instances and ones without. To re-key an FSDP full optimizer state dict (i.e. from :meth:`full_optim_state_dict`) to use parameter IDs and be loadable to a non-wrapped model:: >>> # xdoctest: +SKIP("undefined variables") >>> wrapped_model, wrapped_optim = ... >>> full_osd = FSDP.full_optim_state_dict(wrapped_model, wrapped_optim) >>> nonwrapped_model, nonwrapped_optim = ... >>> rekeyed_osd = FSDP.rekey_optim_state_dict(full_osd, OptimStateKeyType.PARAM_ID, nonwrapped_model) >>> nonwrapped_optim.load_state_dict(rekeyed_osd) To re-key a normal optimizer state dict from a non-wrapped model to be loadable to a wrapped model:: >>> # xdoctest: +SKIP("undefined variables") >>> nonwrapped_model, nonwrapped_optim = ... >>> osd = nonwrapped_optim.state_dict() >>> rekeyed_osd = FSDP.rekey_optim_state_dict(osd, OptimStateKeyType.PARAM_NAME, nonwrapped_model) >>> wrapped_model, wrapped_optim = ... >>> sharded_osd = FSDP.shard_full_optim_state_dict(rekeyed_osd, wrapped_model) >>> wrapped_optim.load_state_dict(sharded_osd) Returns: Dict[str, Any]: The optimizer state dict re-keyed using the parameter keys specified by ``optim_state_key_type``. rzInvalid parameter keys: param_groupsparams)rRrFrIrSrZr[rstrintanyrkeysrrCrD_get_param_to_fqnvaluesitemscopydeepcopysorted_get_fqn_to_paramrErF)rPrrrOrErHrZosd param_keyuses_param_name_maskuses_param_id_mask error_msgnew_osdparam_id_to_paramparam_to_param_namerparam_id_to_param_nameparam_id param_state param_groupparam_name_to_paramparam_to_param_idrparam_name_to_param_ids r^rekey_optim_state_dictz/FullyShardedDataParallel.rekey_optim_state_dictskX !22;?4JJ   $  ( (  & &(    HKG U9Y3 6UUFI'lSd9o4SS $ %c2F.G " #C0B,C23w<3D3D3F2GHIY' ' !$5$@$@ @() $5$>$> >&'J #4#?#? ?%8{K,U3  #4E": 8I8P8P8R1/4#E*1 "1 .1\-?-?-A )Hk'x0+= GG '+mmC4G&HGN #&~6  (.)4H(=$/x8) H% N !%6%?%? ?"3E": %8{K,U3 *=)B)B)D&%J---e44& "&037|/A/A/C +J 'z2K? GG '+mmC4G&HGN #&~6  (.+6h*?&/z:) H% NI VS01 &  s0K:K * KK K 8K!%K'+ K- ctj|}||j}tj|||dt |j dd|j tjk(|t |j ddd S)a Transform the state-dict of an optimizer corresponding to a sharded model. The given state-dict can be transformed to one of three types: 1) full optimizer state_dict, 2) sharded optimizer state_dict, 3) local optimizer state_dict. For full optimizer state_dict, all states are unflattened and not sharded. Rank0 only and CPU only can be specified via :meth:`state_dict_type` to avoid OOM. For sharded optimizer state_dict, all states are unflattened but sharded. CPU only can be specified via :meth:`state_dict_type` to further save memory. For local state_dict, no transformation will be performed. But a state will be converted from nn.Tensor to ShardedTensor to represent its sharding nature (this is not supported yet). Example:: >>> # xdoctest: +SKIP("undefined variables") >>> from torch.distributed.fsdp import FullyShardedDataParallel as FSDP >>> from torch.distributed.fsdp import StateDictType >>> from torch.distributed.fsdp import FullStateDictConfig >>> from torch.distributed.fsdp import FullOptimStateDictConfig >>> # Save a checkpoint >>> model, optim = ... >>> FSDP.set_state_dict_type( >>> model, >>> StateDictType.FULL_STATE_DICT, >>> FullStateDictConfig(rank0_only=False), >>> FullOptimStateDictConfig(rank0_only=False), >>> ) >>> state_dict = model.state_dict() >>> optim_state_dict = FSDP.optim_state_dict(model, optim) >>> save_a_checkpoint(state_dict, optim_state_dict) >>> # Load a checkpoint >>> model, optim = ... >>> state_dict, optim_state_dict = load_a_checkpoint() >>> FSDP.set_state_dict_type( >>> model, >>> StateDictType.FULL_STATE_DICT, >>> FullStateDictConfig(rank0_only=False), >>> FullOptimStateDictConfig(rank0_only=False), >>> ) >>> model.load_state_dict(state_dict) >>> optim_state_dict = FSDP.optim_state_dict_to_load( >>> model, optim, optim_state_dict >>> ) >>> optim.load_state_dict(optim_state_dict) Args: model (torch.nn.Module): Root module (which may or may not be a :class:`FullyShardedDataParallel` instance) whose parameters were passed into the optimizer ``optim``. optim (torch.optim.Optimizer): Optimizer for ``model`` 's parameters. optim_state_dict (Dict[str, Any]): the target optimizer state_dict to transform. If the value is None, optim.state_dict() will be used. ( Default: ``None``) group (dist.ProcessGroup): Model's process group across which parameters are sharded or ``None`` if using the default process group. ( Default: ``None``) Returns: Dict[str, Any]: A :class:`dict` containing the optimizer state for ``model``. The sharding of the optimizer state is based on ``state_dict_type``. NrFrTrq) rOrHrPrErrQrrcrN) rRrrer[rrrr<r)rOrHrPrrs r^rPz)FullyShardedDataParallel.optim_state_dict sX7JJ5Q  #$//1 '>>-#;;\50??,,-#;;=Mt?  r] load_directlyc tj|}tj||d||jtj k(t |jdd||}|r|j||S)a| Convert an optimizer state-dict so that it can be loaded into the optimizer associated with the FSDP model. Given a ``optim_state_dict`` that is transformed through :meth:`optim_state_dict`, it gets converted to the flattened optimizer state_dict that can be loaded to ``optim`` which is the optimizer for ``model``. ``model`` must be sharded by FullyShardedDataParallel. >>> # xdoctest: +SKIP("undefined variables") >>> from torch.distributed.fsdp import FullyShardedDataParallel as FSDP >>> from torch.distributed.fsdp import StateDictType >>> from torch.distributed.fsdp import FullStateDictConfig >>> from torch.distributed.fsdp import FullOptimStateDictConfig >>> # Save a checkpoint >>> model, optim = ... >>> FSDP.set_state_dict_type( >>> model, >>> StateDictType.FULL_STATE_DICT, >>> FullStateDictConfig(rank0_only=False), >>> FullOptimStateDictConfig(rank0_only=False), >>> ) >>> state_dict = model.state_dict() >>> original_osd = optim.state_dict() >>> optim_state_dict = FSDP.optim_state_dict( >>> model, >>> optim, >>> optim_state_dict=original_osd >>> ) >>> save_a_checkpoint(state_dict, optim_state_dict) >>> # Load a checkpoint >>> model, optim = ... >>> state_dict, optim_state_dict = load_a_checkpoint() >>> FSDP.set_state_dict_type( >>> model, >>> StateDictType.FULL_STATE_DICT, >>> FullStateDictConfig(rank0_only=False), >>> FullOptimStateDictConfig(rank0_only=False), >>> ) >>> model.load_state_dict(state_dict) >>> optim_state_dict = FSDP.optim_state_dict_to_load( >>> model, optim, optim_state_dict >>> ) >>> optim.load_state_dict(optim_state_dict) Args: model (torch.nn.Module): Root module (which may or may not be a :class:`FullyShardedDataParallel` instance) whose parameters were passed into the optimizer ``optim``. optim (torch.optim.Optimizer): Optimizer for ``model`` 's parameters. optim_state_dict (Dict[str, Any]): The optimizer states to be loaded. is_named_optimizer (bool): Is this optimizer a NamedOptimizer or KeyedOptimizer. Only set to True if ``optim`` is TorchRec's KeyedOptimizer or torch.distributed's NamedOptimizer. load_directly (bool): If this is set to True, this API will also call optim.load_state_dict(result) before returning the result. Otherwise, users are responsible to call ``optim.load_state_dict()`` (Default: ``False``) group (dist.ProcessGroup): Model's process group across which parameters are sharded or ``None`` if using the default process group. ( Default: ``None``) NrFrq) rRrrarr<rrrload_state_dict)rOrHrPr\rrrresults r^rjz1FullyShardedDataParallel.optim_state_dict_to_loadksN7JJ5Q)HH-#33}7T7TT#;;\5 2I     ! !& ) r]hookc>|js tdtj|D]j}|jt vrtd|j|j tdt|std|||_||_ ly)aM Register a communication hook. This is an enhancement that provides a flexible hook to users where they can specify how FSDP aggregates gradients across multiple workers. This hook can be used to implement several algorithms like `GossipGrad `_ and gradient compression which involve different communication strategies for parameter syncs while training with :class:`FullyShardedDataParallel`. .. warning :: FSDP communication hook should be registered before running an initial forward pass and only once. Args: state (object): Passed to the hook to maintain any state information during the training process. Examples include error feedback in gradient compression, peers to communicate with next in `GossipGrad `_, etc. It is locally stored by each worker and shared by all the gradient tensors on the worker. hook (Callable): Callable, which has one of the following signatures: 1) ``hook: Callable[torch.Tensor] -> None``: This function takes in a Python tensor, which represents the full, flattened, unsharded gradient with respect to all variables corresponding to the model this FSDP unit is wrapping (that are not wrapped by other FSDP sub-units). It then performs all necessary processing and returns ``None``; 2) ``hook: Callable[torch.Tensor, torch.Tensor] -> None``: This function takes in two Python tensors, the first one represents the full, flattened, unsharded gradient with respect to all variables corresponding to the model this FSDP unit is wrapping (that are not wrapped by other FSDP sub-units). The latter represents a pre-sized tensor to store a chunk of a sharded gradient after reduction. In both cases, callable performs all necessary processing and returns ``None``. Callables with signature 1 are expected to handle gradient communication for a `NO_SHARD` case. Callables with signature 2 are expected to handle gradient communication for sharded cases. z9register_comm_hook can only be called on a root instance.z;Communication hook is not supported for hybrid strategies: Nz*A communication hook is already registeredz0The communication hook must be callable but got ) rAssertionErrorrrrbr"rcallabler_comm_hook_state)rrr fsdp_states r^register_comm_hookz+FullyShardedDataParallel.register_comm_hooksN!!# K *::4@ 0J++/II$QR\RnRnQop$$0$%QRRD> FtfM%)J !*/J ' 0r]async_opcGdd}|jr|jtjtj 5t ||j|j|j|jj|_ dddd|j_ ||j|j}|r|S|jy#1swYLxYw)Nc<eZdZdeedej fdZdZy)8FullyShardedDataParallel._unshard..UnshardHandleflat_param_handle unshard_eventc ||_||_yr)_flat_param_handle_unshard_event)rrrs r^rxzAFullyShardedDataParallel._unshard..UnshardHandle.__init__s +<'&3#r]c|jG|jjj}|j|jd|_yyr)r_device_handlecurrent_stream wait_eventr)rrs r^waitz=FullyShardedDataParallel._unshard..UnshardHandle.waitsN**6//>>MMO##--d.A.AB.2D+ 7r]N) rVrWrXr rArtEventrxrr\r]r^ UnshardHandlers' 4#+O#< 4 %{{ 4 3r]rT) r_use_training_staterFORWARD_BACKWARDrFORWARDr,_unshard_stream_pre_unshard_stream record_eventr _prefetchedr)rrrunshard_handles r^r,z!FullyShardedDataParallel._unshards 3 3" <<))..0C0K0K J$,,(<(V>V'+&:&:&G&G&I#  J(,DLL $&t||T5I5IJ ! ! J Js A CC%cvt|jj|j|jyr)r-rrrrrs r^'_wait_unshard_streams_on_current_streamz@FullyShardedDataParallel._wait_unshard_streams_on_current_streams.$    . . 0   $ $ r]rhandle_training_statec#@K|j}||_|jr'|jj}||j_ d||_|jr|j_yy#||_|jr|j_wwxYwwr)rr_training_state)rrrprev_training_stateprev_handle_training_states r^rz,FullyShardedDataParallel._use_training_state&s#11, <<)-)E)E &+@DLL ( J "5D ||/I ,#6D ||/I ,sAB A4&B4'BB)F)rN)NN)TTFFF)g@)NTTNT)NNTFFN)NTNr)NNN)FFN)^rVrWrXrYr/ BACKWARD_PRErzModuler#r r9r0r r rQrPr5rrtrwrbool Parameterr=rxpropertyr`rr@rTrvr rrr staticmethodlistrrrrrr<r:r6r;rrrrrrrrrrrtupleTensorrrrrr no_gradrrrFrIrMrH Optimizerdictr% ProcessGroupr[rarcrgrirnrprSrrPrjobjectrrr,rrr __classcell__)rs@r^rRrRus Y|+/8<,0 8H8U8U48?C?C8<#(!&"& % ,0+r! r!(r!$$45 r! j) r! # (,l: ; r!$$45r!".1r!"(588??";<r! "))d): ;<r!E#u||"345r!!r!r!  !r!"#r!$ Xehh001 2HXehhoo=V4W W %r!*j)+r!h) ))ETEEAXm4AA<<<(s(s( )t) 8 88 ( )88,&"))d!23&8R&P=d=VTV-8!? v  v v p!#BII!#2C!#!#F8!? )  ) ) VSCC6 $ G GGG G  G  G GGR@@(3 ( %U\\)* + ($& %UXX///0 1 &$"5]8K)K#L"QU"($-$-$-LU]]_JMS/eSj)S/6;E3J6GS/ S/S/j<=cTPQ C c #    $-1 6 6 xx6 {{$$6 sCx.6  T#s(^$++,.  6 6 6 ))*6 6 6 c3h!6 6 p 15 $ #(-19 sCx.9 xx9  T#s(^$++,.  9  --.9 9 9 !9 ))*9  c3h9 9 v -1I xxI {{$$I  T#s(^$++,.  I I ))*I  c3hI I V.2 xx {{$$ ))*  c3h   B 15L #CH~L xxL  T#s(^$++,.  L  --.L  c3hL L \$ "&sCx.$ xx$ {{$$$  c3h $ $ L 15#Q 'S#X7Q xxQ  T#s(^$++,.  Q  --.Q }Q  c3hQ Q f 15zsCx.z/zxxz T#s(^$++,.  z --.z c3hzzx6:-1 ] xx] {{$$] #4S>2] ))* ] c3h ] ] ~ $)#-1 WxxW{{$$WsCx.W! W  W ))* W c3hWWr7070h70rB  J+ JDW J Jr]rur zerorrc \|Dcgc]}|j|}}t|dk(r|S|Dcgc]}|j}}|Dchc]}|j}}t|dk7rtd|tj j t j|Dcgc]@}tj j |j|tjBc}|tj} | j|Scc}wcc}wcc}wcc}w)a  Return the gradient norm of parameters ``param`` s, where the gradients are viewed as a single vector. The returned norm is in FP32 even if parameters/gradients are in a low precision. This is because the downstream use of this return value is a reduction across ranks. rr?z4Requires uniform dtype across all gradients but got )r/r) r r0r/rrtlinalg vector_normstackdetachfloat32r.) rur rrrparams_with_gradr9r  grad_dtypes grad_norms r^r!r!7s,2L%UZZ5KLL ! %5 6EUZZ 6E 6*/0$4::0K0 ;1B;- P   (( "  (( (W  mm) I <rTr@rA _optim_utilsrBrCrDrErFrGrHrI_state_dict_utilsrJ_unshard_param_utilsrKrLrMrNrOwraprPrQ__all__ FLAT_PARAMrSrrRrrrrr!rrvrzrr\r]r^rs  99%11 AAIN   : 0-7   >1   Jryy*JD>"' R\\ ""'"' ,,"' LL "'  \\ "'J 88?? %((  c !"@O 88??O #uxx!! !"Or]