Vh- ddlZddlZddlmZddlmZddlZddlmcm cm Z ddl m Z ddlmZmZmZmZmZmZddlmZmZmZmZmZmZddlmZddlmZd Z ejBd ed e"fd Z#d ede jHddfdZ%d ede jHddfdZ&ejNd ede jHdefdZ(d ede"de"de"de"ddf dZ)ejNde jHd ede"de"de"de"f dZ*ejNde jHd ede"de"de"de"f dZ+ejNde jHde"de"de"de"de"f dZ,d ede jHddfdZ-d ede jHddfdZ.y)N) Generator)cast) _FSDPState_get_module_fsdp_state_has_fsdp_params_module_handleHandleTrainingState TrainingState) _lazy_init%_reset_flat_param_grad_info_if_needed_reshard_reshard_grads_unshard_unshard_grads) _p_assert)FlatParamHandle _flat_paramhandlewriteback_gradcdtjdtjffd }|j}jjd|j j ||rfj }|WjjJ|jj}|d|j j |yyy)aP For the handle, writes back the this rank's shard of the unsharded flattened parameter to the sharded flattened parameter. If ``writeback_grad=True``, then writes back to the sharded gradient as well. Precondition: The handle's ``FlatParameter`` 's data points to the padded unsharded flattened parameter. flat_param_or_gradreturncjr0tj|jj\}}|S|S)N)uses_sharded_strategyr_get_unpadded_shardrank world_size)rshard_rs [/home/dcms/DCMS/lib/python3.12/site-packages/torch/distributed/fsdp/_unshard_param_utils.py _get_shardz-_writeback_to_local_shard.._get_shard1sD  ' ''::" !!HE1 L"!N)torchTensor flat_param _local_shardnumelcopy_ sharded_gradgrad)rrr" param_shard existing_grad grad_shards` r!_writeback_to_local_shardr/"s "u|| " "V../K ""#8[%6%6%89?? L++  $$$))5 55#F$5$5$:$:;J .J,,. / 5 5j A %r#statemodulerct||rCttj|jj j tdyy)a De-registers the flattened parameter from the wrapped module, hiding it from ``nn.Module`` methods. We do not use ``del`` because we want ``FLAT_PARAM`` to always be an attribute but dynamically change whether it is visible to ``nn.Module`` methods. N)rrnnModuler1 _parameterspop FLAT_PARAM)r0r1s r!_deregister_flat_paramr8Js8v& RYY &2266z4H'r#ct||}t||r@|jttj |j jt<yy)a2 Registers the flattened parameter to the wrapped module, making it visible to ``nn.Module`` methods. We do not use :meth:`nn.Module.register_parameter` because we want ``FLAT_PARAM`` to always be an attribute but dynamically change whether it is visible to ``nn.Module`` methods. N) rrr&rr3r4r1r5r7r0r1rs r!_register_flat_paramr;XsDE6 *Fv&AGARARRYY &22:>'r#c# Kt||}|sdyt|| |j5dddd|js t ||yy#1swY#xYw#|js t ||wwxYww)ax Assumes that the flattened parameter is unsharded. When in the context, de-registers the flattened parameter and unflattens the original parameters as ``nn.Parameter`` views into the flattened parameter. After the context, re-registers the flattened parameter and restores the original parameters as ``Tensor`` views into the flattened parameter. N)rr8unflatten_as_params_use_orig_paramsr;r:s r!_unflatten_as_paramsr?gsE6 *F  uf- 4++-  **$UF3+  **$UF3+s2 BA&AA&BA#A&&BB writeback rank0_onlyoffload_to_cpu with_gradsc |r-|s |jstd|d|jd|d|r-|jr!|jjs td|r |r td|r|st j dyyy)Nz with_grads=z, use_orig_params=z, offload_to_cpu=z is not supported yetz5offload_to_cpu=True and NO_SHARD is not supported yetz7writeback=True and rank0_only=True is not supported yetzoffload_to_cpu=True and rank0_only=False may result in theunsharded parameters being redundantly copied to CPU memory for GPUs sharing the same CPU memory, which risks CPU OOM. We recommend using offload_to_cpu=True with rank0_only=True.)r>NotImplementedError_handlerwarningswarn)r0r@rArBrCs r!_validate_unshard_params_argsrI~s~U-C-C!*&$5567,-.# $  %--1T1T! C  Z" E  j  H )~r#c#Kt||||||jjt||}d}|r|jt j k7r|}|sdy|jt jk(sJd|jt j |_t||j}|jj} t||| | |r t||rD|jdk7r5t||||r t| dt j|_yt!j"5} |r+|j$r| j'|j)|j*s| j't-|| d| j/|r t1||t||||r t|t j|_ dddy#t j|_wxYw#| j/|r t1||t||||r t|t j|_wxYw#1swYyxYww)zl This unshards the parameters for a single FSDP state ``state`` that corresponds to ``module``. Nz/Expects the handle training to be IDLE but got r)rI_device_handle synchronizer_training_stater SUMMON_FULL_PARAMSIDLEr needs_unshardcurrent_streamrrrr r contextlib ExitStackr enter_contextto_cpur>r?closer/) r1r0r@rArBrC maybe_handlerfree_unsharded_flat_paramcomputation_streamstacks r!_unshard_fsdp_state_paramsr[s1" y*nj $$&!%0L F  ( (,?,R,R R    ! !%8%=%= = 9&:P:P9QR =1CCF)&1 & 4 4 6--<<> UF.0BCvejjAo 9:  6 " > %8%=%=F " ! ! # Bu&">">##FMMO4 ))##$8$GH B -fjA(AB"6*)<)A)A&' B B&9%=%=F "  -fjA(AB"6*)<)A)A&' B BsQD!I;$H(*I;AI/(H,A I/: I;HI;AI,,I//I84I;c#Kt|||||t|||jtjk(r t d|jtj k(r t dt||||||5 tj |_dtj|_ dddy#tj|_wxYw#1swYyxYww)Nz:Cannot manually unshard parameters during forward/backwardzECannot manually unshard parameters when already unsharding parametersr1r0r@rArBrC) rIr training_stater FORWARD_BACKWARDAssertionErrorrNr[rOr]s r!_unshard_params_for_summonras" y*njuf }=== H     !A!A A S   $%   6 6#0#C#CE #0#5#5E  6 6$1#5#5E  6 6s6A=C?CB9C0 C9CCCCrecursec #xK|s6t|}|"tj5ddddy|g|gf}ntj|}tj 5}t |D]%\} }|jt|| ||||'ddddy#1swYyxYw#1swYyxYww)z~ This unshards FSDP-managed parameters for all modules with FSDP applied in the module tree rooted at ``module``. Nr]) rrR nullcontexttraversal_utils_get_fsdp_states_with_modulesrSziprTra) r1rbr@rArBrCoptional_statestates_and_modulesrZr0s r!_unshard_paramsrjs /7  !'')   -.9,JJ6R     5 "45 ME6   *!')#1)           s3$B:B"9B:$5B. B:"B+'B:.B73B:ct||}|syt|jd|jd|j|jt ||y)zO Deregisters the original parameters; registers the ``FlatParameter``. Nz)Inconsistent `_use_orig_params` -- FSDP: z handle: )rrr>_deregister_orig_paramsr;r:s r!rlrl5sdE6 *F   3E4J4J3KL**+ -  ""$'r#ct||}|syt|||j|jr!|j |j y|j dy)zO Deregisters the ``FlatParameter``; registers the original parameters. NT) as_params)rr8 is_shardedr&_use_sharded_views_use_sharded_grad_views_use_unsharded_viewsr:s r!_register_orig_paramsrsEs]E6 *F 5&) **+!!#&&(##d#3r#)/rRrGcollections.abcrtypingrr$'torch.distributed.fsdp._traversal_utils distributedfsdp_traversal_utilsretorch.nnr3$torch.distributed.fsdp._common_utilsrrrrr r %torch.distributed.fsdp._runtime_utilsr r r rrrtorch.distributed.utilsrrrr7no_gradboolr/r4r8r;contextmanagerr?rIr[rarjrlrsr#r!rsw% AA.( $B $B$B$BN I* Ibii ID I S SBII S$ S 4 4BII4)44,          B IB IIIB IBIB IB  IB  IBIBX  6 II 6  6 6 6  6  6 6F ! II! !! !  !  !!H (: (ryy (T ( 4 4RYY 44 4r#