Vhd5 ddlZddlmZmZmZddlZddlmZddl mcm cm Z ddl mcmZddlmZmZmZmZddl mZddlmZddlmZddlmZddlmZdd lmZdd l m!Z!dd l"m#Z#m$Z$m%Z%mZ&dd l'm(Z(m)Z)d gZ*de$de+ejXejXffdZ-de$de.de+ejXejXffdZ/de$de+ejXejXffdZ0de$de.defdZ1de$dejddefdZ3de$dejdfdZ4de jjdejlde.de jjfdZ7dejlde.de.de.dejddejlf d Z8dejlde.d!e#de$fd"Z9dejlde+ejle:effd#Z;de$d$ee#dejlfd%Z<Gd&d eZ=y)'N)AnycastOptional)Shard ShardedTensorShardedTensorMetadataTensorProperties) ShardMetadata)ChunkShardingSpec)_mesh_resources)_set_fsdp_flattened)FSDPExtensions)_create_chunk_sharded_tensor)_remote_device) DeviceMeshDTensor Replicater)_flatten_tensor_unflatten_tensorDTensorExtensionstensorreturnc|j}|jdk(sJd|jd}dgt|j z}|j d}|jdj r3t t|j}|j ||z}|||<tj||jj fS)N&Only 1D DeviceMeshes currently handledr)mesh_dim) device_meshndim placementslensizeis_shardrDSharddimtorchSize _local_tensor)rr placementoffsets num_chunks shard_dim chunk_sizes V/home/dcms/DCMS/lib/python3.12/site-packages/torch/distributed/tensor/parallel/fsdp.py_get_boxr. s$$K   q J"JJ !!!$IcC &&G!!1!-J $$&+// [[+z9 '  JJw !5!5!:!:!< ==idxcxt|\}}tj|Dcgc]}||z c}|fScc}wN)r.r%r&)rr0r)r!vals r- _get_box_forr40s6V$MGT JJW5cc 5 6 ==5s 7c`|j}|j}|Jt||dS)Nr)rget_coordinater4)rrcoords r-_get_local_boxr85s8$$K  & & (E    a ))r/dt current_rankc|j}|jdk(sJdt|\}}tt |t |d|d|j j S)Nrrrank:/ shard_offsets shard_sizesr()rrr8r listr'device)r9r:meshr)sizess r-_create_shard_md_from_dtrE<sg >>D 99>CCC>#B'NGU 7mK,q)9)9)@)@(AB r/dt_pgc  g}tj|}|dkDrdnd}|jdjr|j }nd}t |D]a}t ||\}}|jtt|t|d|dkDr|n|d|jjct||j t|j|j|j S)Nrrr<r=r>)dtypelayout requires_grad)shards_metadatar!tensor_properties)distget_rankrr"r!ranger4appendr rAr'rBrr rHrIrJ) r9rF shards_mdmy_rankscapegoat_rank shard_countir)rDs r-!_create_sharded_tensor_md_from_dtrVHs ImmE"G!A+Q1N }}Q  "jjl  ;   %b!, "7m Ka!eNA2CSCSCZCZB[\      !! WWY*((99**   r/cf|j}|jdk(sJd|jS)Nrr)rr get_group)r9rCs r- _get_dt_pgrYos. >>D 99>CCC> >> r/specrankct|ts|Sd}|jD]G}tt|}|j |k(s'|j |j k7sEd}n|rtj|}t|jD]o\}}tt|}|j |k(s*|j |j k7sHt d|d|j |j|<q|S)z Rewrite ``spec`` to match the device of ``tensor``. FSDP.sharded_optim_state_dict sneakly ships optimizer state to CPU so if the original ShardingSpec produces CUDA metadata, ST construction bombs. FTr<r=) isinstancer rrrr[rBcopydeepcopy enumerate)rZrr[rewriteprUr(s r-_rewrite_spec_if_neededrcus d- . G __  # 668t  fmm ;G   }}T"%doo6 TLAy^Y7I~~4'I,<,<,>&--,O%3eD66==/4R%S" T Kr/ world_sizenum_devices_per_nodepgc t|turt|jdk(sJ|j }t |||||}|jd}t |tj|jg}tj|j} d| j_ tj|| |jd} | St|tur|j} | j dk(sJd|j"}t |||t$j&j)|}t+|} t |t-|t/j0| g}t3|| } d| j_ tj|| | d} | St |||||S)NrrF)sharded_tensor_metadata process_group init_rrefsr)typerr local_shards local_tensorrrr^r_metadatarLrJ+_init_from_local_shards_and_global_metadata_process_grouprrrr'r% accelerator device_countrYrErMrNrV) rr[rdrerf inner_paraminner_stouter_local_shardshardsst_metast_outerrrFs r- _chunk_tensorrys F|}$6&&()Q...))+ /      #//1!4 (DMM*;*D*DE F -- 1227!!/ LL $+ //    f (( 1$N&NN$** /       * * ,   6" (4VT]]5=QR S 4FEB27!!/ LL $+  +       r/rctj|}| td|jdkrtd|jdd|j j }t |tjrt |tst|jDcgc] }t}}t|jDcgc] }t}}td|d<tj|||dj|| S|j}|d}|j!}t|jDcgc] }t}}||d <t|jD cgc] } t}} td|d <||d <tj|||dj|| Scc}wcc}wcc}wcc} w) z Shard a tensor to chunks along the first dimension. The local rank will gets its corresponding chunk as the local tensor to create a DTensor. z4No parent device_mesh is found for FSDP device_mesh.z!Found parent device_mesh of ndim=,zbut meshes must be at least 2D.rF) run_checkrr)r get_root_mesh RuntimeErrorrdetachcloner]r%TensorrrOrr# from_local redistributerto_local) rr[r root_mesh_replicate_placementsshard_placements tp_placements tp_placementrUs r-_chunk_dtensorrs --k:IQRR~~/ /?q A -  ]]_ " " $F &%,,' 670K6;9>>5JK KK16y~~1FGAIKGG$Qi!! I3u ,!'   )) $Q' "6;9>>5JK KK#/R 16y~~1FGAIKGG%ay+!! I3u ,!'   9 LG* LGs+GGG!G&ctt|j}t|dk(r?t |dj tur!|dj }|j}|}|t|dkDr|fSgfS)Nrr)rrrlr rkr)rrv inner_tensors r-_pre_load_state_dictrsz- ( 5 5 7F 6{aD!1!12mCay'' **, c&kAoF 662 66r/ parent_meshc"||jk(sJttj|j}t dt |dz D]}t||<|j|j|}|jS)zGAll gather a DTensor in its FSDP dimension and return the local tensor.rrr~) rrAr^r_rrOr rrr)rrrrUs r-_all_gather_dtensorr)s &,, ,, ,dmmF$5$567J1c*o) *$! 1 $  &&!F ?? r/ceZdZdZdfd Zdej deej ee ffdZ dej de dej fdZ ddej de d e d e d e jd eejdej fd Zdej de dedej fdZdej deej eeffdZdedeedej fdZxZS)rz DTensorExtension is the TensorFlattener extension needed for 2D FSDP + TP. This is the implementation for FSDPExtensions defined in https://github.com/pytorch/pytorch/blob/main/torch/distributed/fsdp/_fsdp_extensions.py rct|d|_||_tj j |j|_yr2)super__init__compute_stream device_handler%_dynamodisablepost_unflatten_transform)selfr __class__s r-rzDTensorExtensions.__init__Es@ "*). (=(=  ) )) %r/rct|Sr2)rrrs r-pre_flatten_transformz'DTensorExtensions.pre_flatten_transformOsv&&r/param_extensionc|jxs|jj}|jj|5t |||j|j}t ||cdddS#1swYyxYw)N)rr)rrcurrent_streamstreamrr )rrrrresults r-rz*DTensorExtensions.post_unflatten_transformUs}$$K(:(:(I(I(K    & &v .  '"00#22 F  '   s 0A>>Br[rdrerfrBc t|||||Sr2)ry)rrr[rdrerfrBs r- chunk_tensorzDTensorExtensions.chunk_tensorhsVT:7KRPPr/rct|||Sr2)r)rrr[rs r- chunk_dtensorzDTensorExtensions.chunk_dtensorss fdK88r/ct|Sr2)rrs r-pre_load_state_dict_transformz/DTensorExtensions.pre_load_state_dict_transform{s$F++r/rct||Sr2)r)rrrs r-all_gather_dtensorz$DTensorExtensions.all_gather_dtensors #6;77r/)rNr2)__name__ __module__ __qualname____doc__rr%rtuplerrrrintrM ProcessGrouprBrrrrArrrr __classcell__)rs@r-rr=sV ' ' u||Xc]* +' ll58 4*. Q  Q Q Q " Q    Q& Q  Q9 99 9  9, , u||T%[( ), 88j)8  8r/)>r^typingrrrr%torch.distributed distributedrM&torch.distributed._shard.sharding_spec_shard sharding_spec shard_spec"torch.distributed.distributed_c10ddistributed_c10dc10d'torch.distributed._shard.sharded_tensorrrrr r :torch.distributed._shard.sharding_spec.chunk_sharding_specr torch.distributed.device_meshr $torch.distributed.fsdp._common_utilsr 'torch.distributed.fsdp._fsdp_extensionsr#torch.distributed.fsdp._shard_utilsrtorch.distributed.remote_devicertorch.distributed.tensorrrrr#6torch.distributed.tensor.parallel._data_parallel_utilsrr__all__rr&r.rr4r8rErrVrY ShardingSpecrrcryrrArrrr/r-rs; && ;;11 AX9DBL:TT   >W >uzz5::'=!> > >>s>uUZZ5K/L> *7*uUZZ-C'D*    $$))$$N7t00   ! !+0<<?B:G LLG G G  G  G  \\ G T> LL> > >  > B 7 LL 7 5<<e $% 7 *% \\(I8I8r/