Vh]&TUddlZddlZddlmZmZddlmZddlmZm Z ddl Z ddl m cm cmcmZddlmZddlmZddlmZmZmZmZe rdd lmZGd d eZeGd d eZGddeZiZe e!e eeffe"d<dZ#defdZ$dZ%eGddeZ&dZ'y)N)ABCabstractmethod) dataclass)Callable TYPE_CHECKING) ShardMetadata)_decorator_func) check_tensorget_chunked_dim_sizeget_split_size(validate_non_overlapping_shards_metadata) ShardedTensorceZdZdZy) PlacementSpecz Base class representing the placement of an entity. Subclasses of this class can be used to specify customized placements which might not be covered by existing APIs. N)__name__ __module__ __qualname____doc__Z/home/dcms/DCMS/lib/python3.12/site-packages/torch/distributed/_shard/sharding_spec/api.pyrrsrrcJeZdZUdZej j ed<dZy)DevicePlacementSpecz Associates placement of an entity with a single device. Args: device(:class:`torch.distributed._remote_device`): The device to place the entity on. devicect|jtjjs/tjj |j|_yyN) isinstancertorch distributed_remote_device)selfs r __post_init__z!DevicePlacementSpec.__post_init__.s>$++u'8'8'G'GH++::4;;GDKIrN) rrrrrr r!__annotations__r#rrrrr#s!    , ,,HrrceZdZdZedej dejdejfdZ e d dejde dd fd Z y) ShardingSpecz: Base class representing sharding specifications. tensor_sizestensor_propertiesreturncy)a Given a global tensor size, define how to shard a tensor like this shape across ranks, return ShardedTensorMetadata Args: tensor_sizes (:class:`torch.Size`): The tensor shape to shard on, a `torch.Size` object that represents the tensor shape to be sharded according to the ShardingSpec. tensor_properties(:class:`torch.distributed._shard.sharded_tensor.TensorProperties): Tensor properties used to create a ShardedTensor. Returns: A :class:`ShardedTensorMetadata` object that encodes the information about the layout of the ShardedTensor and its properties. Nrr"r'r(s rbuild_metadatazShardingSpec.build_metadata8rNtensorsrc_rankrcy)a Given a global tensor on src_rank, shard this tensor across ranks within the process group, return a ShardedTensor. Args: tensor (:class:`torch.Tensor`): Tensor needs to be sharded. Keyword args: src_rank (int, optional): The source rank which is used as the ground truth of the data for the parameter that would be sharded and scattered across the rest of the ranks. Default: 0. process_group (ProcessGroup, optional): The process group to work on. If None, the default process group will be used. Returns: A :class:`ShardedTensor` sharded from the given tensor. Nrr"r.r/ process_groups rshardzShardingSpec.shardLr-rrN)rrrrrrSizesharded_tensor_metaTensorPropertiesShardedTensorMetadatar,Tensorintr3rrrr&r&3sz jj /??   2 2   &EI ll .1   rr&_CUSTOM_SHARDING_SPEC_OPScVt|j}|tvxr |t|vS)zQ Returns whether or not the ShardingSpec has a custom op implementation. )typerr;) sharding_specop class_names r_has_custom_oprAes5m$11J// 8 +J7 7rr?ct|j}t||std|d|t||}|||||S)zA Calls the custom op for this ShardingSpec if it exists. z Custom op: z not registered for )r=rrA RuntimeErrorr;)r>r?typesargskwargsr2r@funcs r_dispatch_custom_oprHpsW m$11J - ,[,@ MNN $Z 0 4D tV] 33rc|j}|tvr it|<tjt|t|S)z Decorator to allow custom registration of ops. Args: sharding_spec_class(type): The ShardingSpec for which we need to add this custom op. func(Callable): The op to override (ex: torch.bmm) )r?op_table)rr; functoolspartialr )sharding_spec_classrGr@s rcustom_sharding_spec_oprN}sC%11J2202!*-   D+DZ+P rceZdZUdZeeed<dZdejde jde jfdZ d d ejd edd fd Zy)EnumerableShardingSpeca@ This is a type of PlacementSpec that allows users to specify a generic sharding scheme by enumerating exactly how each shard is laid out. Args: shards(List[ShardMetadata]): List of :class:`ShardMetadata` objects representing each shard. Note that none of the shards should overlap. shardsc `t|jdk(rtd|jd}|jD]X}|dk7r<|t|jk7r$td|dt|jt|j}Zt |jy)NrzEmpty shard list provided: z%Found inconsistent ranks for shards: z and )lenrQ ValueError shard_offsetsr)r"rankr3s rr#z$EnumerableShardingSpec.__post_init__s t{{ q :4;;-HI I[[ ,Erzdc%*=*=&>> ;D6s5K^K^G_F`au**+D  , 1=rr'r(r)cpt|j|tj|j||Sr)r rQr6r8r+s rr,z%EnumerableShardingSpec.build_metadatas1 T[[,/"88 KK'8  rNr.r/rctd)Nz1EnumerableShardingSpec.shard not implemented yet!)NotImplementedErrorr1s rr3zEnumerableShardingSpec.shards""UVVrr4)rrrrlistrr$r#rr5r6r7r8r,r9r:r3rrrrPrPsv   >  jj  /??    2 2  FJWllW.1W WrrPc g}d}g}g}g}|D]}|j|j|j}|jt||j|j|j|jt |D cgc] \}} | dk7s |} }} t | dk(rt | dk7rd}n|s| d}|| dk7sd}n|tt||tjdD cgc]\} } |  }} } ddl m } | ||}t|D cgc]} | | c} }t|}t|D cgc]} | | c} }t |}t||}tt|Dcgc]}t|||c}}t|Dcgc]}||z }}||k(r||k(r|St!|Scc} }wcc} } wcc} wcc} wcc}wcc}w)a Infer the sharding spec from the metadata of each shard of a ShardedTensor. If the tensor is sharded only on one dimension, we can then verify whether it's a ChunkShardingSpec or not. The way to verify it is to first get the total length and perform a chunk sharding with the given placements to see if we can have the same chunk size as the given shards_metadata. If not, we assume it's enum sharded. Args: shards_metadata (List[ShardMetadata]): List of Metadata of local shards. Returns: A :class:`torch.distributed._shard.sharding_spec.ShardingSpec` object of sharding spec for one sharded tensor. Nrr )key)ChunkShardingSpec)dim placements)append placementrVsum shard_sizes enumeraterTsortedzipoperator itemgetterchunk_sharding_specr^r ranger rP)shards_metadatar`chunk_sharding_dimchunk_offset_listshard_size_listshard_offset_listshard_metadata local_offsetsidxe shard_dims_xr^ chunk_specrdshard_total_lengthrVchunks split_sizechunk_shard_sizeschunk_shard_offsetss r)_infer_sharding_spec_from_shards_metadatar~sAJO).223&44   ]!34~99:  !=!=>(1-(@Kfc1AFcK K z?a   z?a !%  "!+A  :a= 0!%  -0%%z28K8KA8N 1    ;&"! _Ma 23MN  -?PQ!"4 5QR Z#$6? "!= %%7SI   H)(rKrhabcrr dataclassesrtypingrrr0torch.distributed._shard.sharded_tensor.metadatar _shardsharded_tensormetadatar6!torch.distributed._shard.metadatar*torch.distributed._shard.op_registry_utilsr _internalsr r r r'torch.distributed._shard.sharded_tensorrrrr&r;dictstrr$rArHrNrPr~rrrrs#!* NN;FFC  H- H  H+ 3+ ^BD4T(H*<%= =>C 4 4  *W\*W *WZM3r