VhN ddlmZddlmZddlmZddlmZmZm Z ddl Z ddl m Z ddl mZddlmZdd lmZ dd lmZmZmZeed fZeeefZeeeZ ee eeeefZ!d efd Z"de fdZ#de fdZ$ded efdZ%eGddZ&GddZ'Gdde'Z(Gdde'Z)eGddZ*eGddZ+eGdd Z,eGd!d"Z-y#e$r dd lmZmZmZYwxYw)#)Sequence) dataclass)cached_property)AnyOptionalUnionN) OpOverload) DeviceMesh) DTensorSpec) Placement) tree_leaves tree_map_onlyTreeSpec.returnc|jJdtj|jj|jj|jj S)zL This is used to propagate tensor metadata, must be under fake mode z)DTensorSpec does not contain tensor_meta.)dtype) tensor_metatorch empty_stridedshapestrider)args S/home/dcms/DCMS/lib/python3.12/site-packages/torch/distributed/tensor/_op_schema.py!_rebuild_tensor_from_dtensor_metar#sT ?? &S(SS &     oo## opc:|jjddk(S)N_)_schemanamers r_is_inplace_opr#/s ::??2 # %%rc2d|jjvS)Nout)r overload_namer"s r_is_out_variant_opr'6s BJJ,, ,,rspecc 8|yt|tr2dj|jDcgc] }t |c}St|t r.ddj|Dcgc] }t |c}zdzStd|cc}wcc}w)NNone(, )z!Unknown spec type to print: spec=) isinstancer join placementsstrr_pretty_print_spec RuntimeError)r(pss rr3r3=s | D+ &ww81A899 D( #TYYtD! 21 5DEEKK>tfEFF 9Ds B&B ceZdZUdZeeeeedffed<dZ ee eed<dZ ee e e ed<edefdZed Zd d edefd Zdefd Zy)PlacementStrategyaQ A placement strategy describes acceptable sharding placements of the output and the tensor arguments of an operation. note: when the op return value is a single DTensor object, output_specs is DTensorSpec; when the return value is a tuple of Optional[DTensor], output_specs is a tuple of Optional[DTensorSpec]. . output_specsN input_specsredistribute_costrc~t|jtr |jStd|j)z This function requires that the strategy have exactly one DTensorSpec as the output spec. If the output_specs is a tuple, we throw an exception. z;function output_spec expects a single DTensorSpec but got: )r/r9r ValueErrorselfs r output_speczPlacementStrategy.output_spec\s> d'' 5$$ $MdN_N_M`a rc t|jtr|jjSt|jtr-|jd}t|tsJ|jSt d|j)NrzUfunction output_spec expects a single DTensorSpec or a tuple of DTensorSpec but got: )r/r9r meshtupler=r?out_specs rrBzPlacementStrategy.meshis} d'' 5$$)) ) ))5 1((+Hh 4 44== ghlhyhygz{ rindexc|jJdt|j|kDs-Jd|dt|jd|j|j|S)Nz)input_specs of PlacementStrategy is None!zInvalid index z for input_specs of length z: )r:len)r?rFs r input_speczPlacementStrategy.input_specvsz+X-XX+4##$u, UG#>4##$%R(8(8'9 ; ,&&rc|jt|jd}nd}t|j}||S)Nz -> r+)r:r3r9)r?input_specs_stroutput_spec_strs r__str__zPlacementStrategy.__str__~sL    '!3D4D4D!E FdKO O,T->->?!"?"344r)r)__name__ __module__ __qualname____doc__rr rCr__annotations__r:rr;listfloatrr@rBintrIr2rMrrr8r8Hs U8K+@#+E%FFGG37K(;/07 6:xT%[ 129 [    ''K'55rr8ceZdZdZy) StrategyTypezi Base class type for op strategy, We have two StrategyType: OpStrategy and TupleStrategy N)rNrOrPrQrVrrrXrXsrrXceZdZdZdeeddffd ZdefdZde fdZ e dZ e d Z e d Ze d ZxZS) OpStrategyz[ OpStrategy that consists of a list of placement strategies associated with the op strategiesrNc0t|||_yN)super__init__r[)r?r[ __class__s rr_zOpStrategy.__init__s 3=rcdj|jDcgc] }t|c}}|j}d|d|Scc}w)Nr-[z ] @ mesh: )r0r[r2 mesh_shape)r?strategystrategy_list_strrcs rrMzOpStrategy.__str__sH IIT__&Us8}&UV__ $%Z |<<'VsAc:td|jDS)zR Returns the max number of shards across all placement strategies c3HK|]}|jjywr])r@ num_shards).0rds r z,OpStrategy.max_num_shards..sSx8''22S ")maxr[r>s rmax_num_shardszOpStrategy.max_num_shardssS4??SSSrc4|jdjSNr)r[rBr>s rrBzOpStrategy.meshsq!&&&rcH|jdjjSro)r[rBrr>s rrczOpStrategy.mesh_shapesq!&&,,,rcH|jdjjSro)r[r@ndimr>s rrrzOpStrategy.ndimsq!--222rcH|jdjjSro)r[r@rr>s rrzOpStrategy.shapesq!--333r)rNrOrPrQrSr8r_r2rMrUrmpropertyrBrcrrr __classcell__r`s@rrZrZs>4(9#:>t>== TT ''--3344rrZcNeZdZdZdeeddffd ZdedefdZ de fdZ xZ S) TupleStrategya? TupleStrategy represents the output strategy of this op is a tuple of strategy, i.e. If the output of this op is a tuple of tensors or list of tensors with possibly different placement strategies, we should return a TupleStrategy that contains a tuple of OpStrategy, where each child represents the sharding strategy of "each element" of the tuple/list of tensors the op returns. NOTE: if the output of the op is a List[Tensor] and they share the same placement strategy, then we should return a single OpStrategy instead of a TupleStrategy childsrNc0t|||_yr])r^r_ry)r?ryr`s rr_zTupleStrategy.__init__s .4 rrFc\|j|}t|tsJ|jSr])ryr/rZrB)r?rF op_strategys r child_meshzTupleStrategy.child_meshs,kk%( +z222rc djt|jDcgc]\}}t|c}}}d|dScc}}w)Nr-zTupleStrategy(r.)r0 enumerateryr2)r?idxstratchild_strategies_strs rrMzTupleStrategy.__str__sM#yy/8/E FeE |_ F  45Q77 GsA ) rNrOrPrQrrXr_rUr r}r2rMrurvs@rrxrxs? 5x 55$5   88rrxcHeZdZUdZdZeed<dZee e ed<dZ e ed<y)RuntimeSchemaInfoa RuntimeSchemaInfo stores the operator schema related information for runtime (eager) execution. This is mainly used for two ways: 1. to generate hash for args to determine whether to re-run sharding prop or not 2. to determine if we need pytree d static_argnumNstatic_kwargkeyF needs_pytree) rNrOrPrQrrUrRrrrSr2rboolrVrrrrs2M3+/OXd3i(/L$rrceZdZUdZeed<eed<eed<dZe e ed<e de e dffd Ze de edffd Zdefd Zdefd Zdd ZdedefdZdefdZdefdZddedefdZdefdZdedefdZdefdZdefdZ ddZ!y)OpSchemaa OpSchema is a data class that describes an operator input schemas, it includes DTensorSpecs (instead of DTensor) and non-tensor args/kwargs (positional order preserved). It is mainly used by the DTensor's dispatching logic to perform various actions (i.e. sharding propagation, caching sharding decisions, redistribute, etc.) NOTE: this should be used as a read only data class TODO: make this a frozen dataclass Args: op: the operator overload we are intercepting args_schema: contains args except that the DTensor args have been replaced with its DTensorSpec or OpStrategy kwargs_schema: contains kwargs except that the DTensor kwargs have been replaced with its DTensorSpec or OpStrategy r args_schema kwargs_schemaN schema_infor.c|j+|jjrt|jn |j}t d|DS)z args_spec: Tuple[DTensorSpec, ...]: contains a clean list of args spec list with NO non-DTensor positional arguments (i.e. int/float/tuple, etc) mainly used by sharding propagation to propagate the output spec c3BK|]}t|ts|ywr]r/r riitems rrjz%OpSchema.args_spec.. sLdj{.KTLrrr rrCr?argss r args_speczOpSchema.args_specsO+0@0@0M0M (( )!! LdLLLrc|j+|jjrt|jn |j}t d|DS)Nc3BK|]}t|ts|ywr])r/rZrs rrjz)OpSchema.args_strategy..sKdjz.JTKrrrs r args_strategyzOpSchema.args_strategy sO+0@0@0M0M (( )!! KdKKKrcdj|jDcgc] }t|c}}d|jd|d|jdScc}w)Nr-z OpSchema(op=z, args_schema=(z), kwargs_schema=r.)r0rr2rr)r? arg_schemars r__repr__zOpSchema.__repr__s]ii4CSCS TZZ TU 477)$(M*"001 4 !UsAcg}d}|jD]}t|tr1|jt ||j j }Et|trXt|jdk(sJ|jt|jdj|j}t|trI|jd}t|tsJ|j}|jt ||jt |"d|jddj!|d|dS)NrzOp(op=z, args_schema=r-z @ mesh: r.)rr/r appendr2rBrrZrHr[r3r9rcrxryrr0)r?rrcrfirst_op_strtgys rrMzOpSchema.__str__s!#  ## -C#{+""3s8, XX^^ C,3>>*a///""#5cnnQ6G6T6T#UV ^^ C/"%**Q-!/:>>>,77 ""3s8,""3s8, -ytyy/E.FiPZ|[\]]rcd}|jD]Q}t|ts|j!t d|jj DsHd}||_y||_y)NFc3PK|]}t|tj ywr])r/rSymInt)rir6s rrjz)OpSchema.__post_init__..7sPqz!U\\2Ps$&T)rr/r ranyr has_symints)r?ras r __post_init__zOpSchema.__post_init__3sa !! A![)amm.GPAMM.Es#Hq:a-:d:Hrk)rr/r rSall)r?rr is_tensors r#arg_type_tensor_or_tensor_list_likez,OpSchema.arg_type_tensor_or_tensor_list_like<sBw'sK0 #t$HCHHHrc|jjj}t|dkDxr't |dj t jS)Nrr)rr returnsrHr/typer TensorTyper? return_typess rreturn_type_tuple_tensor_likez&OpSchema.return_type_tuple_tensor_likeGsJww.. < 1$  O %"2"2*  rc|jjj}t|djt j Sro)rr rr/rrrrs rreturn_type_tensorzOpSchema.return_type_tensorOs4ww.. ,q/..0@0@AArvalidatec .|jd}t|ttfr |j}n}t|t t tfrIt|tr|jdn|d}t|ttfsJ|j}ntd|jd|rc|jddD]Q}t|ttfs|j|k7s*td|jd|d|jd |S) a This util can be used to get a mesh from the OpSchema that contains multiple DTensors as arguments. When `validate` is True, it will try to validate that all the arguments have the same mesh to avoid unexpected cross mesh errors. NOTE: this util currently does not handle TupleStrategy when `validate=True`, this is because for TupleStrategy there could be different types of checks, i.e.: - for stack and cat like op, we need to check within a TupleStrategy is every input is on the same mesh - for foreach like ops we need to check "zipped" inputs are on the same mesh for each index. rz+Cannot find device mesh from args for op : .rNz1DTensor does not support cross-mesh operation on z! Got meshes:  z>. Please make sure all the arguments have the same DeviceMesh.) rr/r rZrBrSrCrxryr=rr4)r?r first_argrB first_elemrs rget_mesh_from_argszOpSchema.get_mesh_from_argsUs$$Q' i+z!: ;>>D  D%#? @i7  #q\  j; *CD DD??DJ477)STUV V ''+ cK#<=#((dBR&KDGG9U''+fAchhZ8WX  rcjstjd}n,jjjj}t fdt jD}|,t fd|D}tj||fStj|fS)Nc3K|]:\}}j|s|k\rt|tr t|n|<ywr])rr/rSrC)riirr?rs rrjz$OpSchema.__hash__..sD 177:a=>P#1d+E!H 2 sAAc3VK|] }jj|d"ywr])rget)rikr?s rrjz$OpSchema.__hash__..s)#45""&&q$/#s&)) rrHrrrrCrhashr)r?r args_to_hashkwargs_to_hashrs` @r__hash__zOpSchema.__hash__{s 0 01M"O ,,::M"..>>O !$"2"23   &"#9H#N,?@ @,/0 0rotherct|tsy|j|jk7ryt|jt|jk7ry|j st|j}d}n,|j j }|j j}tt|j|jD],\}\}}t|tr||k7ry||k\s&||k7s,y|rB|D]=}|jj|d|jj|dk7s=yy)NFT) r/rrrHrrrrrzipr rr)r?rrrrself_arg other_argkeys r__eq__zOpSchema.__eq__s2%* 77ehh  t C(9(9$: : 0 01M"O ,,::M"..>>O(1   %"3"3 4)   $A$)(K0X5Jm#I(=   & !%%))#t48K8K8O8O9!  ! rc@ttt|jS)z gen_fake_args: generate fake args for the operator, this is mainly used by sharding propagation rules to generate fake args for the operator to run the local tensor operator and get the output spec. )rr rrr>s r gen_fake_argszOpSchema.gen_fake_argss  :Ds rgen_fake_kwargszOpSchema.gen_fake_kwargss  :D .2(+2$$ T TrrceZdZUdZeed<eed<eeed<e eed<e e efed<dZ e eed<dZe eed <y) OpInfoz7 All Runtime Op execution info are packed here compute_meshschemaflat_args_schema local_args local_kwargsNargs_tree_specoutput_sharding)rNrOrPrQr rRrrSrrdictr2rrrrrrVrrrrs[ 6l"  sF{##)-NHX&-15OXn-4rr).collections.abcr dataclassesr functoolsrtypingrrrr torch._opsr torch.distributed.device_meshr &torch.distributed.tensor._dtensor_specr (torch.distributed.tensor.placement_typesr torch.utils._cxx_pytreer rr ImportErrortorch.utils._pytreerCrrrr2rrS PlacementListrrr#r'r3r8rXrZrxrrrrrVrrrsy$!%'' !4>>LL   #v+  Xi() % Xh{6K-L LMN f &z&-:-GVGG ;5;5 ;5|"4"4J8L88  ( w9w9 w9t TT T: 55 5]s C00DD