Vh>fdZddlZddlZddlmZddlZddlmZddlm Z m Z ddl m Z ddl mZdd lmZdd lmZmZdd lmZdd lmZd dlmZd dlmZmZerddlmZGddeZdZ dZ!GddeZ"GddeZ#GddeZ$GddeZ%GddeZ&Gdd eZ'y)!a Distributed computing variable tracking classes for PyTorch Dynamo. This module implements variable tracking for distributed computing components: - Process Groups (for collective communication) - Device Meshes (for distributed tensor sharding) - Placement Types (for specifying distribution strategies) - Distributed Tensors and their operations - Backward hooks for distributed module operations These classes are responsible for tracking distributed operations during graph compilation while maintaining proper guards and handling distributed-specific behaviors. They ensure correct handling of distributed components like process groups, device meshes, and placement strategies while preserving proper semantics for distributed tensor operations in the compiled code. The implementation provides special handling for distributed package availability checks and proper tracking of distributed state and operations across processes. N) TYPE_CHECKING) BackwardState)compiled_autograd variables) trace_wrapped)unimplemented_v2)%call_module_hooks_from_backward_state) GuardBuilder install_guard) AttrSource)istype)VariableTracker)ConstantVariable EnumVariable)InstructionTranslatorc:eZdZdZdfd ZdZedZxZS)DistributedVariablea{ The base distributed variable that encapsulates common methods for the distributed objects (i.e. ProcessGroup, DeviceMesh, etc.). Concrete distributed objects could inherit this class and add object specific logic. i.e. It provides the check on the distributed package existance and hold the tracking value for the corresponding distributed object. c zt|di|tjst ddddg||_y)Nz+torch.distributed package is not available!zOThe PyTorch package doesn't include torch.distributed when builing from source.zESet USE_DISTRIBUTED=1 to enable it when building PyTorch from source.gb_typecontext explanationhints)super__init__r is_availabler value)selfr!kwargs __class__s S/home/dcms/DCMS/lib/python3.12/site-packages/torch/_dynamo/variables/distributed.pyrzDistributedVariable.__init__8sA "6""//1 Em[   c,t|jSN)typer!r"s r% python_typezDistributedVariable.python_typeEsDJJr&c>tjjSr()torch distributedr rr&r%r z DistributedVariable.is_availableHs  --//r&)returnN) __name__ __module__ __qualname____doc__rr+ staticmethodr __classcell__r$s@r%rr-s&  00r&rctjsyddlm}t j |xr||j uS)NFr)DTensor)rr torch.distributed.tensorr8inspect isfunction from_local)r!r8s r% is_from_localr=Ns5  + + -0   e $ D'2D2D)DDr&ctjsyddlm}m}m}m}m}|||||g}tj|xr||vS)NFr)_get_group_size_by_name_get_group_tag_rank_not_in_group$_resolve_group_name_by_ranks_and_tagget_process_group_ranks) rr "torch.distributed.distributed_c10dr?r@rArBrCr:r;)r!r?r@rArBrCconstant_processgroup_functionss r%is_constant_pg_functionsrFVsR  + + - , '#   e $ Q2Q)QQr&c@eZdZdZedZdddedeffd ZxZ S)WorldMetaClassVariablez Tracks torch.distributed.GroupMember and torch.distributed.group, which are instances of the metaclass _WorldMeta. cJ|jsyddlm}t||uS)NFr) _WorldMeta)r rDrJr))clsr!rJs r%is_group_member_typez+WorldMetaClassVariable.is_group_member_typess#!AE{j((r&txrnamer/c|dk(r^t|jd}t|jtj t |jjS|dk(r^t|jd}t|jtj t|jjSt|1||S)NWORLD)basememberNON_GROUP_MEMBER) r sourcer make_guardr ID_MATCHProcessGroupVariabler!rPrrSr var_getattr)r"rMrNrTr$s r%rXz"WorldMetaClassVariable.var_getattr|s 7?T[[AF &++L,A,AB C' (8(89 9 ' 'T[[9KLF &++L,A,AB C ; ;< <w"2t,,r&) r0r1r2r3 classmethodrLstrrrXr5r6s@r%rHrHms< )) -5 -S -_ - -r&rHcFeZdZedZdZ dfd ZxZS)PlacementClassVariablecvtjsyddlm}t |tuxr t ||SNFr) Placement)rr (torch.distributed.tensor.placement_typesr_r) issubclassr!r_s r%is_placement_typez(PlacementClassVariable.is_placement_types1#//1FE{d"Cz%'CCr&c|jSr(r!r*s r%as_python_constantz)PlacementClassVariable.as_python_constant zzr&cdtj|jddtjfvrm|j ratj |j}t |}tj|jddr|j|d|||St|%|||S)N__new__r) r:getattr_staticr!objectrirTPlacementVariable call_methodr call_function)r"rMargsr#new_objvarr$s r%rnz$PlacementClassVariable.call_functions  " "4::y$ ?FNNCT T nnTZZ0G#G,C%%djj*dCJf= w$Rv66r&)rMrrolist[VariableTracker]r#dict[str, VariableTracker]r/r)r0r1r2r4rcrfrnr5r6s@r%r\r\sLDD7 #7&7- 7  77r&r\cZeZdZedZdZdddedeffd Z d fd Z xZ S) rlcPtjsyddlm}t ||Sr^)rr r`r_ isinstancerbs r% is_placementzPlacementVariable.is_placements##//1F%++r&c|jSr(rer*s r%rfz$PlacementVariable.as_python_constantrgr&rMrrNr/c|dk(r)tj|jjSt|||S)Ndim)rcreater!rzrrXr"rMrNr$s r%rXzPlacementVariable.var_getattrs5 5=#**4::>>: :w"2t,,r&cddlm}gd}||vr t|j}t j |ddJdt j ||}|tjur|jdS|D cgc]} | j}} |jD cic]\} } | | j}} } |dk(r||jg|i||S||jg|i|} |j| St |5||||S#t $rd}YwxYwcc} wcc} } w)Nr)r)r __setattr__is_shard is_partial is_replicate __getattr__zno custom getattr allowed!r~)rrr)r!r:rjAttributeErrorrkrr{rfitemsrrm)r"rMrNror#rconstant_fold_functions value_typemethodxkv constant_valr$s r%rmzPlacementVariable.call_methodsP '#  * * !$**- **:}dKS0/0S // DA(.'..t44489qA((*9D9t>v>L*#**<8 8w"2tT6::"   :KsA D$D5/D:$ D21D2rorrr#rsr/r) r0r1r2r4rwrfrZrrXrmr5r6s@r%rlrls],,-5-S-_- *;& *; - *;  *;*;r&rlcZeZdZedZdZdddedeffd Z d fd Z xZ S) DeviceMeshVariablecPtjsyddlm}t ||S)NFr) DeviceMesh)rr torch.distributed.device_meshrr)r!rs r%is_device_meshz!DeviceMeshVariable.is_device_meshs##//1<eZ((r&c|jSr(rer*s r%rfz%DeviceMeshVariable.as_python_constantrgr&rMrrNr/c|dk(r)tj|jjS|dk(r)tj|jjSt |||S)Nndim device_type)rr{r!rrrrXr|s r%rXzDeviceMeshVariable.var_getattrsX 6>#**4::??; ; = #**4::+A+AB Bw"2t,,r&c|dk(r}|Dcgc]}|j}}|jDcic]\}}||j} }}tj|jj |i| S|dk(r-tj|jj S|dk(rs|Dcgc]}|j}}|jDcic]\}}||j} }}t|jj|i| S|dk(r#t|jjSt |-||||Scc}wcc}}wcc}wcc}}w)Nsizeget_coordinate get_group_get_or_create_default_group) rfrrr{r!rrrWrrrrm) r"rMrNror#r const_argsrr const_kwargsr$s r%rmzDeviceMeshVariable.call_methods_ 6>:>?Q!..0?J?BH,,.Q$!QAq3355QLQ#**?4::??J+W,+WX X # ##**4::+D+D+FG G ; :>?Q!..0?J?BH,,.Q$!QAq3355QLQ'$ $$jALA  1 1' (O(O(QR Rw"2tT6::@Q @QsE(E-?E3+E8r) r0r1r2r4rrfrZrrXrmr5r6s@r%rrs]))-5-S-_-;& ; - ;  ;;r&rcReZdZdZdZ dfd Zdfd ZedZxZ S)rWa We don't want a ProcessGroup object to end up in our output graph. But it's common for dynamo to intercept a PG that is then used to get info like rank() or world_size(), as well as passed to utility functions in distributed_c10d which desugar it into plain types like a ranklist and tag. For convenience and proper guarding, we construct a variable type. TODO: make it possible to use ProcessGroupVariable as input to simple functions like _expand_group without dynamo complaining about making a proxy for it. It is not a tensor-like type, and we don't want a proxy- but dynamo assumes torch library functions are dealing with tensor-like types and would have proxies for their args. TODO: should we make this inherit VT instead of UDOV? Do we want any of the default behaviors or just graph-break whenever one of our special cases is not hit? c|jSr(rer*s r%rfz'ProcessGroupVariable.as_python_constant+rgr&c|dk(r7tjj|jj S|dk(r7tjj|jj S|dk(r7tjj|jj St|!||||S)Nrankr_get_backend_name) rrr{r!rrrrrmr"rMrNror#r$s r%rmz ProcessGroupVariable.call_method.s 6>--44TZZ__5FG G 6>--44TZZ__5FG G & &--44TZZ5Q5Q5ST Tw"2tT6::r&cdk(r3tjjjjSdvrtj fdSt S)N group_name)rrc,j||Sr()rm)ror#rNr"rMs r%z2ProcessGroupVariable.var_getattr..Cs(8(8T4(Pr&)rrr{r!rLambdaVariablerrXr|s```r%rXz ProcessGroupVariable.var_getattr>s_ < --44TZZ5J5JK K # #++P w"2t,,r&c`tjsyddlm}ddlm}t |||fS)NFr) ProcessGroup)FakeProcessGroup)rr torch._C._distributed_c10dr+torch.testing._internal.distributed.fake_pgrr)r!rrs r%is_process_groupz%ProcessGroupVariable.is_process_groupHs,#//1;Pel,<=>>r&rrMr) r0r1r2r3rfrmrXr4rr5r6s@r%rWrWsI$;& ; - ;  ; -??r&rWc eZdZdZedededefdZdejjdedededdf fd Z d Z d e ed e eefdeffd ZddZxZS)BackwardHookVariablezU Handles torch.utils.hooks.BackwardHook for module-level backward hooks. module user_hooksuser_pre_hooksc tjstddddgdtf fd }|jj |d\}|jj |\ }|jj |\ }|jj d||fi}tjjjdd d |jjd <t||||S) Nz7Module-level backwards hooks require compiled autograd.rzREnable compiled autograd by setting torch._dynamo.config.compiled_autograd = True.rbw_statec tjjjdt j t t|ft j t t|fS)a  Rather than installing the user hooks in the graph (which don't survive AotAutograd), we install hooks that will call trace_wrapped in the backward pass that CompiledAutograd can turn into actual hook calls. N)fnr hooks_name module_name)r-utilshooks BackwardHook functoolspartialrr )rruser_hooks_nameuser_pre_hooks_names r%_in_graph_bw_hooksz7BackwardHookVariable.create.._in_graph_bw_hooksjsk;;$$11%%%@!)#2$/ %%%@!)#6$/  r&modrnr example_value)rcompiled_autograd_enabledr routputadd_backward_state_hook create_proxyr-rrrnodemetar) rMrrrrbw_state_proxy_proxyrrrs @@@r%r{zBackwardHookVariable.createYs!:: Qh    :')ii&G&GPU&V# ^!#!B!B>!RQYY>>zJ &&        ,1;;+<+<+I+I$PRTV+W (#E6:~NNr&rr/Nc \t|di|||_||_||_||_y)Nr)rrrrrr)r"rrrroptionsr$s r%rzBackwardHookVariable.__init__s2 #7#  $,r&c|jSr()rr*s r%as_proxyzBackwardHookVariable.as_proxyrgr&ror#c^|dvr|j||g|i|St| ||||S)N)setup_input_hooksetup_output_hook) _setup_hookrrmrs r%rmz BackwardHookVariable.call_methodsC < <#4##B>t>v> >w"2tT6::r&c ddlm}|||jjd||j |j fiS)Nr) wrap_fx_proxyrm)builderrrrr)r"rMhook_method_namerors r%rz BackwardHookVariable._setup_hooksD*  II " " $--/2    r&r)r0r1r2r3r4rr{r-fxProxyrrlistdictrZrmrr5r6s@r%rrSs 7O7O$7O( 7O7Or -xx~~ -  -$ - ( -  - ;?# ; S/)* ;  ;  r&r)(r3rr:typingrr-%torch.fx.experimental._backward_staterrrr_trace_wrapped_higher_order_oprexcr external_utilsr guardsr r rTr rrrQrconstantrrtorch._dynamo.symbolic_convertrrr=rFrHr\rlrrWrrr&r%rs( ?+:"B0!4D0/0BER.-0-6 70 7F=;+=;@*;,*;Z8?.8?vg ?g r&