VhEq  UddlZddlZddlZddlmZddlmZddlmZm Z m Z ddl Z ddl m Z ddlmZddlmZmZddlmZmZdd lmZmZdd lmZmZdd lmZdd lm Z dd l!m"Z"ddl#m$Z$ddl%m&Z&ddl'm(Z)ddl*m+Z+ejXe-Z.e j^j`Z0edGddZ1e j^j`jdjfe j^j`jhjfe j^j`jjjfe j^j`jljne j^j`jpjfe j^j`jrjte j^j`jvjfe j^j`jxjfiZ=e=j}Dcic]\}}|| c}}Z?de jjfdZBeGddZCde jtde jtdeDeCde jtfdZEde jtde jtdeDeCde jtfdZFde jjde jtde jtdeDeCde jjf d ZHde jjd!e jjde jjfd"ZIde jjd!e jjde jjfd#ZJe+e0jxjfe0jhjfgZKd!e jjdeLfd$ZMd!e jjdeLfd%ZNde jjddfd&ZOde jjddfd'ZPe0jjfe1e0jjfde0jjfe1ejdeFe1eEdeN(iZU e j^jZWeWjjfe1eWjjfdeWjjfe1eWjjfdiZ\eUje\iZe_e jje1febd)<ej|D]\ZcZde1eddeec<e+egZee+e0jjne0jjne0jjfe0jjfgZjde jjddfd*Zkde jjddfd+Zlycc}}w#e^$rYwxYw),N) defaultdict) dataclass)AnyCallableUnion)compute_overlapping_tensors)enable_python_dispatcher)ReinplaceCountersReInplaceTrigger)kernel_side_table triton_kernel_wrapper_functional)configinductor_prims)get_node_storageis_node_realized)inplaceable_foreach_ops)V)GuardOnDataDependentSymNode)immutable_dict) _is_view_op)_pytree) OrderedSetT)frozencpeZdZUedefed<eed<dZeejjge fed<y) InplaceableOp. inplace_op mutated_argcy)NTnodes S/home/dcms/DCMS/lib/python3.12/site-packages/torch/_inductor/fx_passes/reinplace.pyzInplaceableOp.&s extra_checkN) __name__ __module__ __qualname__rr__annotations__intr%torchfxNodeboolrr$r"rr"s5c""3DK588==/4/0Dr$rgraphctjd||f\}}tj5||i|}ddd|j |||}|j d<|S#1swY-xYw)Nclt|tjjr|jdS|SNval isinstancer+r,r-metar s r"r#z%graph_call_function..4s$D%((--)HTYYu%dr$r3)pytreetree_mapr fake_mode call_functionr6)r/fnargskwargs fake_args fake_kwargs fake_resultr!s r"graph_call_functionrA2su#__R vI{ 4)3{3 4   r4 0D"DIIe K 44s A""A+cheZdZUejj ed<eedfed<e e efed<y)ViewOptarget.r<r=N) r&r'r(r+_ops OpOverloadr)tuplerdictstrrr$r"rCrC?s. JJ ! !! S/ cNr$rCinpsrcview_opsreturnc6|}|D]H}tjd|j|jf\}}|j|g|i|}J |j ||S#t $r+}t d|jd|j|d}~wwxYw)Nclt|tjjr|jdS|Sr2r4r s r"r#z._inplace_generalized_scatter..Ls%Zehhmm-L5)RVr$z-shape error in scatter op, can not broadcast z to )r7r8r<r=rDcopy_ RuntimeErrorshape)rJrKrLtmpviewr>r?es r"_inplace_generalized_scatterrVFs C:!' V YY $"  ;dkk#9 9[9 :  # J ;CII;d399+ V  sA$$ B-&BBc<|j}t|||SN)clonerV)rJrKrLouts r"_generalized_scatterr[Ys ))+C 'S( ;;r$c|d|dd}}|r@t||j|g|ji|j}t ||||dd}t|t |j||g|ji|jS)Nr)rArDr<r=$_decompose_scatter_functional_helper_VIEW_OP_TO_SCATTER)r/rJrKrLview_op view_ops_tailrTs r"r^r^`s &a[(12,]G" 7>>3 )0 9@ 35$Xab\R  GNN+       ..  r$r!cV|jtusJt|g|jS)a_Decompose _generalized_scatter to a sequence of view_scatter operations e.g. _generalized_scatter(inp, src, [(aten.slice, 0, 0, 10), (aten.slice, 1, 10, -10)]) will become view = aten.slice(inp, 0, 0, 10) view_updated = aten.slice_scatter(view, src, 1, 10, -10) inp_updated = aten.slice_scatter(inp, view_updated, 0, 0, 10) )rDr[r^r<)r/r!s r"_decompose_scatter_functionalrcxs+ ;;. .. . / B BBr$c|jttfvsJ|j\}}}|jrJ|jturt |t j|}|}|D]1}t ||j|g|ji|j}3t |t jj|||S)a6Decompose _generalized_scatter using mutations e.g. _generalized_scatter(inp, src, [(aten.slice, 0, 0, 10), (aten.slice, 1, 10, -10)]) will become inp_updated = aten.clone(inp) slice1 = aten.slice(inp_updated, 0, 0, 10) slice2 = aten.slice(slice1, 1, 10, -10) slice2.copy_(src) ) rDr[rVr<r=rAatenrYrPdefault)r/r!rJrKrLrSrTs r"_decompose_scatter_mutatingrgs ;;/1MN NN NCh{{? {{**!%S9 CV!%cUDIIUUVtzz113< Jr$cF|j\}}}td|DS)Nc3@K|]}|jtvywrX)rD_ALWAYS_MUTATING_SCATTER_OPS.0rTs r" z/scatter_always_uses_mutation..sPtt{{::Ps)r<any)r!_rLs r"scatter_always_uses_mutationrps"YYNAq( PxP PPr$c|j\}}t|rytr t|ryjdvrt fd|j Dryy)aChoose between mutating and functional scatter decompositions Reinplacing view scatter ops can be pessimising as it blocks fusion with the input or output tensor computations. However, it is still profitable if the input and output would have been realized anyway. T placeholderget_attrc3K|]=}|jtjjuxr|jdu?yw)rN)rDrerPrfr<)rluserrJs r"rmz+should_reinplace_scatter..s>5FJ tzz)))Adiilc.AA5sAAF)r<rproprnusers)r!_src _view_opsrJs @r"should_reinplace_scatterr{sf 99Cy$D)!1$!7 vv,,5NRjj52 r$ctj|jdt|jdtD]x}|j tuxs t |}|j|5|r t||}n t||}ddd|j|j|zy#1swY.xYw)z1Replace _generalized_scatter with normal aten opsr:)rwrDN) itertoolschain find_nodesr[rVrDrpinserting_beforergrcreplace_all_uses_with erase_node)r/r! use_mutationnew_nodes r"decompose_generalized_scatterrs O4HI O4PQ KK7 7 2+D1   # #D ) F6udC8E  F ""8, ! F Fs 4B==C cJittdtjjffd dtjjffd }j D];}t |jr |!|jtvs4||=y)aV This canonicalizes view scatter ops into a generalized form, defined as: def scatter(inp, src, views): tmp = inp.clone() for view in views: tmp = view(tmp) tmp.copy_(src) We also fuse consecutive view scatter ops of the form a = scatter(view2(self), src, [view1]) b = scatter(self, a, [view2]) which can be rewritten as b = scatter(self, src, [view2, view1]) a = view2(b) This is both more efficient as we only do a single scatter, and also easier to reinplace since there is only one use of `self` r!c|jd}j|||<g|t|j|jdd|j|<y)Nrr]r<r=)r<getrCrDr=)r!rJnode_to_view_basenode_to_view_ops r" handle_viewsz3canonicalize_view_scatter_ops..handle_viewssiiil"3"7"7S"A$! S !!  YYqr]{{ ! r$c d t|jdk\sJ|jdd\ tt|j|jdd|j   fd}|sQ j |5t t g}ddd|j j|y j\}}} j 5t t|g|}|j| j| jrRt t|j|g|jddi|j } | j| j dddy#1swYxYw#1swYyxYw)Nrc jtury j\}}}j}j||}||uxr|gk(S)NF)rDr[r<r) src_inp_src_src_src_scatter_view_opinp_basesrc_baserJrrscatter_view_oprKs r"can_fusezLcanonicalize_view_scatter_ops..handle_view_scatter..can_fuseszz!5569hh 3GX3(,,S#6H(,,Wg>Hx'OG,DI %II- r$) lenr<rC_SCATTER_OP_TO_VIEWrDr=rrAr[rrrx)r!rr_src_inpsrc_srcsrc_scatter_view_opnew_srcrJrrKr/rrrs @@@r"handle_view_scatterz:canonicalize_view_scatter_ops..handle_view_scatter s499~"""99Ra=S  ,12;;   z''- .($%    & &x 0   T " 14.'.  # #C ( "*$ 7#67 H  & &x 0   T "yy-' 4YYqr]  kk W%))'2   S !/ " "   " "sF)B(F&F#&F/N) rlistr+r,r-nodesrrDr)r/rr!rrrs` @@@r"canonicalize_view_scatter_opsrs(=?9DT9JO  588==  ;"%((--;";"z & t{{ #   [[/ /  % &r$)r%rc  !"#iitt}tt#i!t t |j D]\} t|j |z dz ! <#t j  jtjjk(sj jdjdvs jd} jd}|jt j"k(r|jdjt$k(r<|jdj&d|jd jdk(s]|jdjt(vs>|jdjt*j,j.j0k(r|jd} ||f< |<|j3 jd!fd#fddi" "fd}|j D] t4j7 jd x}n j|j8} |sI|j; s[j7| f}||jd"|<|j< _ jt*j,j.j>k(r| jd} j&} | d } tAt| } tCt | } || | jtDjF}| jHd <A jt*j,j.j0k(r} jd} dd l%m&}|| \}}|Dcgc]} j&||}}|| j&| jNtDjP}| jHd < jtRvr< j&d }tUjV|}ddl,m-}ddl.m/}ta||r|jbjd}nta||rstgjhrHt*jjjl|jnjd}n9|jbjd}n"|jnjd}n tqd| j&d j&d|tDjr}tC j&} || d<tu|  _>t(j7 jd x} c j|j8}tw fd|Ds |s|D]}| f}|jd"|<|j< _ "jyD]:\ }|"vr "|}|"vr |" < j{||j} <y cc}w)a Reinplaces in-placeable operations. If there are no uses of a view of the mutated arg after the current node, it is possible to inplace the op. This above algorithm could be justified by observing side effects. While we traverse the graph in forwards direction, only latter nodes could view side effects of the current node. If the current node is not used later as well as no view of this node is used later in the graph, then it is safe to inplace as there would be no way to observe the side effects. This condition is slightly different for graph inputs where they can only be inplaced if the above condition is true and there's a copy_ in the epilogue that signals that the caller wants to observe the mutation. Unlike JIT Inductor, AOTInductor currently unlifts weights and buffers from input args, so instead of checking mutation on placeholder, AOTInductor checks mutation on get_attr. This is subject to change in future. r]rrrr=c2 |}| |nd} fd |D]}}|jD]l} |}||kr|||kr |r|jtjjj j ur||jdurkyy)Nct|jrtfd|jDS|jtvS)Nc3.K|] }|ywrXr)rluis_meta_only_users r"rmzqreinplace_inplaceable_ops_core..any_use_of_views_after_node..is_meta_only_user..sDA,Q/Ds)rrDallrx META_ONLY_OPS)r!rs r"rz^reinplace_inplaceable_ops_core..any_use_of_views_after_node..is_meta_only_users34;;'DDDD;;-/ /r$rTF)rxrDr+opsrerPrfr<) r!shared_view_nodes copy_nodernode_loc copy_node_locrTrvuser_locr node_orders @r"any_use_of_views_after_nodezCreinplace_inplaceable_ops_core..any_use_of_views_after_nodesd#1:1F 9-D  0 & D  %d+x'!,(1J$T*KK599>>#7#7#?#??#tyy|3-  0r$c dtfd}t|ttfr?t d|D}t |t |k7ryt fd|DSt|y t|}|Dcgc](}||jd|jdgr|*}}|jdvr# j|d}|y|||ryy td |Dry|d| Scc}w) NrMcP tt|dk7S#t$rYywxYw)NrT)rrr)lss r"_overlapzEreinplace_inplaceable_ops_core..can_inplace.._overlaps/ 6r:;q@@.  s  %%c32K|]}t|ywrXr)rlargs r"rmzFreinplace_inplaceable_ops_core..can_inplace..s(V3)9#)>(VsFc30K|] }|ywrXr)rlr can_inplacer!s r"rmzFreinplace_inplaceable_ops_core..can_inplace..sE#{4-Esr3rr)rrTc38K|]}|jdvyw)rrN)rwrks r"rmzFreinplace_inplaceable_ops_core..can_inplace..sVD77Vs) r.r5rrGrrrrr6rwrrn) r!rrunique_storagesrvrrr copy_nodesstorage_to_nodess ` r"rz3reinplace_inplaceable_ops_core..can_inplaces/ D  kD%= 1((V+(VVO?#s;'77EEE E K ( 0,-=k-JK ' ))%0!&&-@A   >>8 8"{D9I *'9+ VDUV V2'4[ ; s-Dc"d}d}|D]6}t|ttfr|D] } ||| z },|||z }8tj d|||||t j |t|t j||y)Nrc|jjdd}|]t|jtr?t|j tr!|j|j zSy)Nr3r)r6rr5 element_sizer*numel)r!ts r"byteszJreinplace_inplaceable_ops_core..log_inplace_results..bytessW eT*A q~~/5qwwy#.~~'!'')33r$zFor node %s, attempted to reinplace %s. We were unable to reinplace %s; %s (if non-empty) are possible missed reinplacing opportunities that may be bad for memory usage and performance. Total size of missed opportunities with static shapes is : %s bytes.) r5rrGloginfor add_missed_opportunitiesradd_missed_bytes) node_nameold_tensors_to_clonetensors_to_clone missed_args missed_nodestrigger missed_bytesrr!ns r"log_inplace_resultsz;reinplace_inplaceable_ops_core..log_inplace_results s  ! ,D$u .-A E!H,L-d +  ,        227C .reinplace_and_refine_tensors_to_clone..tensor_with_same_storage_already_reinplaced..Fs"JK$Q'+EEs)r5rrGrnr)rrrs r"+tensor_with_same_storage_already_reinplacedzreinplace_inplaceable_ops_core..reinplace_and_refine_tensors_to_clone..tensor_with_same_storage_already_reinplacedDs>#e}-OR$K04NN Nr$rr])rrr*rr<r AUTO_FUNC_V2rxrDoperatorgetitemr5rrGaddrappend)rr=rrrrrrrshould_attempt_reinplacerrvrrrrcopy_args_to_copy_nodesrr! replace_dicts @@r"%reinplace_and_refine_tensors_to_clonezMreinplace_inplaceable_ops_core..reinplace_and_refine_tensors_to_clone9s')%/c4i0@%A%C"   O(* -C&= = +K,W,( $(Kk,J377d8KL (.7nnQ.?L+"2"?"?? $ =  ;;(*:*::tyy|s?R1.sV#T{&==Vs)?rrrr enumeratereversedrrrrrDrerPrfr<rwrrr r=rr+r higher_orderauto_functionalizedrinplaceable_opsrrr%rauto_functionalized_v2rangerHr rr6*torch._higher_order_ops.auto_functionalizer_name AUTO_FUNC_V1inplaceable_triton_opsr get_kerneltriton.runtime.autotunerrtriton.runtime.jitrr5r;r&r is_fbcodeversionhipbase_fnAssertionError TRITON_OPSrritemsrr)$r/mutated_inputsidstrKrinplaceable_oprr _mutable_opr= all_basesbases_to_clonebase_tensors_dctnew_bases_to_clonerrrorrkernelrr kernel_name mutated_argsr replacementrrrrrr!rrrs$ @@@@@@@@@r"reinplace_inplaceable_ops_corer {s&!J_&N"4(!#JXekk23-4u{{+a/!3 4)$/077= ;;$**,, ,1E 2 ))A,C))A,CzzX---HHQK&&*JJ **84SXXa[ATYYq\QHHQK&&*AAHHQK&&%))*@*@*T*TThhqk26 #S#J /"JsO   tyy| ,1-4!F8t*BX8:LF F P _8-11$++tD DN Q))N$>$>?K4-.2L2LT2R477d8KL (.7nnQ.?L+,77 [[EII22II I))A,K[[F|,I"3y>2N#Ii$89 ,Q   -- - 5GDII0 1 [[EII22FF F))A,K S"2;"?  a, t{{1~/I    E  !! --   5EDII0 1 [[2 2\2J&11*=F : 6&+.$ii00 FI.##%}}((0&,nn&=&= &,ii&8&8 "(.."9"9K$%ABB E ./ H% ++   $++&F)9F% &(0DK599$++tL LN 99^%?%?@LVVV4.'@C 7d DI.7nnQ.?L+@-77 _8@*//1k\)&{3K\)( T "";/  } s Y Y ct5t|t|t|dddy#1swYyxYwrX)r rr r)r/s r"reinplace_inplaceable_opsr s5 ! #-%e,&u-%e,---s"6?)mr}loggingr collectionsr dataclassesrtypingrrrr+torch._C._dynamo.guardsrtorch._dispatch.pythonr torch._dynamo.utilsr r *torch._higher_order_ops.triton_kernel_wrapr r torch._inductorrrtorch._inductor.fx_utilsrrtorch._inductor.loweringr!inplaceable_foreach_ops_loweringstorch._inductor.virtualizedr%torch.fx.experimental.symbolic_shapesrtorch.fx.immutable_collectionsrtorch.fx.passes.reinplacer torch.utilsrr7torch.utils._ordered_setr getLoggerr&rrrerdiagonal_scatterrfdiagonalselect_scatterselectr* slice_scattersliceTensoras_strided_scatter as_stridedrrr_r,GraphrArCrrVr[r-r^rcrgrjr.rpr{rr index_put index_put__unsafe_index_put_unsafe_index_put_r_c10d_functionalc10d_functional all_reduce all_reduce_all_reduce_coalescedall_reduce_coalesced_inplaceable_collective_opsupdateAttributeErrorrHrErFr) outplace_oprrsym_size sym_stride sym_numelsym_storage_offsetrr r )krs00r"r=sC#!'' ?;C3G*M91)/g! yy~~ $EEE IINN##++UYY^^-D-D-L-L IINN!!))599>>+@+@+D+D IINN  ((%))..*>*>*E*E IINN%%--uyy~~/H/H/P/P  )<(A(A(CD1q!tD uxx~~     !LL48L \\&< <!LL<48L< \\< 88>>  6l   XX]] 0C 88>>C!&C XX]]C" 88>>!& XX]]@ *   Quxx}}QQ 588==T8D*d&d&Dd&P NNM$//*A*A1E""M.2S2SUV$W-$ , ii00O""**M  ' ' / /-  ,,44m  1 1 9 917 "56 GIejj33]BCHF@FFHHK+8Q+GK(H$%E$FG    ''  l%((..lTl^ -UXX^^--wEh    s W<B XX  X