VhddlmZddlZddlZddlZddlZddlmZmZm Z ddl Z ddl Z ddl m Z mZddlmZe r ddlmZmZmZeeefZ ej4Z eeeeeffZ eeefZ GddZGd d Z d dd Z!dd Z"Gd dejFZ$Gdde$Z%Gdde$Z&Gdde jNZ(y)) annotationsN)AnyFinal TYPE_CHECKING)_pass diagnostics)_pytree) GeneratorIteratorSequenceceZdZUdZded<ded<ded< ddZedd Zedd Zedd Z edd Z edd Z ddZ ddZ ddZeddZe ddZe ddZe ddZy) _ModuleMetaaMeta information about a module. This class is used to represent the module information in a more structured way. It parses raw module information from a single item from `node.meta["nn_module_stack"].items()`. See the uses of `from_raw_meta`, `from_fx_tracer_produced_raw_meta`, and `from_dynamo_produced_raw_meta` for how to create an instance. Attributes: _module_class: The class of the module. E.g. `torch.nn.module.sparse.Embedding`. _module_name: The name of the module. E.g. `L__self___h_1_mlp_c_proj`. _raw_meta: The raw meta '(module_name, node.meta["nn_module_stack"][module_name])'. zFinal[type | str | None] _module_classz Final[str] _module_namezFinal[tuple[Any, Any]] _raw_metac.||_||_||_yN)rrr)self module_name module_classraw_metas ]/home/dcms/DCMS/lib/python3.12/site-packages/torch/onnx/_internal/fx/passes/modularization.py__init__z_ModuleMeta.__init__3s ()!c@|j}|jd}|S)zHThe display name of the module. E.g. `h_1_mlp_c_proj`. L__self___)r removeprefix)rnames rmodule_display_namez_ModuleMeta.module_display_name=s$  . rc|jy|j}t|tr|jdz|jz}|j ddS)z^Qualified name of the module class. E.g. `torch_nn_module_sparse_Embedding`. ._)r isinstancetype __module__ __qualname__replace)rmod_clss rqualified_module_class_namez'_ModuleMeta.qualified_module_class_nameHsT    %$$ gt $((3.1E1EEGsC((rc|jyt|jtr|jjS|jS)z=Name of the module class. E.g. `Embedding`. r!)rr$r%__name__rs rmodule_class_namez_ModuleMeta.module_class_nameUsA    % d(($ /%%.. .!!!rc|jS)zFName of the module. E.g. `L__self___h_1_mlp_c_proj`. )rr-s rrz_ModuleMeta.module_nameas    rc|jS)zrReturns the raw module meta data. I.e. (module_name, node.meta['nn_module_stack'][module_name]). )rr-s rrz_ModuleMeta.raw_metais ~~rct|tsy|j|jk(xr|j|jk(SNF)r$rrrrothers r__eq__z_ModuleMeta.__eq__qsA%-   !3!3 3 :""e&9&99 rcDt|j|jfSr)hashrrr-s r__hash__z_ModuleMeta.__hash__ysT&&(:(:;<rrrs r from_fx_tracer_produced_raw_metaz,_ModuleMeta.from_fx_tracer_produced_raw_metas %-! \; h??rcP|\}\}}t|jdd||S)z@Create a module meta from raw meta produced by FX dynamo tracer.@r)rsplit)r>rr_qualified_namers rfrom_dynamo_produced_raw_metaz)_ModuleMeta.from_dynamo_produced_raw_metas3 8@4 4o|;,,S1!4lHMMrcHt|tr6t|dk(r(t|dtrtj |St|tr6t|dk(r(t|dtrtj |Stdt|)NzIUnknown type of raw meta item from node.meta['nn_module_stack'].items(): )r$tuplelenr%rrArF TypeError)r>rs r from_raw_metaz_ModuleMeta.from_raw_metas x 'H "8A;-??I I x 'H "8A;.< ), 'L__self___h_1_attn': ( "L['self'].h[1].attn", ) } If produced by fx.symbolic_trace: { 'h.1': , 'h.1.attn': } zFinal[list[_ModuleMeta]] _module_stackcg|_|ytj|}|jD]+}|rd}|jtj |-yr2)rbcopyitemspushrrM)rnn_module_stack_metais_exported_programritems rrz_ModuleStackMeta.__init__s_   ' 9912NN$ 7D#&+# IIk//5 6  7rc,t|jSrrKrbr-s r__len__z_ModuleStackMeta.__len__s4%%&&rc |j|Srrb)rindexs r __getitem__z_ModuleStackMeta.__getitem__s!!%((rc,t|jSr)iterrbr-s r__iter__z_ModuleStackMeta.__iter__sD&&''rc2t|jdk(SNrrkr-s ris_empty_or_rootz!_ModuleStackMeta.is_empty_or_roots4%%&!++rch|jrtjS|jdS)zReturns the top module meta in the stack. I.e., the meta for leaf module. Example: Consider the following module stack: stack = [GPT, block1, Attention_1, MLP] stack.top() == MLP )rvrr?rbr-s rtopz_ModuleStackMeta.tops0  "**, ,!!"%%rc|jry|jyt|t|kryt|D]\}}|||k7syy)aDetermines if self is a superset of the provided module stack. I.e., If self includes all elements from the provided module stack, plus additional elements on top. If self is empty or root, this method always return False. Example: Consider the following module stack: stack_1 = [GPT, block1, Attention_1, MLP] stack_2 = [GPT, block1] stack_1.is_superset_of(stack_2) == True stack_2.is_superset_of(stack_1) == False stack_3 = [GPT, block2, Attention_1] stack_1.is_superset_of(stack_3) == False stack_3.is_superset_of(stack_1) == False FT)rvrK enumerate)r module_stacki parent_keys ris_superset_ofz_ModuleStackMeta.is_superset_ofse0  "  ( ( * 2 t9L) )&|4 MAzAw*$ rc:|jj|y)z"Pushes a module meta to the stack.N)rbappendr module_metas rrfz_ModuleStackMeta.push s !!+.rcVt|tsy|j|jk(Sr2)r$rarbr3s rr5z_ModuleStackMeta.__eq__$s'%!12!!U%8%888rcv|jDcic]}|jd|jd!c}Scc}w)zJReturns the raw module stack meta data, i.e. node.meta['nn_module_stack'].rrI)rbrrs rrz_ModuleStackMeta.raw_meta)sE $11    #[%9%9!%< <   s$6c"d|jdS)NzModuleStackMeta(r:rnr-s rr;z_ModuleStackMeta.__repr__1s!$"4"4!5Q77rc6|jjS)z2Returns the module display name of the top module.)ryrr-s rrz$_ModuleStackMeta.module_display_name4sxxz---rc6|jjS)z:Returns the qualified module class name of the top module.)ryr*r-s rr*z,_ModuleStackMeta.qualified_module_class_name9sxxz555rc6|jjS)z+Returns the module class of the top module.)ryrr-s rrz_ModuleStackMeta.module_class>sxxz'''rN)T)rgzO_FX_TRACER_NN_MODULE_STACK_META_TYPE | _DYNAMO_NN_MODULE_STACK_META_TYPE | NonerhrUrV)rorWrRr)rRzIterator[_ModuleMeta])rRrUrX)r|rarRrU)rrrRNonerS)rRz"dict[str, tuple[str, type]] | NonerQ)rRrO)r,r&r'r[r\rrlrprsrvryrrfr5r]rr;rr*rr_rrraras8,+%) 77 " 7&')(, &%&% %N/9   8..66((rracNt|jjd|S)Nnn_module_stackrh)rametaget)noderhs r_module_stack_meta_from_noderDs%  '(>Q rcT|j|d||xxdz cc<|d||S)NrrIr#) setdefault) module_namesrs r_get_unique_module_namerLs9K+"]!L56 77rcpeZdZdZeej ddZeej ddZy)_IRNodearBase class for IR nodes. IR nodes are used for Modularize pass only. They add a layer of abstraction on top of torch.fx.Node. [NOTE: Modularize Pass Implementation] The main job of the pass is to group `fx.Node`s that belong to the same `nn.Module` forward call, and then create `call_module` node and sub `fx.GraphModule` from them. Each `fx.Node` possesses an `nn_module_stack` meta data that contains information about the module call stack. See `_ModuleStackMeta` for examples. Analysis step ------------- Each module call is identified by a set of base stack layers. For each module call, the pass creates a `_ModuleNode` and groups the sequence of nodes that shares the same base stack layers. For example, stack_of_node_0 = [GPT, block0] stack_of_node_1 = [GPT, block1] stack_of_node_2 = [GPT, block1, Attention1, MLP] stack_of_node_3 = [GPT, block1, Attention1] stack_of_node_4 = [GPT, block2] All nodes belong to the `GPT` module call, since they share the base stack layers [GPT]. [node_1, node_2, node_3] are grouped for `GPT.block1`, because they share the base stack layers [GPT, block1]. And [node_2, node_3] for `GPT.block1.Attention1`, [node_0] for `GPT.block0`, and [node_4] for `GPT.block2` respectfully. After the analysis step, a hierarchical representation is generated. For above example, the representation is: _ModuleNode(GPT) _ModuleNode(block0) _LeafNode(node_0) _ModuleNode(block1) _LeafNode(node_1) _ModuleNode(Attention1) _ModuleNode(MLP) _LeafNode(node_2) _LeafNode(node_3) _ModuleNode(block2) _LeafNode(node_4) Construction step ----------------- The second step is to build the actual `call_module` node and the sub `fx.GraphModule`. This is done recursively from the leaf `_ModuleNode` to the root. For example, the first submodule to be built is `GPT.block1.Attention1.MLP`. Below pair is generated from `_ModuleNode(MLP)`. fx.GraphModule(GPT.block1.Attention1.MLP) graph: node_2 new_mlp_node = `call_module[GPT.block1.Attention1.MLP](...)` Next, the `GPT.block1.Attention1` submodule is built. Below is generated from `_ModuleNode(Attention1)`. fx.GraphModule(GPT.block1.Attention1) graph: new_mlp_node node_3 new_attention1_node = `call_module[GPT.block1.Attention1](...)` Until every submodule is built, the new modularized `fx.GraphModule` is generated. Alternatives ------------ The current algorithm adopts a top down approach. A bottom up approach is similar. In contrast to these two, an alternative flat order approach is also possible, where each node is traversed and copied to the corresponding submodule. The advantage of the current approach lies in the encapsulation of the fx.GraphModule construction for each individual submodule within a single `build_module` method, which can be called separately once the analysis phase is completed, making debugging more convenient. Regarding construction step, an alternative implementation is to utilize `fx.Interpreter` for traversing all the nodes under the flattened root module and copying the nodes into their respective submodule under construction. This approach is not adopted because 1. It uses the flat order approach discussed above. This means one cannot individually construct a submodule and examine it while debugging. 2. The graph execution functionality of `fx.Interpreter` is not necessary for the purpose of this pass. Ignoring that, `fx.Interpreter.run` achieves the same effect as a for loop over all the nodes. cy)z5The module stack meta data associated with this node.Nr_r-s r stack_metaz_IRNode.stack_meta rcy)z*The stack trace associated with this node.Nr_r-s r stack_tracez_IRNode.stack_tracerrNrRrarRz str | None) r,r&r'r[r]abcabstractmethodrrr_rrrrRsM`D    rrceZdZdZ ddZeddZeddZddZddZ ddZ ddZ dd Z dd Z dd Zdd Zy ) _ModuleNodeaRepresenting a sequence of fx.Nodes to be formed into a fx.GraphModule. This class encapsulates metadata and provides building block methods to construct this layered abstraction from a sequence of flat fx.Nodes. Attributes: - _stack_meta: Metadata of the module stack. - _nodes: List of IR nodes in the module. - _reference_root_module: Reference to the root flat fx.GraphModule instance. c.||_g|_||_yr) _stack_meta_nodes_reference_module)rreference_root_modulers rrz_ModuleNode.__init__s&%' !6rc|jSrrr-s rrz_ModuleNode.stack_metasrcP|jsJ|jdjSru)rrr-s rrz_ModuleNode.stack_traces"{{{{{1~)))rc"d|jdS)Nz ModuleNode(r:rr-s r__str__z_ModuleNode.__str__sT--.a00rc4|j|jk(S)zIDetermines if the provided node pertains to the same module as this node.)rrrs ris_same_module_asz_ModuleNode.is_same_module_ass$//11rcL|jj|jS)zHDetermines if this node represents a parent module of the provided node.)rrrs ris_parent_module_ofz_ModuleNode.is_parent_module_ofs--doo>>rc ,|j|s|jdk(r|jj|y|jdk(r|jj|y|j |r|jr|jdnd}t |t r4|j |s|j|r|j|ytj|j}|j|jt|jt |j|}|jj||j|ytd|d|jd|jd) zAdds a leaf node to the module. The leaf node must belong to the same or a child module. This method will recursively construct _ModuleNode instance based on the stack_meta information of the leaf node. call_module placeholderrxNzNode z (z) does not belong to module r")rfx_oprrrr$r add_leaf_noderddeepcopyrrfrKrAssertionErrorr)r leaf_node last_noders rrz_ModuleNode.add_leaf_nodesM  ! !) , =0P KK  y ) __ - KK  y )  % %i 0,0;; BDI)[1--i8..y9'' 2"]]4??;  4 4S5I JK'**  ""9-'' 2  {"Y%9%9$::V##$A' rc#K|jD]K}t|tr|jEd{,t|tsJ|j My7(w)zEReturns an iterator for the sequence of fx nodes this instance holds.N)rr$rfx_nodes _LeafNodefx_noders rrz_ModuleNode.fx_nodessOKK #D$ ,==?**!$ 222ll"  #*s3A A)A cJt|j}t|dkDsJdit|dfd }|D]B}t j ||j t j ||jDtjS)aExtract module inputs from the sequence of fx nodes this instance holds. All node args that are produced by nodes outside of the module are considered module inputs. The order of returned module inputs is the same as the their use order. ### Known limitations The original ordering of module inputs is not preserved. There is no meta information to be found from the `fx.GraphModule` that can be used to recover the original ordering. Returns: Sequence of module inputs. r.Cannot extract module inputs from empty nodes.cdt|tjjr |vrd|<yyyr)r$torchfxNode)arg module_inputsnode_sets r#_extract_arg_if_node_outside_modulezF_ModuleNode.module_inputs.._extract_arg_if_node_outside_module1s/#uxx}}-#X2E%) c"3F-r)rr) listrrKsetpytreetree_mapargskwargskeys)rnodesrrrrs @@rrz_ModuleNode.module_inputssT]]_%5zA~OOO~35 '*5z * ND OO? K OO? M NM&&())rct|j}t|dkDsJdi}t||D]&}t fd|j Ds"d||<(t|j S)a!Extract module outputs from the sequence of fx nodes this instance holds. All nodes that are used by nodes outside of the module are considered module outputs. The order of returned module outputs is the same as the their creation order. ### Known limitations The original ordering of module outputs is not preserved. There is no meta information to be found from the `fx.GraphModule` that can be used to recover the original ordering. Returns: Sequence of module outputs. rrc3&K|]}|v ywrr_).0userrs r z-_ModuleNode.module_outputs..Os?D4x'?sN)rrrKranyusersr)rrmodule_outputsrrs @rrz_ModuleNode.module_outputs:szT]]_%5zA~OOO~46'*5z ,D?DJJ??'+t$ ,N'')**rcV|jj}tjj }id fd |j }|D]R}|j |j|j|<tj|j|_ T|jD]b}t|tr#|j}|j||<7t|t sJ|j#|}|j } |j%} t'||j(j*} |j,j/| ||j1| t3fd| D} t5| dkDrt3d| D| jd<t7| D]\} }|j9t:j<| | f|j}tj|j|_ tj|jd|jd<|jdj?||<n/| | d <tj| d j| _ |j@x}|| jd <|j(jB}|Jtj|| jd<| jdj?e|jD}tGtItK|jLd k7rT|j%} |j%Dcgc]}| }}|jOt5|dk(r|d n|}tjjQ|j,||}|jjRx}?tUjVtTjXj[||jd <|Scc}w)ag Constructs the fx.GraphModule for this node, registering submodules as necessary. Args: module_names: A dictionary of module names and their counts. This is used to generate unique module names for submodules. This should be an empty dictionary when the method is called on a root module. c|Srr_)rcopy_envs r_arg_transformz0_ModuleNode.build_module.._arg_transform`s D> !r) arg_transformc3.K|] }|ywrr_)rrrs rrz+_ModuleNode.build_module..sLtnT*LsrIc3RK|]}|jjd!yw)valN)rr)r ref_outputs rrz+_ModuleNode.build_module..s$33=JOO''.3s%'r)r type_exprrrroutputonnx)r torch.fx.NoderRr).rr*rrGraphrrrr%rdrrr$rr node_copyr build_modulerrrrr add_submodulerrJrKr{ call_functionoperatorgetitempopitemrrrnextrrreversedopr GraphModulerrGraphModuleOnnxMeta PackageInfofrom_python_class)rrr.fx_graph ref_inputsrir_noder submoduleref_submodule_inputsref_submodule_outputsunique_submodule_namesubmodule_noder}r getitem_noderraw_module_stack_meta new_nodes new_outputs graph_modulerrrs @@rrz_ModuleNode.build_moduleSs!,,HH88>>#79 "'')  7D%11$))TYYGHTN"&))DII"6HTN  7{{> =G'9-!//$,$6$6>%7%!g{3 33,,\:I#*#8#8#: $+$:$:$< !$;g00DD% !  " " 0 01F R&11%L7KLLN()A--23AV3.##E*&//D%E8MAz#+#9#9 ((,a0",//$:$L )- *//(BL%<@99"(9:<L%%&78!%%&78@@B+7HZ(8"6D.q12&*ii0Ea0H0M0M&N# '222 ?5@##M2$+$6$6$?$? !(4 4459YY?T5UN   1 2    1 2 : : <}> =@NN  Xi() * - - 9$($7$7$9 !BFBUBUBWXJ8J/XKX??"%k"2a"7 A[Dxx++  " "H.? !,,99 9L F(-(A(A!!33LA)L  f %Ys0 P&N)rtorch.fx.GraphModulerrarrrQ)rrrRrU)rrrRr)rRz$Generator[torch.fx.Node, None, None])rRzSequence[torch.fx.Node])rdict[str, int]rRr)r,r&r'r[rr]rrrrrrrrrrr_rrrrsn 7%97GW7  **12?*X#*8+2frrcjeZdZdZd d dZed dZed dZed dZeddZ d dZ y)rzRepresenting a single fx.Node.c6||_t|||_y)Nr)_noderr)rrrhs rrz_LeafNode.__init__s 7 &9 rc.|jjS)z!Syntax sugar for self.fx_node.op.)rrr-s rrz_LeafNode.fx_opszz}}rc|jS)z-Returns the fx.Node this instance represents.rr-s rrz_LeafNode.fx_nodeszzrc|jS)z=Returns the module stack meta data associated with this node.rr-s rrz_LeafNode.stack_metasrcL|jjjdS)z2Returns the stack trace associated with this node.r)rrrr-s rrz_LeafNode.stack_traces||  $$]33rc"d|jdS)Nz LeafNode(r:r r-s rrz_LeafNode.__str__s4::,a((rNF)rrrhrUrQ)rRrrr) r,r&r'r[rr]rrrrrr_rrrrs_(   44)rrc:eZdZdZ d dfd ZddZxZS) ModularizeaKTransforms a flattened `fx.GraphModule` into a modular structure. In the flattened `fx.GraphModule`, each `nn.Module` forward call has been traced as a sequence of `fx.Node`s. All these `fx.Node`s are flattened and reside in the same `fx.GraphModule`. `fx.GraphModule` could be from `torch.export.ExportedProgram` or directly generated by `torch._dynamo.export` with torch.nn.Module. This pass generates a new `fx.GraphModule`. It groups the flattened `fx.Node`s that belong to the same `nn.Module` forward call into a sub `fx.GraphModule`. It then replaces the sequence of flattened `fx.Node`s with a single `call_module` node, which is linked with the sub `fx.GraphModule` by `node.target`. The sub `fx.GraphModule` is registered as a submodule of the new `fx.GraphModule`. The process is done based on information from the `nn_module_stack` metadata of each node, i.e. `node.meta["nn_module_stack"]`. For more implementation details, see [NOTE: Modularize Pass Implementation]. An fx submodule under this context can typically be interpreted in three different ways: 1. As an embodiment of an nn.Module class, which is considered stateless. Its execution path can vary depending on the configuration of module initialization, which should also be part of the inputs. 2. As a representation of an nn.Module instance. It maintains the state initialized in the module. The execution path can vary based on actual input data. 3. As a captured call of an nn.Module instance, where the execution path is set. The generality decreases along this list. Within the scope of this function, the pass creates fx submodules according to the third interpretation. The first interpretation is the most general case. It requires complex analysis and additional metadata and code information to construct its general form. Consider an example nn.Module that generates arbitrary submodules based on an initialization configuration file. It's impractical to extract this logic for the generated fx submodule to function with arbitrary configuration. The second interpretation demands less analysis and is sturdier than the first. In most use cases, it's equivalent to the third. It only differs in exceptional situations where a complex nn.Module instance is called multiple times, each with a different set of inputs leading to a unique execution branching path. The third interpretation is the most specific scenario. It necessitates the minimum analysis and creates the most stable representation. The drawback is that it generates more redundancy than the other two methods. If needed, a subsequent post-processing pass can be applied to consolidate completely identical functions and reduce duplication. ### Known constraints Two successive calls to the same module instance will be conflated. They are indistinguishable. This is due to limitations of the current fx metadata "nn_module_stack". [NOTE: Modularize pass ordering] This pass groups fx nodes into subgraphs that reside within the `call_module` fx node. Other fx passes (including some outside the exporter) might not recognize `call_module`. They may assume that all nodes are flattened. Hence it is recommended to invoke this pass as the last pre onnx export fx pass. If not for this consideration, this operation could potentially be relocated anywhere earlier in the pipeline. Example: >>> # xdoctest: +REQUIRES(env:TORCH_DOCTEST_ONNX) >>> import torch >>> from torch.onnx._internal.fx import passes >>> from torch.onnx._internal.diagnostics import infra >>> >>> class CustomModule(torch.nn.Module): >>> def __init__(self) -> None: >>> super().__init__() >>> self.embedding = torch.nn.Embedding(10, 32) >>> self.relu = torch.nn.ReLU() >>> >>> def forward(self, x): >>> out = self.embedding(x) >>> out = self.relu(out) >>> return out >>> >>> class TestModule(torch.nn.Module): >>> def __init__(self) -> None: >>> super().__init__() >>> self.layer = CustomModule() >>> self.linear = torch.nn.Linear(32, 10) >>> >>> def forward(self, x): >>> out = self.layer(x) >>> out = self.linear(out) >>> return out >>> >>> gm, _ = torch._dynamo.export(TestModule(), aten_graph=True)( ... torch.tensor([0, 1, 2]) ... ) >>> gm.print_readable() >>> gm = passes.Modularize( ... infra.DiagnosticContext("test_context", "1.0"), ... gm, ... ).run() >>> gm.print_readable() cBt|||||_||_yr)superrmodulerh)rdiagnostic_contextrrh __class__s rrzModularize.__init__As$ +V4 #6 rc|jjjtjj |j|jj}t |td|j}|jjjD](}|jt||j*|jiS)N)rgrhr) rgrapheliminate_dead_coderrrrrarhrrrr)rreference_moduleroot_module_noders r_runzModularize._runKs --/ 88// T[[=N=NO&  %)t?W?W   {{((.. G  * *'t7O7OP   ,,R00rr )rzdiagnostics.DiagnosticContextrrrhrU)rRr)r,r&r'r[rr __classcell__)rs@rrrs5aN%* 797%7" 71rrr )rrrhrUrRra)rrrrNrRrN)) __future__rr collectionsrdrtypingrrrrtorch.fxtorch.onnx._internal.fxrr torch.utilsr rcollections.abcr r r rJrNr%rY OrderedDict$_FX_TRACER_NN_MODULE_STACK_META_TYPErZdict!_DYNAMO_NN_MODULE_STACK_META_TYPErrarrABCrrr Transformrr_rrr)s"  ,, 6)=="'sDy!1e'2'>'>$S#CsDy)9$9:\$(.I)I$J!JJ J ZU(U(r6; .28 m cggm `w'wt))B11r