Vht' ddlZddlmZddlmZmZddlZddlZddl m Z m Z m Z m Z ddlmZddlmZmZGddeZGd d eZGd d eZGd deZdej.j0de dededej2def dZGddeZGddZdedefdZdededefdZ dededefdZ!d%d e"d!edefd"Z#d#ede$e"e%ffd$Z&y)&N) OrderedDict)cast TypedDict) _MemRefType _ModMemStats _ModState MemTracker)RuntimeEstimator) SACEstimatorSACTradeOffStatscNeZdZUeeed<eeed<eeed<eeed<y)ModOrder fw_pre_order bw_pre_order fw_post_order bw_post_orderN)__name__ __module__ __qualname__liststr__annotations__R/home/dcms/DCMS/lib/python3.12/site-packages/torch/distributed/_tools/ilp_utils.pyrrs*s)s)99rrc"eZdZUeed<eed<y) ModRuntimefwbwN)rrrfloatrrrrrrs  I IrrceZdZUeed<eed<eed<eed<eed<eed<eed<eed<eed <eed <eed <eed <eed <eed<eed<eed<eeed<eeed<eeed<e eefed<y)ModStatsfqnparam_per_modulegrad_per_module grad_totalact_fw_per_moduleact_bw_per_moduleact_grad_per_module act_totalinput_per_moduleoutput_per_modulefw_runtime_per_modulebw_runtime_per_moduleis_leaf sac_runtime sac_memory n_segmentsslopes intercepts breakpointstradeoff_curveN) rrrrrintr boolrrrrrr"r"s HON    MOO KU eu --rr"c(eZdZUeed<eeed<y) ModuleInfo mod_order mod_statsN)rrrrrrr"rrrr:r:IsH~rr:model mem_trackerruntime_estimator sac_estimatordevreturnc ttj|j}|jj Dcic]\}}||d|dd}}}t |jt |jt |jt |jd} |jtj|j} | gd} |jD]} |j| dx} s| j| jdx}rW|j }|j"}|j$}|j&}|j(}|j*}|j,}d}ndx}x}}gx}x}}t/}d }id | jd | j0d | j0d | j2t4j6d|t8j:dt=d| j2t4j>d|t8j@| j2t4jBd|t8j@z | jDz dt=d| j2t4jFd|t8j@d| j2t4jFd|t8jH| j2t4j6d|t8jHz d| j2t4j>d|t8j@d| jJd| jDd|| jdd|| jdd|d|d|d|d||||d}| djM|| Scc}}w)a Collect modulewise stats for a given model, including memory, runtime, and AC tradeoff stats. Args: model: nn.Module object runtime_estimator: RuntimeEstimator object with runtime stats mem_tracker: MemTracker object with memory stats sac_estimator: SACEstimator object with AC tradeoff stats dev: device the model was run on (used to extract memory stats from MemTracker) Returns: ModuleInfo: A dictionary with module order and module stats. rr)rr)rrrr)r;r<NFrTr#r$r%r&r'r(r)r*r+r,r-r.r/r0r1r2r3)r4r5r6r<)'dictcopydeepcopymemory_tracking mod_runtimesitemsrmod_fw_pre_ordermod_bw_pre_ordermod_fw_post_ordermod_bw_post_orderpwlf_sac_tradeoff_curvesac_mod_tradeoff_statsmodulesgetmod_fqnr0r1r2r3r4 fit_breaksr6r parameter_mem snapshotsrPRE_BWrGRADmaxPOST_FWACTPRE_FW output_memPEAK_BWTEMP input_memappend)r=r>r?r@rA mod_mem_statsr#vmod_runtime_statsr;mod_sac_tradeoff_stats module_infomod mod_mem_stattradeoff_statsr0r1r2r3r4r5r6r/mod_stats raggregate_statsrkNs9,:> k112:M(44::<0 C AdG1T7 ++00.??@.??@/AAB/AAB I))+:>--,,; K }}76(,,S$7 7< 7!7!;!;L!>899 9j:4666k$>&""<#=#=&"l44Y5E5EFrJ3O$$ &"$S **9+<+<=bA#F{W",,Y-=-=>rB3G XY"--.&&"$S **9+<+<=bA#F{W&&""& **9+<+<=bA#F{GWGWX",,Y-=-=>rB3G#((%&".\33I4E4EFrJ3OOO/&"4#L$:$:5&"6$\%<%<7&"8():<;O;O)PQU)V9&":():<;O;O)PQU)V;&"<7=&">{?&"@jA&"BjC&"D&E&"F)*"0K&"HN  $ + +H 5o76r c0sOc"eZdZUeed<eed<y)Nodeindexpos_fw_post_orderN)rrrr7rrrrrmrms Jrrmc,eZdZdeddfdZdeddfdZy)GraphnrBNcfg|_i|_tj||f|_g|_y)N)nodes name2nodenpzeros ad_matrixr)selfrrs r__init__zGraph.__init__s,!# *,1a&)(*rnodec^|jj|||j|d<yNr#)rtraru)ryr{s radd_nodezGraph.add_nodes& $&*tE{#r)rrrr7rzrmr~rrrrqrqs(+#+$+ +T+d+rrqrfc6|d}|ddt|tk(sJt|}t|}|dd|_t|fd|d<t |D]L\}}t t |}||d<|jj|d|d <|j|Nt|D]S}t||D]B}t|j|d|j|drd |j||<BSU|S) z Parse module info and create a graph (tree) of modules. The graph will be used by MILP solver to find optimal SAC and/or FSDP configurations. r<r;rrc,j|dSr})rn)xrs rz#parse_module_info..s!3!3AeH!=r)keyrnr#ro) lenrqrsorted enumeraterrmrnr~rangeis_self_or_submodulertrx) rfr<n_nodesgi one_mod_statsr{jrs @rparse_module_infors6 K(I{+N;L y>S. .. .)nG gA!+.?AO &= K &i0=$ .W $%OO$9$9$u+$F ! 4 7^q'" A#AGGAJu$5qwwqz%7HI$% Aq!   Hrname_descendant name_ancestorc||k(xs|dz|vS)z[ check if name_descendant is a submodule of name_ancestor, or if they are the same .rrrs rrrs m + U}s/Bo/UUrc|dz|vS)zN if name_descendant is a submodule of name_ancestor, but not the same rrrs r is_submodulers 3 / 11rbunitcb|dk(r |dz ddS|dk(r |dz ddS|dk(r |d z dd S|dd S) zN return a string that represent the number of bytes in a desired unit KiBiz.2fz KiBMiBiz MiBGiBi@z GiBz bytesr)rrs r display_bytesrsa u}e)C%% u}e)C%% u}e)C%%WF rgraphc\|jdd}t|j}d}t|D]M}|j|d}|j|d}|j|d}t|||z|z|z}O|jdd|jddz}||fS)aV Get the baseline peak memory and runtime. Baseline here means there is no FSDP or AC. Memory includes the parameters, gradients, activations, and activation gradients. Memory does not include e.g., optimizer states, embedding tables, etc. Returns: int: peak memory in bytes float: compute time in ms rr$r&r)r*r-r.)rtrrrY) rP_1 num_nodespeak_memrTG_iAG_iTA_i compute_times r get_peak_memory_runtime_baseliner s ++a.+ ,CEKK IH 9 ;{{1~l+{{1~34{{1~k*xtd!2T!9: ;  A./ ++a.0 1 2 l ##r)r)'rF collectionsrtypingrrnumpyrvtorch$torch.distributed._tools.mem_trackerrrrr *torch.distributed._tools.runtime_estimatorr &torch.distributed._tools.sac_estimatorr r rrr"r:nnModuledevicerkrmrqrrr8rrr7rtupler rrrrrsA #"  HQy (.y(.V l 88??ll(l l  l  l^8 + +! :! %! HV#VcVdV2#2c2d2 S   $E$eCJ6G$r