Vh`&ddlmZddlZddlZddlZddlZddlmZmZm Z m Z ddl m Z ddl mZddlmZmZddlmZdd lmZerdd lmZdd lmZmZej6eZej<Gd d Zej<GddZ ej<GddZ! ddZ" ddZ# ddZ$ ddZ% ddZ& d dZ'd!dZ(d!dZ)e'e(e)gf d"dZ*y)#) annotationsN)Callable TYPE_CHECKING TypedDictUnion)signpost_event) OrderedSet)MultiOutputLayout NoneLayout)get_dtype_size)V)Dep)BaseSchedulerNodeSchedulerBuffercZeZdZUdZded<dZded<ejeZ ded<y) MemoryPlanningInfoForBufferrint size_alloc size_freedefault_factoryOrderedSet[BaseSchedulerNode] succ_nodesN) __name__ __module__ __qualname__r__annotations__r dataclassesfieldr rF/home/dcms/DCMS/lib/python3.12/site-packages/torch/_inductor/memory.pyrrs3JIs0A 0A0A"1J-r"rceZdZUdZded<dZded<ejeZ ded<ejeZ ded <ejeZ ded <y ) MemoryPlanningInfoForNoderrindexsizerz7OrderedSet[Union[SchedulerBuffer, FreeableInputBuffer]] pred_buffersr pred_nodesrN) rrrr&rr'rr r r(r)rr!r"r#r%r%"sqE3ND#M *5I1B 0A0A"1J-1B 0A0A"1J-r"r%cXeZdZUded<ej eZded<d dZd dZ y) FreeableInputBufferstrnamerr mpi_bufferc|jSN)r-selfs r#get_namezFreeableInputBuffer.get_name8s yyr"c,t|jSr0)hashr-r1s r#__hash__zFreeableInputBuffer.__hash__;sDIIr"N)returnr,)r7r) rrrrrr rr.r3r6r!r"r#r+r+1s. I.?k.?.?3/J+r"r+cdd}tjt}t}|D]{}|jj D]`}|j |vs|j jdr.||j j|||||j <b}t}|jD]"\}} t|t||| ||<$|S)z Create and keep track of all input buffers that can be freed during the program Returns: A dictionary containing all freeble input buffers, keyed by their names. cld} |js|j}|S#t$rY|SwxYw)Nr)has_unbacked_symbols numbytes_hintKeyError)depress r#_dep_size_hintz.get_freeable_input_buf.._dep_size_hintLsH ++-'')      s & 33)primals_arg)rr)r=rr7r) collections defaultdictr dict read_writesreadsr- startswithadditemsr+r) nodes graph_inputsr?dep_name_to_succ_nodesdep_name_to_sizenoder=name_to_freeable_input_bufdep_namers r#get_freeable_input_bufrQ?s   +(,vA##)) ACxx<'0C0C#1'sxx044T:-;C-@ *  AABF 6 < < > */B  '*84 0 "8,  &%r"cddlmddlmt  d dfd |j D]}|j vs|S)a Compute the size of each scheduler buffer, including (1) memory allocated when it is created and (2) memory deallocated when it is freed. We specially handle the case of MultiOutputLayout. Consider the following case: buf0 = some_ops_with_multi_outputs(...) buf1 = buf0[0] # assume 10 bytes buf2 = buf0[1] # assume 20 bytes In such cases, buf0: at creation, 30 bytes allocated, when deleted, 0 bytes freed buf1: at creation, 0 bytes allocated, when deleted, 10 bytes freed buf2: at creation, 0 bytes allocated, when deleted, 20 bytes freed Returns: A dictionary mapping a scheduler buffer to a tuple of (size_alloc, size_free). r ) MultiOutput) OutputNodect|jjtrd |j <yt|jjt rd}|j D][}t|jr|jjD]%}t|js||dz }']|rdn|df |j <|Stjjj|jjdt|jjz}|rdn||f |j <|S)N)rrrT)fallback) isinstancerNlayoutr r3r users get_outputsrgraphsizevars size_hint get_numelr get_dtype) sched_bufuser_of_MultiOutputLayoutruserbufbuf_sizerSrT_compute_and_update_buf_sizesched_buf_to_sizes r#rezGcompute_size_for_scheduler_buffer.._compute_and_update_buf_sizesL inn++Z 86< i002 3  --/@ AJ! Ndii499002NC!#((K8"&B3&MM N N/J7 i002 3 ww''11((*Q2y~~779:;H/H7 i002 3Or")F)r`rraboolr7r)irrS schedulerrTrDvaluesr3) name_to_bufr`rSrTrerfs @@@@r#!compute_size_for_scheduler_bufferrlrsz( %48FGL"?C :!'')4     '8 8 ( 3 4 r"c,t|}tjt}|D]1}|jD] }||j j |"3|jD]'}t||d||d||||_ )y)z For each SchedulerBuffer, assign its size info and successor nodes. A buffer's successor nodes determines when a buffer can be freed. rr )rrrN) rlrBrCr unmet_dependenciesr-rHkeysrr.)rJrkrfrLrNr=buf_names r#1assign_memory_planning_info_for_scheduler_buffersrqs:+F  +7** 7C "388 , 0 0 6 77  $$& +F(215'1!4-h7, H( r"cL ddlm t|D] \}}td|j D}t t tf}|jjD]j}|j|vr-||jvr|j||j>|j|vsM|j||jlt fd|D} t d|j D} t|||| | |_y)zL Assign to each scheduler node its predecessor and successor nodes. r )rc3HK|]}|jjywr0)r.r).0buffers r# zBassign_memory_planning_info_for_scheduler_nodes..sW&**55W "c3\K|]#}t|r|j%ywr0)rWdefining_op_name)rt pred_bufferrname_to_fused_nodes r#rvzBassign_memory_planning_info_for_scheduler_nodes..s1 ;8 {;;= > s),c3VK|]!}|jjD]}|#ywr0)r.r)rtru succ_nodes r#rvzBassign_memory_planning_info_for_scheduler_nodes..s9 #..99    s'))r&r'r(r)rN)rir enumeratesumrZr rr+rErFr-rnrHr%mpi_node) rJr{rkrOr&rNrr(r=r)rrs ` @r#/assign_memory_planning_info_for_scheduler_nodesrs+ ' tWDDTDTDVWW !%9L(L"MNP ##)) GCxx;&3$2I2I+I  SXX!6777  !;CHH!EF  G   +    **,  2%!!   % r"c tjGdd}tt|D] \}}||< g}|j D]\}}||vrt |dz n't fd|jjD} |j|||jj|jjd| t|D]\}}|jD]} | j|vrt |dz n1t | jjD cgc]} |  c} |} |j|| | jj| jj|| tt |dzD cgc]} d} } |D]G}| |jxx|jz cc<| |j dzxx|jzcc<Id}d}g}tt |dzD]'}|| |z }|j|t ||})||fScc} wcc} w)a  Given a list of nodes in their execution order, estimate the peak memory, by keeping track of the liveliness of SchedulerBuffers and FreeableInputBuffers. Returns: int: peak memory List[int]: memory usage at each node (or each step). c@eZdZUded<ded<ded<ded<ded<y) (estimate_peak_memory..BufferInfoz+Union[SchedulerBuffer, FreeableInputBuffer]rurrr start_stepend_stepNrrrrr!r"r# BufferInfors;; r"rr c3(K|] }| ywr0r!)rtr} node_to_steps r#rvz'estimate_peak_memory..s,5 Y'sr)default)r dataclassrDr~rIlenmaxr.rappendrrZr3rrangerr)rJrO graph_outputsrsteprN buf_info_listrp input_bufrr`r}_memorybuf_info max_memory cur_memorymemories_at_nodestrs @r#estimate_peak_memoryrs26L&" d! T"')M9??A )=( JN9B9M9M9X9X   $$..$$..    & & d))+ I %%'=8E Q*3)=)=)H)H%%Y/!    ((33((22   6s5zA~. /Aa /F /"<x""#x':'::#x  1$%););;%< JJ 3u:> "1fQi   ,Z0 1 ) **E$0s 9 I . IcGddt}Gddt}tt}t}|D]D}t|jj dd|<|ddk(s4|j |Ft|jt|jzD]=} dt| jj| j|vrd ndzi|| <?td |jDd} |D]D} | |vr| || jjz } $| |vs)| || jjz } Ft| |D]}|jjD]2} || dd k(s|d xx| jjz cc<4|j!D]2} || ddk(s|d xx| jjz cc<4g} d} | t|krV|rSt#|fd  }|j%|| j'|| d z } |jj(z t|d z|jjD]<}|ddkDsJ|dxxd zcc<|ddk(s,|j |>|jjD]j} || ddkDsJ|| dxxd zcc<|| dd k(s,| jjD]&}|d xx| jjz cc<(l| t|kr|rS| t|kDr t+d| S)a A bfs-based greedy topological order. LPMF stands for "Least Peak Memory First". The idea is from this paper: Buffer memory optimization for video codec application modeled in Simulink https://www.cs.york.ac.uk/rts/docs/DAC-1964-2006/PAPERS/2006/DAC06/PDFFILES/P0689.PDF The algorithm maintain the max memory so far. At every iteration, for each scheduleable node, it computes: - how much memory needs to be allocated for the output buffers of this node; - how much memory can be freed as a result of executing this node. This gives us two values for each node: (1) mem1: memory during the execution of the node; (2) mem2: memory after executing the node, after some input buffers are freed. The greedy approach select as follows: (i) if there are nodes whose mem1 values are below the max memory so far, then pick the node with the lowest mem2 value; (ii) otherwise, pick the one with the lowest mem1 value. c"eZdZUded<ded<y)'topological_sort_lpmf..NodeInforindegreememory_to_freeNrr!r"r#NodeInforps  r"rceZdZUded<y))topological_sort_lpmf..BufferInfor outdegreeNrr!r"r#rrtsr"rr)rrrrr c3HK|]}|jjywr0r.r)rtrs r#rvz(topological_sort_lpmf..s$  &&rwrct|jjz|jj|dz |jjfS)Nr)rrr'r&)rN live_memoryr node_infos r#z'topological_sort_lpmf..sLK$--"4"44jA ""Yt_5E%FF ##r"keyz4Failed to schedule, while loop ran too long for lpmf)rrDr rrr)rHlistrjr.rr3rrrr(rZminremoverr' RuntimeError)rJrOrkrrrrnodes_to_schedulerNrc output_memoryrpschedule num_iters selected_noder}rrrs @@@r#topological_sort_lpmfrVs49Y486INRfH8B|(DMM445 $ T?: &! +  ! !$ ' (K&&()D1K1R1R1T,UU  S^^667LLNm3q<   3::<KM!W { " [2==GG GM 3 3 7ALLVV VM W [-0JN==-- NC}[)Q.$ 01S^^5M5MM1 N##% NC}[)Q.$ 01S^^5M5MM1 N N)+HI c%j %6     / &Q  }--222 [1 y/0@AA '//:: 1IY' 3a7 77 i  , 1 ,#J/14!%%i0  1!))66 WCC=-1 11 SM+ &! + &}[)Q.!$!:!:WIi()9:cnn>V>VV:W  W7 c%j %6D3u:QRR Or"c  Gddt}t tjGdd}d fd }g}|D]V}t |j j dd |< |dd k(s4tj|||||Xg}d }|t |kr|rtj|j}t | |d <|j||d z }|j jD]N} | dd kDsJ | dxxd zcc< | dd k(s,tj|||| | P|t |kr|r|t |kDr td |S)a A BFS topological sort that selects nodes whose dependencies are executed the earliest. This follows a FIFO idea. Specifically, at every iteration, for each node that is schedulable, we gather the order in which its predecessor nodes are executed, and this sorted list of execution orders of predecessor nodes defines the priority. We select the node whose predecessors nodes are executed the earliest. The FIFO idea aims to reduce the liveness duration of buffers created. c"eZdZUded<ded<y)&topological_sort_bfs..NodeInforrorderNrr!r"r#rrs   r"rc*eZdZUded<ded<ddZy).topological_sort_bfs..NodeWithPriority list[int]priorityrrNc|j|jk(rA|jjj|jjjkS|j|jkSr0)rrNrr&)r2others r#__lt__z5topological_sort_bfs..NodeWithPriority.__lt__sP}}.yy))//%**2E2E2K2KKK==5>>1 1r"N)rNodeWithPriorityr7rg)rrrrrr!r"r#rrs 2r"rc|ddk(sJttfd|jjD}|S)Nrrc3.K|] }|dyw)rNr!)rt pred_noders r#rvz?topological_sort_bfs.._node_priority..s 2; )$W-s)sortedr rr))rN exec_ordersrs r#_node_priorityz,topological_sort_bfs.._node_prioritysKz*a/// ?C}}?W?W  r")rrrrrr z3Failed to schedule, while loop ran too long for bfs)rNrr7r)rrDrrrrr)heapqheappushheappoprNrrr) rJrrrrrNrrrr}rs @r#topological_sort_bfsrs9486I22213'*4==+C+C'DrR $ T?: &! + NN!#3N44H$#O )+HI c%j %6 &78== ,/M - ) &Q '//:: IY' 3a7 77 i  , 1 ,#J/14%$^I%> J   c%j %6"3u:PQQ Or"cnttgtdfd |D]}|jD]}||< |D]B}|jjt d|jj Dz|<Dt|fdD] }| S)a This is a DFS topological sort. The setup is similar to `topological_sort_schedule` in scheduler.py. The difference is the order nodes are visited in the outer loop. In `topological_sort_schedule`, nodes are visited in their original order. In this function, nodes are visited based on their priority -- for each node, we compute the total memory of all buffers it reads from or writes to, and we visit the nodes in ascending order of this priority. c|vrtj||jDcgc]}|jvr|j!}}t|fdD] }| j |yycc}w)Nc:||jjfSr0rr&nsize_with_readss r#rz5topological_sort_dfs..visit..2s/!*N>N)Or"r)rHrnr-rr) rr= dep_nodesrN name_to_noderesultseenrvisits r#rz#topological_sort_dfs..visit)s D= HHQK//88|+SXX&I O d   MM!  s$A;c3HK|]}|jjywr0r)rtpred_bufs r#rvz'topological_sort_dfs..<s!9 .6H   ) )9 rwc:||jjfSr0rrs r#rz&topological_sort_dfs..?s_Q-?AQAQ,Rr"r)rrr7None)r rDget_buffer_namesrr'rr(r)rJrNr-rrrrrs @@@@@r#topological_sort_dfsrs+5,D15L&(F48FO  &))+ &D!%L  &&  $ 2 2S9 :>--:T:T9 6 !  u"RS d  Mr"c tjdt|tjGdd}t ||}t ||t||||g}t|||\} } |j||| dtjd| |D]} | tk(r | ||||} n| |} t| t|k(sJt| ||\} } |j|| | | jtjd| j| td d d |Dcic]}|j|j c}i t#|d }|j$S#t$r,}tjd| j|Yd}~ d}~wwxYwcc}w)z Try a few heuristics based topological sort algorithms, and pick the one whose resulting topological order has the lowest peak memory estimation. z&Reordering for peak memory -- %d nodesc,eZdZUded<ded<ded<y)1reorder_for_peak_memory..PeakMemoryResultlist[BaseSchedulerNode]rr peak_memoryr,methodNrr!r"r#PeakMemoryResultrXs&& r"rbaselinezBaseline peak memory: %dz%s peak memory: %dzFailed to reorder for %s: %sNinductorrorm)categoryr- parametersc|jSr0)r)xs r#rz)reorder_for_peak_memory..s ammr"r) torch_loginforrrrQrqrrrrr Exceptionerrorrrrrr)rJrkr{rKrmethodsrrOpeak_memory_diff_methodsestimated_peak_memoryrrrreelem best_results r#reorder_for_peak_memoryrEs"NN;SZHBX |B6e[I3 !;0J 8: 4 )= 1## 5zBNN-/DEP P..5{Mu u:U+ ++11=NK % + +  V__E  NN/+ NP&  >VWdDKK!1!11W .4KLK    P OO:FOOQ O O PXs*B FF; F8 !F33F8)rJrrKOrderedSet[str]r7dict[str, FreeableInputBuffer])rkdict[str, SchedulerBuffer]r7zdict[str, tuple[int, int]])rJrrkrr7r) rJrr{dict[str, BaseSchedulerNode]rkrrOrr7r)rJrrOrrrr7ztuple[int, list[int]]) rJrrOrrkrrrr7r)rJrr7r)rJrrkrr{rrKrrrrz,list[Callable[..., list[BaseSchedulerNode]]]r7r)+ __future__rrBrrloggingtypingrrrrtorch._utils_internalrtorch.utils._ordered_setr rhr r utilsr virtualizedr dependenciesrrirr getLoggerrrrrr%r+rQrlrqrrrrrrr!r"r#rs" <<0/-!!= G  h '          0& "0&!0&$0&f<+<<~ " +   <# "# 4# ,# !? #  # L^+ "^+ >^+#^+ ^+Bz "z >z,z# z  zzEP'b = N "N+N5N" N # N : NNr"