Vh_=ddlZddlZddlmZddlmZmZmZddlZddl m Z m Z ddl m Z ddlmZej eZdej&d ee dffd Zd ee d ee d eee ee ffd Zd ee d ee ee ffdZdee dee d eeeeffdZdeej&deeej&deej&dee d eeeej&dfeeeefff dZ ddee deeeefd eeej&dffdZ ddeeed eeej&dffdZ dZ!y)N)Iterator)AnyOptionalUnion) GradientEdgeNode) Parameter)map_debug_infotreturnc|jrH|j<|j|}|j}||jddSt d|jS)z Get the grad function or grad accumulator for a tensor. Accumulate grad nodes are lazily created, so we need to a dummy view in order to trigger its creation. rzRAttempted to get grad_fn, but got None.Is this being created in a no-grad context?) requires_gradgrad_fnview_asnext_functions RuntimeError)r viewed_trs V/home/dcms/DCMS/lib/python3.12/site-packages/torch/distributed/pipelining/_backward.py_get_grad_fn_or_grad_accrsf 199,99Q<""  ))!,Q/ />  yyroots target_nodesc|t}t}tj}|D],}|||vs |j||j |.|r^|j }||}|D]A}||vs| ||vr|j| |j||j |C|r^||fS)a This function returns the reverse closure of the given roots, i.e. the set of nodes that can be reached from the roots by following the reverse edges of the graph. The target_nodes are the nodes that we want to include in the closure. )set collectionsdequeaddappendpopleft) rrreverse_edges_dictclosurevisited_target_nodesqnode reverse_edgesfns rreverse_closurer((sG5!,!2!2!4A  G 3 KK  HHTN yy{*40  BW} \!$((, KKO HHRL   ( ((rctj}t}tjt}|D],}|||vs |j ||j |.|r`|j}|jD]>\}}| t||dk(r|j |||j |@|r`|S)Nr) rrr defaultdictlistrrr rlen)rr$ root_seenr!r%r'_s rconstruct_reverse_graphr/Gs!,!2!2!4A5I1<1H1H1N   I 5 HHTN MM$   yy{(( 4EB~)"-.!3HHRL"2&--d3  4  rinputsparamsc$t|t|\}}t}|D]p}t|g||\}}|h|d} |D]R} |j| d} | 7| dj | d| d<| dj | d| d<| } N| || <Trt} t} g}|j D]O} t | | vs| jt | |j| | j | d} Q|S)a Given a list of inputs and a list of parameters, return a list of parameter groups, where each group contains the parameters and the intermediates that are connected to the parameters. The returned list of parameter groups is a list of dictionaries, where each dictionary contains the following keys: - "params": a set of parameters - "intermediates": a set of intermediates The returned list of parameter groups is a list of dictionaries, )r1 intermediatesNr1r3) r(rdictgetunionvaluesidrr)r0r1r!inputs_closurer. param_groupsparamr" intersected param_group input_nodeexisting union_paramsseen_idsunique_param_groupss rget_param_groupsrCYsL"(7IJNA/3vL7. G^%7 g(' & 7J#'' D9H#%-h%7%=%=k(>S%T",4_,E,K,K0-)' +6 Z( 77("eLH#**,E k?( * LLK )  & &{ 3'--k(.CDL E rstage_outputs_or_loss output_grads input_valuesweights.c ttdtt|}ttdtt|}ttdtt|}t |}t |||}g} |D]D} t | dD]1\} } d} | j| | | }| j|3F|"|Dcgc]}tj|}}tjj|||d}t |D]4\} }|j || |_ |xj|| z c_ 6|D]}|j| D]}|j||fScc}w)aj Compute the gradients for only the stage inputs with respect to the stage outputs (if non-last stage) or loss (if last stage) After computing input gradients, we save the intermediate nodes in `param_groups` for later use in stage_backward_weight. We don't need to save any other intermediate nodes that aren't needed for dW because when we do dW calculation, we start from saved intermediates. Detaching the stage_outputs_or_loss at the end of this function is important as it frees up the memory that the autograd graph is anticipating to be used later (but doesn't actually need). Nr3cfd}|S)Ncdjdddgtdzd<|d<y)Ngradsr3)r5r,) grad_inputsir=s rhookz4stage_backward_input..get_hook..hooksE"w5=04v'890 G,/:K(+r)r=rMrNs`` rget_hookz&stage_backward_input..get_hooks: rT)r0 grad_outputs retain_graph)r+filtermaprr/rC enumerateregister_prehookrtorch ones_likeautogradgraddetach_remove)rDrErFrGstage_output_grad_fnsstage_input_grad_fnsweight_grad_fnsr!r:handlesr=rM intermediaterPhandle stage_outputdinputsinpr s rstage_backward_inputrfs )-tS13HIJ)(,tS1<@A(#'tS17;<#O11FG#o/ALG## (_)EF #OA| "228K3KLF NN6 " ##$?T .:EOOL )  nn!!! "GL)#3 88 qzCH HH "H ##   L  7 sFr:c <i}g}t|D]2\}}t|}||f||<|j|j4|D]}t d|dD} t d|dD} |d=t d|dDsJt jj| | t|dt |} |d=t|d| D]6\}} ||\}}|j| |_"|xj| z c_8t |S)Nc34K|]}t|dywrNr).0rMs r z(stage_backward_weight..s# #$LA # r3c34K|]}t|dywrirj)rkws rrlz(stage_backward_weight..sPQl1a0Prmr1c38K|]}t|dk(yw)r N)r,)rkgs rrlz(stage_backward_weight..s=13q6Q;=srK)rQrR) rUrrrZtupleallrWrYsumzip) rGr:rRgrad_acc_to_weight weight_gradsindexweightgrad_accr=intermediate_edges weights_edgesdweightsdws rstage_backward_weightrsS13L"7+) v+F3'-u}8$FKK() $ " "# (3O(D#  P+h:OPP   (= G(<====>>&&  [157;% '    H 5x@ "LHb.x8MFE{{"   r!  "7 "D  routputs_with_grads_idxsc  |$|Dcgc]}|| }}|Dcgc]}|| }} g g fd}||||tjj g}|D]I}t|tjr|j |j 9|j dK t|Scc}wcc}w#t$r8}dt|dt|dt|d} t| |d}~wwxYw)a This is a helper function to: 1. compute the gradients for the stage inputs, and 2. accumulate gradients for the stage module's parameters. Given the input value(s) and the corresponding gradient for the output value(s), compute and accumulate gradients for all parameter values (leaves in the autograd trace) as well as return a list of the gradients for the input values Nc.t|tjrt|js |jyt|tjt dfsJdt |j |j |yt|ttfrp|yt|ttfsJdt |dt |t|t|k(sJt||D]\}}||||yt|trp|yt|tsJt|jt|jk(sJ|jD]}||||||yy)Nz)Expected Tensor or None gradient but got z!grad_value expected to have type z but got ) isinstancerWTensorrrtyperrrr+r,rur4rkeys) output_valgrad_valextract_tensors_with_gradsovgvkoutput_grad_tensorsstage_output_tensorss rrz2stage_backward..extract_tensors_with_grads,s *ell3!//J4F4F4N!(U\\4:,FG?X?OPG%++J7#**84J 6#!(UDM:7Z8H7ISWX`SaRbc::#h-777!*h7FB.2 J-#!(D111:??,-X]]_1EEEE#*A."1 x{4N r) grad_tensorsz= Failed to run stage backward: Stage output: z Output gradient: z Input: z ) rWrYbackwardrrrrZ Exceptionr rrr) rcrErFrrMrrLvaleexc_msgrrs @@rstage_backwardrsB *1HIA QI I1HIA QI I\+46<>& ` # ,(B   ,  57  )C#u||,""388,""4(  ) (  CJIn +%l345(678|,-.   7#*+s" B; CBC D3DDc>||S||Stj||S)z] Coalesce two values, even if one of them is null, returning the non-null value. )rWr)lhsrhss r_null_coalesce_accumulaters)  {  yyc""r)F)N)"rloggingcollections.abcrtypingrrrrWtorch.autograd.graphrrtorch.nnr _debugr getLogger__name__loggerrrr+rrrr(r4r/strrCrfrintrrrOrrrs$'' 3"   8 $ tTz1B,) :)%(Y) 3t9c$i  )>4:$tT$Z7G2H$3 J3 $T 3 $sCx.3lO! -O!4 -.O!u||$O!i O!  5%,,', -tDcN/C CD O!fTY- i -04T#s(^0D- 8ELL !3 &'-h48 s&d3i0 s  8ELL !3 &' sp #r