Vh&HddlZddlZddlZddlmZddlmZddlmZddl m Z eje Z ejdZGddeZeZGd d eZeZGd d eZeZGd deZeZGddeZeZy)N)Optional) warning_once)HigherOrderOperator)_dtypec&eZdZdfd ZdZxZS)Wrapc$t|dy)Nwrapsuper__init__self __class__s L/home/dcms/DCMS/lib/python3.12/site-packages/torch/_higher_order_ops/wrap.pyrz Wrap.__init__s  cBddl}ddlm}|fd}|S)Nrdisableci}|SN)resultargsfunckwargss rwrapperzWrap.__call__..wrappers4*6*FMr torch._dynamor)rrrrtorchrrs ``` r__call__z Wrap.__call__s' )    yrreturnN__name__ __module__ __qualname__rr" __classcell__rs@rr r s ! rr c&eZdZdfd ZdZxZS)WrapWithSetGradEnabledc$t|dy)Nwrap_with_set_grad_enabledr rs rrzWrapWithSetGradEnabled.__init__(s 56rcJddlddlm}|fd}|S)Nrrcbj5icdddS#1swYyxYwr)set_grad_enabled)r enable_gradrr! wrapped_funcsrrz0WrapWithSetGradEnabled.__call__..wrapper1s3'' 4 5#T4V4 5 5 5s%.r)rr2r3rrrrr!s ```` @rr"zWrapWithSetGradEnabled.__call__+s' )  5  5yrr#r%r*s@rr,r,'s 7 rr,c BeZdZfdZdedeededeefdZxZ S)WrapWithAutocastc$t|dy)Nwrap_with_autocastr rs rrzWrapWithAutocast.__init__=s -.r device_typedtypeenabled cache_enabledc V ddl ddlm}| fd} | S)Nrrchj5icdddS#1swYyxYwr)autocast)rr;r8r9r:rr!r3srrz*WrapWithAutocast.__call__..wrapperOs7 UG]K 5#T4V4 5 5 5s(1r) rr8r9r:r;r3rrrrr!s ``````` @rr"zWrapWithAutocast.__call__@s, )  5 5  5yr) r&r'r(rstrrrboolr"r)r*s@rr5r5<s</   ~ rr5c*eZdZdZdfd ZdZxZS)WrapActivationCheckpointa This operator is used to wrap torch.utils.checkpoint. This avoids TorchDynamo to look into saved tensor hooks and directly passes the control to AOT Autograd, which is ok with tracing saved tensor hooks. As a result of AOT tracing torch.utils.checkpoint code, we have a backward graph with recomputed forward nodes. However, we might deprecate this operator soon. The difficulty arises in the functionalization of rng ops. Today, there are two different functionalization of rng ops - one at AOT autograd and other at Inductor. And they are difficult to map to each other. The rng states also complicate pattern matching in Inductor. Due to the ease of implementation, we are currently inclined towards functionalization at Inductor level, which means that duplication/recomputation is done as a compiler pass in the partitioners. See TagActivationCheckpoint for more information. c(t|ddy)Nwrap_activation_checkpointF cacheabler rs rrz!WrapActivationCheckpoint.__init__ls 5Grcddlmcm}ddlm}d|d<d|d<|j 5ddlm}|||jg|i|cdddS#1swYyxYw)Nr InterpreterF use_reentrantpreserve_rng_state checkpoint) torch.fx.tracebackfx tracebacktorch.fxrIpreserve_node_metatorch.utils.checkpointrMrun)rfunctionrr fx_tracebackrIrMs rr"z!WrapActivationCheckpoint.__call__osg 21("'',#$  , , . J 9k(377I$I&I J J Js "AAr#)r&r'r(__doc__rr"r)r*s@rrBrBZs"H JrrBc@eZdZdZdfd ZedZdZdZxZ S)TagActivationCheckpointa This operator is supposed to be used only with torch.compile stack. This accepts a Fx graph module which needs to be checkpointed. This operator adds "recomputable" tag to the nodes of the Fx graph that should be recomputed. The goal is to: 1. Avoid using Dynamo to trace through saved tensor hooks. 2. For selective checkpointing case, let AOTAutograd trace through saved tensor hooks but has special logic with TorchDispatchMode to override the usual saved_tensor_hooks fn logic in order to tag the nodes. 3. Rely on the partitioners to actually duplicate the nodes. This sits well in the torch.compile stack, because by the time graph reaches partitioner, inductor has already run its functionalization of rng ops (by setting fixed seed for each random op, see `replace_random_passes`). Therefore, the duplication of nodes, by design, respects the rng states in the forward and recomputed forward in backward. c(t|ddy)Ntag_activation_checkpointFrEr rs rrz TagActivationCheckpoint.__init__s 4Frcpddlm}tj|}t }|j D]}|dvr|j ||j d|jDcic] }||vs|||}}|jDcic] }||vs|||}}||fScc}wcc}w)a  checkpoint fn can have mixed kwargs between checkpointed fn and checkpoint fn itself. For example >> def gn(x, y, z=None): >> a = torch.matmul(x, y) >> if z is not None: >> return torch.matmul(a, z) >> return a >> def fn(x, y, z): >> return torch.cos(checkpoint(gn, x, y, use_reentrant=False, z=z)) In the above case, z belongs to checkpointed function gn, but use_reentrant belongs to the checkpoint function. This function splits the kwargs into checkpoint_kwargs and gmod_kwargs (or checkpointed_fn_kwargs). We do sorting to ensure same graph from run to run for better debuggability. It is not required for correctness. rrL)rUrrrK)rSrMinspect signatureset parametersaddkeys)rrMckpt_signaturecheckpoint_keysnamecheckpoint_kwargs gmod_kwargss r divide_kwargsz%TagActivationCheckpoint.divide_kwargss& 6 **:6%"-- &D55    % & 01,2;;= #'DO-N-NDIIk* O rcddlmcm}ddlm}d|j vr{t tdd|d<d|d<|j d|d<|j|d }|j5dd l m }|||jg|i|cdddS|j|d }|j5||j|cdddS#1swYyxYw#1swYyxYw) NrrH_checkpoint_context_fnz Detected that context_fn is passed to torch.utils.checkpoint under torch.compile. Please make sure the checkpointed region does not contain in-place ops (e.g. torch.relu_). FrJrK context_fnT)rxrL) rNrOrPrQrIrurlogr{rRrSrMrT)rrwrrrVrIrMs rr"z TagActivationCheckpoint.__call__s11( #tyy 0  ',F? # ,1F' (#'99-E#FF< >>$t>4D002 J=!+d"3"7"7I$I&I J J >>$u>5D 002 4,{4(,,d3 4 4 J J 4 4s-"C<C'C$'C0r#) r&r'r(rWr staticmethodrhr{r"r)r*s@rrYrYs-$G$.$.L &4rrY)r] itertoolsloggingtypingrtorch._loggingr torch._opsr torch.typesr getLoggerr&rcountrqr r r,r.r5r7rBrDrYr[rrrrs'*g!iooa $ v0$45*6&'"J2"JJ67r41r4j45r