Vh*ddlZddlZddlZddlZddlmZddlmZmZddl m Z m Z m Z m Z mZddlZddlmZmZmZmZmZmZmZmZmZddlmZe rddlmZdd lmZdd l m!Z!m"Z"gd Z#ejHe%Z&Gd d eZ'Gdde(Z)Gdde)Z*Gdde)Z+dddde,de e-de e-fdZ.e'j^e._/e'j`e._0e'jbe._1dddde,de e-de e-de2e)dffdZ3ejhGddZ5ejhGdd e5Z6ejhGd!d"Z7ejhGd#d$e5Z8ejhGd%d&e5Z9ee6e8e9fZ:d'e:d(e e-e-ge;d)fd*d+dedd4d5e de fd6e d7e d8e e,de f d9Z?dDdeeeD z_Dim.__add__..T QYr(typeintNotImplementedErrorr _deriveclsr=s `r)__add__z _Dim.__add__MsK ;c !%#E7$s||n=__ {{.//r(c ||zSr:r'rEs r)__radd__z _Dim.__radd__V U{r(ctturtdd|jd|j fdS)NzAttempted to subtract z from r8c|z Sr:r'r;s r)r>z_Dim.__sub__..`r?r(r@rEs `r)__sub__z _Dim.__sub__YsL ;c !%(vcll^D__ {{.//r(c4td|jd)NzAttempted to negate zN. (Only increasing linear operations with integer coefficients are supported.))rCr rEs r)__rsub__z _Dim.__rsub__bs&!"3<<.1[ [  r(cttusdkrtdd|jd|j fdS)NrzAttempted to multiply z with zu, where a positive integer was expected. (Only increasing linear operations with integer coefficients are supported.)c|zSr:r'r;s r)r>z_Dim.__mul__..or?r(r@rEs `r)__mul__z _Dim.__mul__hsR ;c !UaZ%(vcll^D__ {{.//r(c ||zSr:r'rEs r)__rmul__z _Dim.__rmul__qrJr(cPddlm}t|||jS)Nr)sympify)sympyrVstrr )rFfnrVs r) _derived_namez_Dim._derived_namets!2gcll+,--r(cJt|j|tf||dS)NrootrY) _DerivedDimrZrB)rFrYs r)rDz _Dim._deriveys#3,,R03&3b:QRRr(N)r r!r"r# staticmethodr6rGrIrMrOrRrTrZrDr'r(r)r+r+8sD 8 800 0. Sr(r+c0eZdZdZedZedZy) _StaticDimz Meta class for static :func:`Dim` types. This class is only for setting and checking static dim constraints, and the user should never interact with it. c|jSr:valueselfs r)minz_StaticDim.min zzr(c|jSr:rcres r)maxz_StaticDim.maxrhr(N)r r!r"r#propertyrgrjr'r(r)rara}s/r(rac6eZdZdZedZedZdZy)r^a Metaclass for derived :func:`Dim` types. Currently we only support increasing linear expressions with integer coefficients. In other words, a derived Dim can always be written in the form Ax + B, where x is a regular Dim (i.e., non-derived Dim), A and B are integers, and A is positive. (In particular, the latter ensures that x < y => Ax + B < Ay + B.) These restrictions on the form of derived Dims makes the metatheory simpler: e.g., it simplifies computing ranges for derived Dims, solving for underlying regular Dims, deciding equalities between derived Dims, and so on. The function lambda x: Ax + B is expressed by `fn`, where x is a normal Dim, `root`. The range of a derived Dim is computed by mapping `fn` over the range of its `root`. c>ddlm}ddlm}|jj | ur| S|j ||jj }|j}|dk\s/Jd|jd|jd|j dt|S)NrIntegerr-zExpected derived min value of z9 to be >= 0. Please specify an appropriate min value for (currently ).) rWror2r.r]rgrYr rB)rfror. _min_symintr]s r)rgz_DerivedDim.mins "5 99==VG #7Ngggdiimm45 yya ,T]]O<;;?==/J((2 '  ;r(c ddlm}ddlm}|jj |ur|S|j ||jj }|j}|tjdz ksCJd|jdtjdz d|jd|j d t|S) Nrrnr-rzExpected derived max value of z to be <= z.. Please specify an appropriate max value for rprq) rWror2r.r]rjrYsysmaxsizer rB)rfror. _max_symintr]s r)rjz_DerivedDim.maxs "5 99==F "Mgggdiimm45 yyckkAo- ,T]]O:ckkTUoEVW;;?==/J((2 ' - ;r(cjtjtfjfddS)Nc2j|Sr:)rY)r<rYrfs r)r>z%_DerivedDim._derive..s4771:r(r\)r^rZrBr])rfrYs``r)rDz_DerivedDim._derives2   r " FYY&> ?  r(N)r r!r"r#rkrgrjrDr'r(r)r^r^s4   &  & r(r^rgrjr3rgrjc&ddlm}|dn|}||n|}||kDs Jd|d||js Jd|t|tf||d}t t jt jdddd |_ |S) an :func:`Dim` constructs a type analogous to a named symbolic integer with a range. It can be used to describe multiple possible values of a dynamic tensor dimension. Note that different dynamic dimensions of the same tensor, or of different tensors, can be described by the same type. Args: name (str): Human-readable name for debugging. min (Optional[int]): Minimum possible value of given symbol (inclusive) max (Optional[int]): Maximum possible value of given symbol (inclusive) Returns: A type that can be used in dynamic shape specifications for tensors. rr-z(Cannot create Dim with inconsistent min=r1z)Dim name must be a valid identifier, got ryrr __main__) r2r. isidentifierr+rBgetattrinspect getmodulestackr!)r3rgrjr._min_maxdims r)rrs 2 1D[6cD $;SB3%vcUSS;    R"KD6 RR  tcVT$7 8C'--/!,Q/0*jCN Jr(namesreturn.c0tfd|DS)zh Util to create multiple :func:`Dim` types. Returns: A tuple of :func:`Dim` types. c3:K|]}t|yw)ryN)r).0r3rjrgs r) zdims..s?Ts,,?s)tuple)rgrjrs`` r)rrs ?? ??r(c&eZdZUdZeed<eed<y)_ConstraintTargetz2 This represents input tensor dimensions. t_idrN)r r!r"r#rB__annotations__r'r(r)rrs I Hr(rc\eZdZUdZeed<ded<d dZdZdZd Z d Z d Z e d Z y) _Constraintz This represents a Dim describing a constraint target. `name` is the name of the Dim. `constraint_range` contains the min/max bounds of the Dim. r3rconstraint_rangeNcddlm}ddlm}ddlm}||}||j j|||zd}t|j|j|j|S)Nrrr- ValueRangeslowerupperFvr warn_only) %torch.fx.experimental.symbolic_shapesrr2r.torch.utils._sympy.value_rangesrrrrrrr3)rfrrrr.rrs r)_clone_with_rangez_Constraint._clone_with_rangeseP5? =E1$$''+E*OO  II HH II    r(c&|j|S)Nrrrfrs r)__ge__z_Constraint.__ge__)%%E%22r(c,|j|dzS)Nrrrrs r)__gt__z_Constraint.__gt__,%%EAI%66r(c&|j|S)Nrrrfrs r)__le__z_Constraint.__le__/rr(c,|j|dz S)Nrrrrs r)__lt__z_Constraint.__lt__2rr(ctd)NzCannot determine truth value of _Constraint. If you are trying to combine _Constraint's with logical connectives, you can specify them separately instead.) TypeErrorres r)__bool__z_Constraint.__bool__5s  7  r(c|j|j|jjj|jjj dSN)rrrgrjrrrrrrres r)serializable_specz_Constraint.serializable_spec@sFII88((++11((++11   r()rN)r r!r"r#rXrrrrrrrrkrr'r(r)rr sF I.. (3737      r(rc0eZdZUdZeed<ded<eed<y) _PhantomRoota This represents the root of a derived Dim where the root does not directly specify the shape of any input dimension, but the derived Dim does. e.g., the input shapes 2*dim and dim + 1 are related via a "phantom" dim. The fields `name`, `constraint_range`, and `val` carried by a phantom root help create a symbol for it. Any derived dims with this phantom root are backed by expressions over this symbol. r3rrvalN)r r!r"r#rXrrBr'r(r)rrQs  I.. Hr(rcTeZdZUdZeed<ded<eeefed<e ed<e dZ y) _DerivedConstraintaC This represents a derived Dim, whose root is either a regular constraint target (which directly specifies the shape of some input dimension) or a phantom root (which does so indirectly). It can be thought of as a subclass of `_Constraint`, except that it does not support <, <=, >, >= operations. r3rrr]rYc|j|j|jjj|jjj dSrrres r)rz$_DerivedConstraint.serializable_specssFII88((++11((++11   r(N) r r!r"r#rXrr rrrrkrr'r(r)rrcs< I.. !</ 00L   r(rc eZdZdZedZy)_RelaxedConstrainta This represents a dim marked with Dim.AUTO/DYNAMIC (i.e. mark_dynamic() or maybe_mark_dynamic()), which leaves relations & min/max ranges for inference, instead of requiring explicit specification. The intention is for constraint violations to not be raised if produce_guards() finds equalities or relations between a _RelaxedConstraint and another type of _Constraint. c4|j|jdS)Nrrrres r)rz$_RelaxedConstraint.serializable_specsII88  r(N)r r!r"r#rkrr'r(r)rr~s  r(r constraint get_sourcesr shape_envr source_pairs)rrderived_equalities)rrphantom_symbolsrrelaxed_sourcesc||j|j}|sy|^} |jfd| Dt|trj|j |vr6||j \} } || | } |jfd| Dy|j|jf||j <yt|t rit|jts5||jj|jjd} n|jj |vr||jj } n|j|jjtjjj|jj tjj j"j$j&|jj(} | ||jj <|j*} |j-| | fyt|t.r|j1yy)z Updates `source_pairs`, `derived_equalities`, and `phantom_symbols` (which become fields of `EqualityConstraint`) based on a given input `constraint`. Nc3&K|]}|f ywr:r'r other_sourcesources r)rz&_process_equalities..sQ<.Qc3&K|]}|f ywr:r'rs r)rz&_process_equalities..s +7& rr)rr dynamic_dimconstraint_dim)rrextend isinstancerr3rr]r create_symbolrtorch_dynamorConstantSourcefx experimentalsymbolic_shapes DimDynamicr&rrYappendradd)rrrrrrrrsources other_sources shared_t_id shared_dimr]rYrs @r)_process_equalitiesrs*//:>>:G $F]Q=QQ*k* ??e #&+JOO&< #K' Z@M    ;H  '1ooz~~%FE*// " J 2 3*//<8z33Z__5H5HI!LD##6&z';';<!.."++ ==//>>z?S?ST % 5 5 E E P P X X#-??#C#C / 9=  4 45 ]] !!64"45 J 2 3F# 4r( tree_namefunctreedynamic_shapesrc0 d fd t |g|d iS#t$ri}d|jdvrR|sJdsJdd fd  fd  t| \}}|D]}t| \}} ||gd }~wwxYw) a" Customized tree_map for mapping pytrees to dynamic_shapes. For built-in types (e.g., standard collections) this behaves exactly like tree_map. OTOH for a user-defined class C registered with pytree, we cannot assume that a C containing tensors can be mapped to a C containing dynamic shapes (i.e., C may not be a polymorphic container). In that case we use the flattened form of C instead. Thus a C(**tensors) that flattens to (**tensors) will map to (**dynamic_shapes). Args: func: function to apply to each (int, float, str, bool, None, torch.Tensor) tree: input pytree dynamic_shapes: zero or more (typically one) dynamic_shapes to match Returns: output pytree mapping func to each (int, float, str, bool, None, torch.Tensor) c$t|tvSr:)r r )ts r)is_leafz$_tree_map_with_path..is_leafs a 55r(ct|}|tvr)tt|j|dg|diS||g|S)Nrr)r rr flatten_fn)pathrrtypfrrs r)rz_tree_map_with_path..fsgQ / !&$//215   a1.1 1r(rmismatchrz2Cannot be a mismatch if there is no dynamic_shapesz7Must provide a tree_name when there might be a mismatchc|turt||S|ttfvr|Jt |St d|)NzDid not expect type )dictrlistrrAssertionError)type_contextis r)_keyz!_tree_map_with_path.._keysKD=%gaj11T5M)"?*?&q>)$';E7%CDDr(cJddlm}m}||jdd|d)Nr UserError UserErrorTypez,Detected mismatch between the structure of `z` and `dynamic_shapes`: dynamic_shapes_validation case_name)torch._dynamo.excrr INVALID_INPUT)msgrrrs r)raise_mismatch_errorz1_tree_map_with_path..raise_mismatch_errors2F!//B9+Mefiejk9r(c t|}t|tryt|tr d |d|jd|d|j|jk7r* d |d|jd|d|j t |j t |j k7r= d |dt |j d|dt |j d |jt urt|jt|jk7r* d |d|jd|d|j t t|j|j }|jDcgc]}|| }}n |j }tt|j |D]2\}\}}  || | |j|j|gz4ycc}w) N`z` is a z, but `dynamic_shapesz` is notz` has z elements, but `dynamic_shapesz elementsz ` has keys ) r rrrAlenchildren_specsrsortedrzip enumerate)rrr rendered_path_remapkdynamic_shapes_children_specsrtree_dynamic_shapes__comparerrrs r)rz%_tree_map_with_path.._compares#!'t dH-nh7(I;}oWTYYKH..;_HF99 3 33(I;}oWTYYKH..;_GNDWDWCXZt**+s>3P3P/QQ(I;}oVC@S@S WW W9 E ) V(g>LAy, 9 %1*g%F"?OR8 9 QHs# BA$BBc>t|tr|j}|szt|tjj rt j|jnt j|}||ni}|j|i|jS|Sr:) rrmodulernnModuler~ signatureforwardbind arguments)rrkwargs_is_torch_jit_tracers r) _combine_argsr#Is!_% HHJ !UXX__-   aii (""1%  "-2y~~t.v.888 Kr(c0eZdZdZdZdZdZdZddZy) ShapesCollectionaD Builder for dynamic_shapes. Used to assign dynamic shape specifications to tensors that appear in inputs. This is useful particularly when :func:`args` is a nested input structure, and it's easier to index the input tensors, than to replicate the structure of :func:`args` in the :func:`dynamic_shapes` specification. Example:: args = ({"x": tensor_x, "others": [tensor_y, tensor_z]}) dim = torch.export.Dim(...) dynamic_shapes = torch.export.ShapesCollection() dynamic_shapes[tensor_x] = (dim, dim + 1, 8) dynamic_shapes[tensor_y] = {0: dim * 2} # This is equivalent to the following (now auto-generated): # dynamic_shapes = {"x": (dim, dim + 1, 8), "others": [{0: dim * 2}, None]} torch.export(..., args, dynamic_shapes=dynamic_shapes) ci|_yr:)_shapesres r)__init__zShapesCollection.__init__ps  r(ct|tjsJdt|t |}||j vr"|j |}||k(s Jd|d|y||j t |<y)Nz'Cannot assign shape to non-tensor type z1Shapes assigned to tensor do not match: expected z, got )rrTensorrAidr')rfrshaper_shapes r) __setitem__zShapesCollection.__setitem__ss u||  ? 4T!WI > ? !u 4<< \\$'F YB6(&QVPWX Y#(DLLA r(cpt|}||jvri|j|<|j|Sr:)r+r')rfrrs r) __getitem__zShapesCollection.__getitem__s4!u t|| #!#DLL ||D!!r(c,t|jSr:)rr'res r)__len__zShapesCollection.__len__s4<<  r(Nctfd}t|||}t||}tfdjDr t d|S)zu Generates the :func:`dynamic_shapes` pytree structure according to :func:`args` and :func:`kwargs`. cxt|}|jvr j|j|Syr:)r+r'r)rrrrft_idss r) find_shapez3ShapesCollection.dynamic_shapes..find_shapes5a5Dt||# $||D))r(c3&K|]}|v ywr:r')rrr5s r)rz2ShapesCollection.dynamic_shapes..s:Tt5 :rzSome tensors that were assigned shapes were not found in args. Maybe such tensors were copied when passing them as args? Maybe such tensors are contained in classes that were not registered with pytree?)setr#ranyr'r)rfmrr!r6 combined_argsrr5s` @r)rzShapesCollection.dynamic_shapessZ  &av6 ,ZG :T\\: :d  r(r:) r r!r"r#r(r.r0r2rr'r(r)r%r%Ys , (" !r(r%c2d}tj|y)NzTUsing None as a dynamic shape dimension is deprecated. Please use Dim.STATIC instead)logwarning)rs r)%_warn_on_None_dynamic_shape_dimensionr?s ( KKr(r;c ddlmm|t|dk(ryt |t t frt||j}ifd  fd t |tt t fsJt |trt |j}t |j}t|t|k7rTd|d|d}t|d k(r#|d|vrt ||dtr|d z }n|d z }j|d  fd}t|||dy)z Checks the dynamic_shapes specification for correctness, using combined args + kwargs as reference for inputs structure. rrNc |jvr|j\}}|j|k7s|j|k7rstj |j||}tj |j|j|j}j d|d|d|dy|j|jf|j<y)NzFound different definitions z and z! for the same symbolic dimension !)r rgrjr+r6r)rr4r5this_that_rrboundss r)check_same_boundsz0_check_dynamic_shapes..check_same_boundss <<6 ! -JD$ww$#''T/ cllD$? cllCGGSWWE!//25'ugF77:e1>#2%(GGSWW#5F3<< r(c t|tr|jD]l\}}t|tr || t ,t|t t frCjd|d|dt|d|d dyt|ttfr{t|D]l\}}t|tr || t ,t|t t frCjd|d|dt|d |d dy|%jd |dt|d dy) Nz%Unexpected dimension mapped to index z in input tensor shape  specified at `dynamic_shapeszP` (expected None, an int, a Dim, Dim.AUTO, Dim.STATIC, or Dim.DYNAMIC, but got z instead)rrzUnexpected dimension #zO` (expected None, an int, a Dim, Dim.AUTO, Dim.STATIC, or Dim.DYNAMIC, but got zUnexpected input tensor shape z` (expected either a list/tuple of dimensions, or a dict mapping indices to dimensions, where each dimension is an int, a Dim, Dim.AUTO, Dim.STATIC, or Dim.DYNAMIC)) rritemsr+r?rBrrr rrr)rtensorr,rrrrrFs r) check_symbolsz,_check_dynamic_shapes..check_symbolss eT "++- 3c4(%c*[9;$S3/:#%33?sBYZ_Y`a77=d|nE$$'5 3#>  t} -#E* 3c4(%c*[9;$S3/:#%3303J5'R77=d|nE##&%y2#>   ++07TU[\`UaTbc`a6  r(zWWhen `dynamic_shapes` is specified as a dict, its top-level keys must be the arg names z of `inputs`, but here they are z. rzSince here `inputs` is a list/tuple enclosing a single dict, maybe you just forgot to enclose `dynamic_shapes` in a list/tuple?zwAlternatively, you could also ignore arg names entirely and specify `dynamic_shapes` as a list/tuple matching `inputs`.rrc t|tjr |||y|6t|}jd|d|dt |d|d dy)NzCannot associate shape rHz` to non-tensor type z at `inputsz` (expected None)rr)rrr*r rrA)rr dynamic_shaper rrrKs r) check_shapez*_check_dynamic_shapes..check_shapesx a & $= 1( &t !//-m_.to_constraint..root_value>s\\#(("3"3T\BF66&>D $ Q!@&IH#8A;''!66%aSV\\!_,=>##'&Hr(rFr)r3rr)rWrrtorch.utils._sympy.solverWrrrr^r]r rrrrrgrjrr+rYrrarrdr+)rrJrrrr_dim_rootroot_constraintr]rrWrWrr%derived_constraints_with_phantom_root phantom_rootssymbolss``` @@r) to_constraintz._process_dynamic_shapes..to_constraint7sP6?   c; 'xxH  G+")(*;*;".update_symbols.._create_static_dimsc%j3&7E2BC Cr()r8_dynamo_weak_dynamic_indices_dynamo_dynamic_indices_dynamo_dynamic_range_dynamo_static_indices_dynamo_unbacked_indicesrrrIrBr+r rrr$rrmaybe_mark_dynamicr% mark_staticr& mark_dynamicrr+rrrranger) rrJr,rirrr constraintsrerfs r)update_symbolsz/_process_dynamic_shapes..update_symbolss D /2e+),&'*u$(+%*-%' eT "++- 93cC;/!#s+0C@!.sFA!>JCLL)00<X.hmm+ 88C/ 11&!< 0 00 2261=&&'9"V*a'HI[MM--fa8 9 t} -#E* 93cC;/!#s+0C@!.sFA!>JCLL)00<X.hmm+ 88C/ 11&!< 0 00 2261=&&'9"V*a'HI[MM--fa8 9 ]6::<( 5 ))&!4 5r(cPt|tjr |||yyr:)rrr*)rrrMrts r) assoc_shapez,_process_dynamic_shapes..assoc_shapes" a & 4M 2 'r(rOr)rrrrrrrrArPrrr]r3r)r;rrv$derived_constraint_with_phantom_rootphantom_root_name dynamic_dimsrrrsrcrdrerfrts @@@@@@@@r)_process_dynamic_shapesrz s;^!4!9 .5$-0,^,]-A-A-CD ,7t+z'_get_dim_name_mapping..s *Q-r(rr)rrrBr+r r^r]r)r name_to_dimrs r)_get_dim_name_mappingr~sK- - ;  c3   T "(+K %#{+14 CHH--.c8, ,,- r(rc $ddl}ddlddlm}m}ddlm} |jddj}i|jdD]}|j}|jd |x} r| jd} d \} } |jd |x} rt| jd} |jd |x}rt|jd} t| | | | <|jd\} }j|}t!|j"rt|| <|| <t%|t'}j)D]\}}t!|tt*t,j.fsJt!|j.rE||sJ||<|j1t3t5t7|j8t!|t,s|j1|j:j<j)D]\}}|vr ||vrJifd}t?|||S#t$r}||jd|d}~wwxYw)a When exporting with :func:`dynamic_shapes`, export may fail with a ConstraintViolation error if the specification doesn't match the constraints inferred from tracing the model. The error message may provide suggested fixes - changes that can be made to :func:`dynamic_shapes` to export successfully. Example ConstraintViolation error message:: Suggested fixes: dim = Dim('dim', min=3, max=6) # this just refines the dim's range dim = 4 # this specializes to a constant dy = dx + 1 # dy was specified as an independent dim, but is actually tied to dx with this relation This is a helper function that takes the ConstraintViolation error message and the original :func:`dynamic_shapes` spec, and returns a new :func:`dynamic_shapes` spec that incorporates the suggested fixes. Example usage:: try: ep = export(mod, args, dynamic_shapes=dynamic_shapes) except torch._dynamo.exc.UserError as exc: new_shapes = refine_dynamic_shapes_from_suggested_fixes( exc.msg, dynamic_shapes ) ep = export(mod, args, dynamic_shapes=new_shapes) rNr)_is_supported_equivalencezSuggested fixes:rz`Suggested fixes not found in error message given to refine_dynamic_shapes_from_suggested_fixes() z(.*) = Dim\('(.*)'.*\))NNz!.* = Dim\('.*', min\=([0-9]+).*\)z.* = Dim\('.*'.*max\=([0-9]+)\)ryz = cD|t|tr|S|j vr |j}t| jrt | vr t |St t |j}|j vr |j}n|j vsJ |j} jjj||\}}|}|dk7rt||z}|dk7r|t|z}| t |<|S|St|trn|jj vrV|j vr |jS|j |jj}| |j<|S|S)Nrr)rrBr ExprrXnextiter free_symbolsr3polys polytoolsdivr^r]rY) rrdummyfixr\r]modulus remainder_dimderived_dim_cacher} shape_fixesrWs r) apply_fixesz?refine_dynamic_shapes_from_suggested_fixes..apply_fixesCs ;*S#.J \\[ (cll+C#uzz*s800,SX66!$s'7'7"89F{{k1*6;;7%{{k999*6;;7).)>)>)B)B3)O&GYC!|!'lS0 A~!C N225%c#h/J [ )chh.?.?;.N||00(66vvk#((*;*;<=26!#,,/  r() rerWrrrrrsplitstrip ExceptionrmatchgrouprBrrVrNumberr~r8rIr+r^rrrXrrrr]r r)rrrrrrshape_fixes_msgexcrrr3rr match_min match_maxr]rootsr crrr}rrWs @@@@r)rrsn@:O))$67:@@BK$$T*)iikHH6< <5 <;;q>D#JD$HH%I3OOyO9??1-.HH%GMMyM9??1-. #Dd =K 5)JD$==&D$ -$'I D!%) D!%)((7KeE!!#'1!c4ejjABBB a $,Q/ //KN IIc$tANN345 6 a % IIaffoo &'!!#.1K1:--.13"H {NN KKi   ' ' n  s"I-- J6J  J)F)G dataclassesr~loggingrt collectionsrenumrrtypingrrrr r rtorch.utils._pytreer r r rrrrrrexported_programrrWr torch._guardsrrrr__all__ getLoggerr r=rrAr+rar^rXrBrr$r%r&rr dataclassrrrrrrrrr8rrr#r%r?rTrzr~rr'r(r)rs #@@    .$V g! t BS4BSJ"@ $@ F,0dc8C=hsm: == __  '+ @  @sm @19# @ 49 @     D #D D N    "  *  4   *    ; 24FF G 8$8$3*d8n458$8$ U38_$ % 8$ u/01 8$ U8U3E-F#PQR 8$#x-(8$]8$~ $ y 38 y yy} y  yx c3h  HHVpXS>pX$sCx.%*d3iEFpXfqS>q$sCx.%*d3iEFq *qh$sCx.%*d3iEF,}L }L$sCx.%*d3i?@}L 4S>5:tCy 01}Lr(