Vhw UddlZddlmZddlmZmZmZddlZddlm Z m Z m Z m Z m Z ddlmZedefiZeed<ej&eZej,ej.ej0ej2hZej6dej8dej:d ej<d ej>d ej@d ejBd iZ" ejFdejHdejJdejLdiZ'e(eejRe fe*e+e+ffed<e'jYe"dejRde-fdZ.dZ/dZ0dZ1ejdjgddZ4e1e4Z5dddddde jlfdejndejnde de*e+d fd!ejRd"ee+d#ee+d$ee8d%eejRd&eejRd'e-d(ee de*ejnejnffd)Z9e5 d9d*eejnde:de;e+d!ejRd"eee+e8e-fd#eee+e8e-fd$ee8d%eejRd&eejRd'e-d(ee:deejndeejnde*ejnejnffd+Z<ejzdde jlfd*ejnde*e+d fd,ejnd-eejnd.ejRd"eee+e8fd#eee+e8fd(ee dejnfd/Z>e5dde jlj~fd*ejnde;e+d,ejnd-eejnd.ejRd"eee+e8e-fd#eee+e8e-fd(ee:dejnfd0Z@e jlj~fd*ejnde;e+d,ejnd-eejnd"ee+e8fd#ee+e8fd(ee:dejnfd1ZAdde jlfejd2d*ejnde*e+d fd,ejnd-eejnd3ejRd"eee+e8fd#eee+e8fd(e d.ejRdejnfd4ZCe5dde jlj~ejfd*ejnde;e+d,ejnd-eejnd3ejRd"eee+e8e-fd#eee+e8e-fd(ee:d.ejRdejnfd5ZDe jlj~ejfd*ejnde;e+d,ejnd-eejnd"ee+e8fd#ee+e8fd(ee:d.ejRdejnfd6ZEGd7d8e ZFy):N)ABCMeta)AnyOptionalUnion)AffineQuantizedObserverBaseget_block_size MappingType TorchAODTypeZeroPointDomain)NodeABC)r)r)r)r)r)r?)r)r)ir)ii)ii_DTYPE_TO_QVALUE_BOUNDSdtypereturnctjtjtjtjh}||vSN)torch float8_e4m3fnfloat8_e4m3fnuz float8_e5m2float8_e5m2fnuz)r fp8_typess _/home/dcms/DCMS/lib/python3.12/site-packages/torch/ao/quantization/pt2e/_affine_quantization.py_is_float8_typer"4s:     I I c4|tvr?tj|jtj|j}}n"|t vrt d|t |\}}||}||}||k\s Jd|d|||ks Jd|d|||fS)zGet quant_min and quant_max args based on dtype and also verify that they are within the range of possible quant_min/quant_max for dtype zUnsupported dtype: z9quant_min out of bound for dtype, quant_min_lower_bound: z quant_min: z9quant_max out of bound for dtype, quant_max_upper_bound: z quant_max: ) FP8_TYPESrfinfominmaxr ValueError)r quant_min quant_maxquant_min_lower_boundquant_max_upper_bounds r!_get_and_check_qmin_qmaxr.?s   KK  " " KK  " " 5 - -.ug6777Nu7U44) ) - - ""7!8 YK Q - - - ""7!8 YK Q - i r#c t|t|k(sJg}g}d}tt|D]}||||k7rw||dkDro||||zdk(sJd|d||d|d|||j||||z|j|||j|dz|dz }|j||||dk7r|j||dz }||fS)aGiven block_size and input size find the parameters for reduction: Output: shape_for_reduction: the shape we use to `view` input to prepare it for reduction reduction_dims: the dims we'll do reduction over Example:: Input: block_size: (3, 3, 2, 10) input_size: (3, 3, 10, 10) Output: shape_for_reduction: (3, 3, 5, 2, 10) reduction_dim: [0, 1, 3, 4] rrzExpecting input size at z dimension: z" to be divisible by block_size at )lenrangeappend) block_size input_sizeshape_for_reductionreduction_dimscur_dimis r!_get_reduction_paramsr:^s< z?c*o -- -NG 3z? # a=JqM )jma.?a=:a=0A5 *1#\a=/!CA3lS]^_S`Rac 5 & &z!} 1 'E F  & &z!} 5  ! !'A+ . qLG & &z!} 5!}!%%g. qLG%&  ..r#c"ddlmfd}|S)aThis decorator is used to preserve some high level operators for torch.export.export while still allow them to be decomposed for inductor path requirement: make sure `fn.__name__[1:]` is the operator name you want to register NOTE: This should be applied at the top, after all other decorators have been applied NOTE: We haven't tested the case when `fn` accepts tensor subclass instance as input, e.g. uint4 tensor subclass instance, and we'll probably need to figure out what would make sense for downstream system (like executorch) to accept as well Example: lib = torch.library.Library("my_namespace', "FRAGMENT") register_custom_op = _register_custom_op(lib) @register_custom_op def _the_op_that_needs_to_be_preserved(...) ... # after this, `_the_op_that_needs_to_be_preserved` will be preserved as # torch.ops.my_namespace.the_op_that_needs_to_be_preserved operator after # torch.export.export / torch._export.export_for_training r)register_decompositioncddlm}jddk(sJdjtfddDrJdjjdd}||i z}j |j |d j }tttj||}|g|S) Nr) infer_schema_z-Expecting function name starts with `_`, got c3:K|]}|jvywr)__name__).0cfns r! z9_register_custom_op..decorator..s !"A  sz.<>zEExpecting op to be defined in normal functions, not lambda or local: r) mutates_argsCompositeImplicitAutograd) torch._library.infer_schemar>rAanydefineimplnsgetattrrops)rDr>op_nameschema lib_namespaceoplibr<s` r! decoratorz&_register_custom_op..decorators< KKNc ! I :2;;- H I ! &+   a RSUS^S^R_ ` a ++ab/<<< 6 "9: WUYY 6 @$t$R( r#)torch._inductor.decompositionr<)rSrTr<s` @r!_register_custom_oprVs2E* r# pt2e_quantFRAGMENTTmin_valmax_val mapping_typer4. target_dtyper*r+eps scale_dtypezero_point_dtype preserve_zerozero_point_domainc ltd|j||||||| | | | j|| Sd|| S)alA variant of :func:`~torchao.quantization.quant_primitives.choose_qparams_affine` operator that pass in min_val and max_val directly instead of deriving these from a single input. This is used for observers in static quantization where min_val and max_val may be obtained through tracking all the data in calibration data set. Args: Mostly same as :func:`~torchao.quantization.quant_primitives.choose_qparams_affine`. with one difference: instead of passing in `input` Tensor and use that to calculate min_val/max_val and then scale/zero_point, we pass in min_val/max_val directly N)_choose_qparams_affinename) rYrZr[r4r\r*r+r]r^r_r`ras r!"choose_qparams_affine_with_min_maxresa0 "  "3"? FJ r#inputc & t|||\}}|tjjtjjtj jfvs Jd||t vr'|tjjk(s Jd||| |j}| |j}|)tj|jj}t||jk(sJd|jd|t||j\} }|j| }tj ||d} tj"||d} nr| | Jd| j| jk(sJd | | j}| | j}|)tj| jj}| rStj$| tj&| }tj(| tj&| }n| }| }|tjjk(s|tjjk(r|tjjk(r,tj(| |}|t+||z d z z }nW|tjjk(sJ|t+|z }|t+|z }||kD}tj,|||}| s t/d | (| t0j2jk7r t/d tj4|| }tj6|t9||zdzd z }n|tj jk(sJ||z t+||z z }tj4|| }| t0j:jk(rd}nl| r3|tj<||z z }tj4|||}n7| t0j>jk(sJd||zdzd z }|||zz}||jA|}|jA||fS)aoop definition that has compatible signatures with custom op library The op does the following: 1. figure out the dimension for reduction based on block_size 2. find min_val/max_val based on the dimension for reduction 3. calculate quantization parameters based on min_val/max_val based on args like `preserve_zero` and `zero_point_domain` zUnsupported mapping type: z2 3  y K1166 6 Y I, X Y 6   ++K  #${{  ;++ekk*..C  Ouyy{ * D EIIK=zl C D *.C  / +^ ./**UF**UF  G$7 c b c 7 MMW]] * k j k *  !--K  #&}}  ;++gmm,00Cii)9)9')BC ii)9)9')BC     --222 ;@@EE E ;0055 5))[L+>K5Y)>#?!#CDE;#H#H#M#MM MMy!11Dy!11D$;DKKdD1ET   )!_%8%8%=%==f  Es+__UCY1F1Ja0O,PQ {55:::::{*eI 4I.JJ Es+  4 4 9 9 9J&)II "[[Y J &)>)>)C)CCNMNC&2Q6!; (59+<< ]])9]: 88+8 & 22r#rr output_dtypec Jt|||||||||jSdS)a Args: input (torch.Tensor): original float32, float16 or bfloat16 Tensor block_size: (Tuple[int, ...]): granularity of quantization, this means the size of the tensor elements that's sharing the same qparam e.g. when size is the same as the input tensor dimension, we are using per tensor quantization scale (float): quantization parameter for affine quantization zero_point (int): quantization parameter for affine quantization output_dtype (torch.dtype): requested dtype (e.g. torch.uint8) for output Tensor quant_min (Optional[int]): minimum quantized value for output Tensor, if not specified, it will be derived from dtype quant_max (Optional[int]): maximum quantized value for output Tensor, if not specified, it will be derived from dtype zero_point_domain (ZeroPointDomain): the domain that zero_point is in, should be either integer or float if zero_point is in integer domain, zero point is added to the quantized integer value during quantization if zero_point is in floating point domain, zero point is subtracted from the floating point (unquantized) value during quantization default is ZeroPointDomain.INT Note: How can block_size represent different granularities? let's say we have a Tensor of size: (3, 3, 10, 10), here is the table showing how block_size represents different granularities: granularity type | block_size per_tensor | (3, 3, 10, 10) per_axis (axis=0) | (1, 3, 10, 10) per_axis (axis=1) | (3, 1, 10, 10) per_group (groupsize=2) | (3, 3, 10, 2) per_group (groupsize=2) for axis = 3 | (3, 3, 2, 10) Output: quantized tensor with requested dtype N)_quantize_affinerdrfr4rrrr*r+ras r!quantize_affinermsGZ   "3"?  FJ  r#c t|||\}}|tvrtj}t |||||||j |S)a,op definition that has compatible signatures with custom op library Note: zero_point_domain is optional specifies how we quantize the floating point to quantized data: INT: quantized_val = (float_val / scale) (integer) + zero_point (integer) FLOAT: quantized_val = (float_val - (zero_point (float) - scale * mid_point)) / scale None: quantized_val = (float_val / scale) | this is primarily used for floatx quantization Where we do not want to round values to nearest integer and instead scale and cast. )r._SUB_BYTE_UINT_BOUNDSruint8_quantize_affine_no_dtype_castr~rs r!rrsY(4L)YWIy,,{{ )    br#c|jtjtjtjfvsJd|jt ||j k(sJd|j d|t||j\}}|j} |j|}|} |D]} d| | < |j| }||j| }|tjjk(r4tjtj|d|z z|z||} n|tj jk(r:|Jdtjtj|d|z z||} n|2|Jdtj||j#z||} nb|tj$jk(sJ||zdzdz } ||| zz }tjtj||z |z ||} | j| } | S) aW The op does the following: 1. figure out the dimension for reduction based on block_size, also reshape the input to align with the shape after reduction 2. quantize the input based on the quantization parameters scale and zero_point and args like zero_point_domain 3. reshape the quantized result to origianl shape zUnsupported input dtype: rhrirg?8zero_point should be None when zero_point_domain is NONE8zero_point should be None when zero_point_domain is Noner0)rrfloat32float16bfloat16r1rkr:rpshaperqr rwrdrxr|r{ reciprocalr})rfr4rrr*r+rar6r7original_shapeshape_after_reductionr9quantrrYs r!rrsF$ ;;    1 #5;;-0 1  J599;&@  }N:,?@&*?EJJL+'[[N JJ* +E/ %#$a % JJ, -E__%:; O//444 KKu- . ;Y   o2277 7   F E F  EKKu(=> 9U  "   F E F  EE$4$4$66 9M O$9$9$>$>>>>*Q.!3 uy00 KKE1 2Iy  JJ~ &E Lr#r input_dtypec Rt|||||||||j| Sd| S)a Args: input (torch.Tensor): quantized tensor, should match the dtype `dtype` argument block_size: (List[int]): granularity of quantization, this means the size of the tensor elements that's sharing the same qparam e.g. when size is the same as the input tensor dimension, we are using per tensor quantization scale (Tensor): quantization parameter for affine quantization zero_point (Tensor): quantization parameter for affine quantization input_dtype (torch.dtype): requested dtype (e.g. torch.uint8) for output Tensor quant_min (Optional[int]): minimum quantized value for input Tensor quant_max (Optional[int]): maximum quantized value for input Tensor output_dtype (torch.dtype): dtype for output Tensor, default is fp32 zero_point_domain (ZeroPointDomain): the domain that zero_point is in, should be either integer or float if zero_point is in integer domain, zero point is added to the quantized integer value during quantization if zero_point is in floating point domain, zero point is subtracted from the floating point (unquantized) value during quantization default is ZeroPointDomain.INT Output: dequantized Tensor, with requested dtype or fp32 Nr)_dequantize_affinerd rfr4rrrr*r+rars r!dequantize_affiner sMD   "3"?!  FJ!  r#c |tvr&|j|k(sJd|d|j|tjtjtj fvs Jd|t |||\}}t||||||||S)zCop definition that has compatible signatures with custom op libraryz Expected: z, got: zUnsupported output dtype: )rrrrrrr.!_dequantize_affine_no_dtype_checkrs r!rr:s// KK; & :  }GEKK= 9 : &     3 $L>2 3  4KIVIy ,    r#c2t||jk(sJd|jd|t||j\}} |j} |j |}|} | D]} d| | < |j | }||j | }|t jjk(r\|jtjd} |"| |jtjz } | j|} | |z} n|t jjk(r |Jd|j|} | |z} n|I|Jdt|jsJd|j|j|} | |z} nT|t jjk(s Jd |||zdzd z }||z } | j|} | |z} || |z } | j | j|S) aThis function converts AQT tensors to their high precision floating point representation The op does the following: 1. figure out the dimension for reduction based on block_size, also reshape the input to align with the shape after reduction 2. dequantize the input based on the quantization parameters scale and zero_point and args like zero_point_domain 3. reshape the quantized result to origianl shape and change dtype to the output_dtype rhrirT)copyrrzNdequantiztion with no zero point domain is only supported with FP8 types, got zUnexpected zero point domain: r0)r1rkr:rprrqr rwrdr~rint32r{r"rr})rfr4rrr*r+rarr6r7rrr9dequantrs r!rr^sU& J599;&@  }N:,?@&*?EJJL+'[[N JJ* +E/ %#$a % JJ, -E__%:; O//444((5;;T(2  ! ekk ::G**\*E/ o2277 7   F E F ((<(E/  "   F E F  KK  j [\a\g\g[h i j ((<(E/ !6!6!;!; ; @ +,=+> ? @ ;*Q.!3 )#**\*5  ! z !G << ' * *< 88r#ceZdZdejfdZdeejejffdZdejjde fdZ y) AffineQuantizedMinMaxObserverrfc&|jdk(r|S|j}|j|_|jJdt |j |j|_t|j|j\}}|j|}tj||d}tj||d}t|dr t|ds||_||_|S|jj |j k(s+Jd|jj d|j |j j |j k(s+Jd |j j d |j tj"|j|}tj$|j |}|jj'||j j'||S) Nrzgranularity is NoneFrjrYrZz=Can't update existing min_val - shape mismatch, self.min_val:z != min_val:z=Can't update existing max_val - shape mismatch, self.max_val z != max_val:)numeldetachroriginal_dtype granularityrrr4r:rprqrrrrshasattrrYrZr'r(copy_)selfrfinput_detachedr6r7rYrZs r!forwardz%AffineQuantizedMinMaxObserver.forwards ;;=A L,22+B-BB+()=)=t?O?OP.C OO^002/ +^(,,-@A**^O**^OtY'wtY/G"DL"DL  ""gmm3 Nt||OaOaNbbnovo|o|n}~ 3 ""gmm3 Nt||OaOaNbbnovo|o|n}~ 3ii g6Gii g6G LL  w ' LL  w ' r#rcHt|dr t|dsJdt|j|j|jg|j |j |j|j|j|j|j|j S)NrYrZzhExpecting the observer has min_val and max_val, please run the observer before calling calculate_qparams) rrerYrZr[r\r*r+r]r^r_r`ra)rs r!calculate_qparamsz/AffineQuantizedMinMaxObserver.calculate_qparamsstY'G )-  v u v 2 LL LL        NN NN HH     ! !     " "  r#model observer_nodec tdddlm}|j\}}|jj |5|j Jd|jJd|||jd|}|||jd|}|jjtjjj|jd|j |||j|j|j |j"j$fi}|jjtjjj&||j |||j|j|j |j"j$fd|ji} |j)| |jj+|dddy#1swYyxYw) Nzcalling convertr)create_getattr_from_valuez$Expecting block_size to be populatedz(Expecting original_dtype to be populated_scale _zero_pointr)printtorch.ao.quantization.fx.utilsrrgraphinserting_beforer4r call_functionrrNrWrargsr\r*r+rardrreplace_all_uses_with erase_node) rrrrrr scale_nodezero_point_nodeq_nodedq_nodes r!convertz%AffineQuantizedMinMaxObserver.converts  L 224z [[ ) )- 8& 2??. V0V V.##/ :9 :/25%++xQVWJ7u{{M:O[[.. $$44!&&q)OO#%%NNNN**//  Fkk// $$66OO#%%NNNN**//  !4!45 G  / / 8 KK " "= 1M& 2& 2& 2s FG%%G.N) rA __module__ __qualname__rTensorrtuplerfx GraphModuler rr#r!rrsNU\\@ 5u||)C#D &+2UXX11+2$+2r#r) NNNNNTrwNN)Gloggingabcrtypingrrrrtorch.ao.quantization.observerrrr r r torch.fxr objectr __annotations__ getLoggerrAloggerrrrrr%uint1uint2uint3uint4uint5uint6uint7rrint8int16rrdictrrrzupdateboolr"r.r:rVlibraryLibrary quant_libregister_custom_oprwrrurestrlistrcno_gradrrdrrrrrrrrr#r!rs''  56)R (S(   8 $        KK KK KK KK KK KK KK  KK JJ  KK& KK& TeEKK$=>c3hOP 455;;4 >'/T0f MM ! !, ; (3 $#)-.23B3F3F& \\& \\&&c3h & ++ & } &}& %&%++&&u{{+&& 0& 5<< %&&R 4837)-.2',&*&*@3 ELL !@3@3S @3++ @3 c5$./0 @3 c5$./0 @3 %@3%++&@3u{{+@3@3 }@3ell #@3ell #@3 5<< %&@3@3F.2-13B3F3F5 <<5c3h5 <<5& 5 ++ 5 c5j)* 5c5j)*5 05 \\55p4837'6':':'?'?  << S   << &  ++  c5$./0 c5$./0  }  \\  T(7':':'?'?> <<>S > <<>& > S%Z > S%Z > }> \\>N.2-1)8)<)<,!& , <<,c3h, <<,& ,  , c5j)* ,c5j)*,',++, \\,^4837'6':':'?'? %   << S   << &    c5$./0 c5$./0  } ++  \\  T(7':':'?'? % G9 <<G9S G9 <<G9& G9 S%Z G9 S%Z G9 }G9++G9 \\G9T_2$?_2r#