o \h@sddlmZddlZddlZddlZddlZddlmZddlm Z ddl Z ddl Z ddl m Z m Z mZddl mZddlmZmZmZmZddlmZdd lmZmZmZzdd lmZWn eyidZYnwz dd lmZm Z Wn eydZdZ Ynwzdd l!m"Z"Wn eydZ"Ynwd Z#dZ$dZ%dZ&dZ'dZ(dZ)dZ*dZ+dZ,iZ-dde.e DZ/GdddeZ0GdddeZ1GdddeZ2Gdd d eZ3ej j4e 5d!ej j6e 5d"ej j7e 5d#ej j8e 5d$ej j9eej j:eej j;e iZd%e j=d&e j>d%fej j4e j=d'e j?d%e j=d(e j?d%fej j8e j=de j@d%e j=d)e j@d%fej j7e j=d*e jAd%e j=d+e jAd%fej j;e j=de d%e j=d,e d%fej j:e j=d-ed%e j=d.ed%fiZBej j6e j=de j>d%e j=d/e j>d%fej j4e j=d0e j?d%e j=d(e j?d%fej j8e j=de j@d%e j=d1e j@d%fej j7e j=d2e jAd%e j=d+e jAd%fiZCej j6e j=de j>d%e j=d(e j>d%fej j4e j=d3e j?d%e j=d4e j?d%fej j8e j=de j@d%e j=d+e j@d%fej j7e j=d5e jAd%e j=d6e jAd%fej j;e j=ded%e j=d.ed%fej j:e j=d7ed%e j=d8ed%fiZDd9d:d;d<ZEdd=d>ZFdd@dAZGdBdCZH ?   dddQdRZI dddTdUZJ  ddd^d_ZKdd`daZLddbdcZMddgdhZNddldmZOGdndodoZPGdpdqdqZQGdrdsdsZRdtduZSdvdwZTdxdyZUdzd{ZVdddZWddZXdddZYdddZZdddZ[dddZ\dddZ]dddZ^dddZ_dddZ`dddZadddZbdddZcdddZddddZedddZfdddZgdddZhdddZidS)) annotationsN)Enum)Path) ModelProto TensorProtoexternal_data_helper)onnx_pb) make_graph make_model make_nodemake_tensor_value_info)ReferenceEvaluator)GraphOptimizationLevelInferenceSessionSessionOptions float8e4m3fn)int4uint4)to_array_extendedz onnx.quantizez0.1.0zai.onnxz com.microsoftQuantizeLinear_QuantizeLinear_InputDequantizeLinear_DequantizeLinear_Output _quantizedlcCs(i|]}ttt|trtt||qS) isinstancegetattrrint).0krrg/home/air/segue/gemini/backup/venv/lib/python3.10/site-packages/onnxruntime/quantization/quant_utils.py 9s(r"c@(eZdZdZdZddZeddZdS)QuantizationModercC|jSNnameselfrrr!__str__DzQuantizationMode.__str__cC zt|WStytwr')r$KeyError ValueError)moderrr! from_stringG   zQuantizationMode.from_stringN)__name__ __module__ __qualname__ IntegerOps QLinearOpsr, staticmethodr2rrrr!r$@ r$c@r#)QuantizedValueTyperr%cCr&r'r(r*rrr!r,Sr-zQuantizedValueType.__str__cCr.r')r;r/r0)vrrr!r2Vr3zQuantizedValueType.from_stringN)r4r5r6Input Initializerr,r9r2rrrr!r;Or:r;c@sHeZdZdZdZdZdZdZdZdZ dd Z e d d Z e d d ZdS) QuantTyperr%cCr&r'r(r*rrr!r,gr-zQuantType.__str__cCr.r')r?r/r0)trrr!r2jr3zQuantType.from_stringcCs|tjkrtjS|tjkrtjS|tjkrtjS|tjkr tj S|tj kr(tj S|tj kr0tj S|tjkr8tjStd|d)NzUnexpected value qtype=.)r?QInt8rINT8QUInt8UINT8QUInt16UINT16QInt16INT16 QFLOAT8E4M3FN FLOAT8E4M3FNQUInt4UINT4QInt4INT4r0r*rrr! tensor_typeqs       zQuantType.tensor_typeN)r4r5r6rGrIrOrMrKrSrQr,r9r2propertyrUrrrr!r?^s r?c@r#) QuantFormatrr%cCr&r'r(r*rrr!r,r-zQuantFormat.__str__cCr.r')rWr/r0)formatrrr!r2r3zQuantFormat.from_stringN)r4r5r6 QOperatorQDQr,r9r2rrrr!rWr:rWint8uint8int16uint16dtypeiiiiiiiii@ii@rAzero_point_indexcGsg}t|D]H\}}tt|tjr|t|nt|tjr(||n t d|d|||krN|d}|j tj ksF|j tj krNt d|j qt |dkrYt|S|dS)Nzarg z is not an array: rhzzero_point cannot be r%r) enumeratenumpy issubdtypetypenumberappendarrayrndarray TypeErrorr`float32float16lentuple)rjargsnew_argsiar<rrr! _check_types  r|c Cs|tvs Jd|d|tjjtjjtjjtjjfvr|dkr(td|d|jt j kr2tj }n|jt j kr requested. Only INT8, UINT8, INT16, and UINT16 are supported.rz2zero_point is expected to be null for float 8 not rFzUnexpected dtype Constant zero_point)valuer)XscalerYqurr)rrF) reduce_range symmetric)out) ONNX_TYPE_TO_NP_TYPE onnx_protorrPFLOAT8E4M3FNUZ FLOAT8E5M2FLOAT8E5M2FNUZNotImplementedErrorr`rlrtFLOATruFLOAT16r0r r r onnxhelper make_tensorr r r|runget_qmin_qmax_for_qTypemaxminasarrayastyperoundclip)qTypearrrrlowhigh onnx_type onnx_modelrefr`qminqmaxcliplowcliphigharr_fp32rrr!quantize_nparraysN       rFc Cs|dks|dkrtd|d|t|tjd|jd}t|tjd|jd}|dur;t||tj||jd}|rOtt|t|}| }| }||ks]Jd|d|tj||tj d}tj|tj dtj|tj d}t||} | dksJd| t |jj krtjd |jd} tjd|jd} | | gS|rtjt ||tjd tj d|jd} ntjt ||| |jd} | |j} | | gS) aCalculate the scale s and zero point z for the quantization relation r = s(q-z), where r are the original values and q are the corresponding quantized values. r and z are calculated such that every value within [rmin,rmax] has an approximate representation within [qmin,qmax]. In addition, qmin <= z <= qmax is enforced. If the symmetric flag is set to True, the interval [rmin,rmax] is symmetrized to [-absmax, +absmax], where absmax = max(abs(rmin), abs(rmax)). :parameter rmin: minimum value of r :parameter rmax: maximum value of r :parameter qmin: minimum value representable by the target quantization data type :parameter qmax: maximum value representable by the target quantization data type :parameter symmetric: True if the floating-point range should be made symmetric. Defaults to False. :parameter min_real_range: Minimum floating-point range (i.e., rmax - rmin) to enforce. Defaults to None. :return: zero and scale [z, s] rBqmin and qmax must meet requirement: qmin <= 0 <= qmax while qmin:, qmmax:r_Nzqmin=z > qmax=z scale issue?g@)r0rlminimumrqr`maximumrrabsfloat64finfotinyrr) rminrmaxrrrmin_real_rangeabsmaxdrdqrrrrr!compute_scale_zps4   rc sd}|tvr?|tjkr2ddlmddlm}|}fddtdD}tj dd|Dtj d }nt d |d |t|<n |tjkrLddlm}|}|durXt d |d t t|}tj d|d }tj |||jd }||gS)arCalculate the scale s for a float8 type (E4M3FN). The function assumes the coefficient distribution and the float 8 distribution are similar to two gaussian laws. :return: zero and scale [z, s] More details in notebook `quantization_fp8.ipynb `_. Nrfloat8e4m3_to_float32rcsg|]}|qSrr)rrzrrr! Ksz+compute_scale_zp_float8..cSs$g|]}t|st|s|qSr)rlisnanisinf)rfrrr!rMs$r_zQuantization to element_type=z not implemented.zUnexpected element_type rF)FLOAT8_DISTRIBUTIONSrrPonnx.numpy_helperr#onnx.reference.custom_element_typesrrangerlrqrtr0rsstdr`) element_typerzp_dtyper all_valuesvaluesstd_f8zerorrrr!compute_scale_zp_float8:s*       rdata numpy.ndarray quant_typeonnx.TensorProto.DataTyperboolrr float | None rmin_override rmax_overridereturn#tuple[numpy.ndarray, numpy.ndarray]cCsPt|tjstdt|d|dur|}n t|r|nd}|dur(|}n t|r0|nd}tj||j d}tj||j d}tjd|j d} |t j krh|rUt dt |} t|| \} } t| | dd S|t jt jt jt jt jt jfvrt|||d \} } t|rt||| | ||\} } ntjd| j d} t| | dd Std |d) a Returns the zero_point and scale for the given data. :param data: The data for which to compute quantization parameters. :param quant_type: The quantization data type. :param symmetric: whether symmetric quantization is used or not. :parameter reduce_range: True if the quantization range should be reduced. Defaults to False. :parameter min_real_range: Minimum floating-point range (i.e., rmax - rmin) to enforce. Defaults to None. :parameter rmin_override: The value of rmin to use if not None. Otherwise, uses min(data). :parameter rmax_override: The value of rmax to use if not None. Otherwise, uses max(data). :return: zero point and scale z%Weight must be given as an array not rFNgr_rz1Unsupported option reduce_range=True for float 8.rrirz Unexpected value for quant_type=)rrlrrrsrnrvrrrqr`rrP RuntimeErrorrrr|rHrJrNrLrTrRrrr0)rrrrrrrrrrrrrrrrr!compute_data_quant_params_s>   r2tuple[numpy.ndarray, numpy.ndarray, numpy.ndarray]c Cst|||||||\}}|tjkrIt||||} t| tjd@dkrDt |} t d| d| d| d| d ||| fS|tj tjtjtjtjtjfvret||||} ||| fStd|d)al :param data: data to quantize :param qType: data type to quantize to. :param symmetric: whether symmetric quantization is used or not. :parameter reduce_range: True if the quantization range should be reduced. Defaults to False. :parameter min_real_range: Minimum floating-point range (i.e., rmax - rmin) to enforce. Defaults to None. :parameter rmin_override: The value of rmin to use if not None. Otherwise, uses min(data). :parameter rmax_override: The value of rmax to use if not None. Otherwise, uses max(data). :return: minimum, maximum, zero point, scale, and quantized weights To pack weights, we compute a linear transformation - when data `type == uint8` mode, from `[rmin, rmax]` -> :math:`[0, 2^{b-1}]` and - when data `type == int8`, from `[-m , m]` -> :math:`[-(2^{b-1}-1), 2^{b-1}-1]` where `m = max(abs(rmin), abs(rmax))` and add necessary intermediate nodes to transform quantized weight to full weight using the equation :math:`r = S(q-z)`, where - *r*: real original value - *q*: quantized value - *S*: scale - *z*: zero point rbz+One of the quantized value is NaN data in [z, z], quantized_data in [z].zUnexpected value for qType=rF)rrrPranyrrlr\ravelrrrrrHrJrNrLrTrRr0) rrrrrrrrrquantized_datanp_datarrr! quantize_datas@    rweightonnx.TensorProtorraxis int | Nonequant_weight_name str | Nonec Cst|}d}|durt||||}n?|j|}t|j} d| |<g} t|D]$} || |} || } || }t|| | |}| t | | q(t | |}|rW|n|j t }|tjjkrt}||_|j|j||_ ||_tdurt|}|j|jks||krtd|jd|ddd|ddd|jdt|dd d |S|tjjtjjfvr|jtjtjfvrtd |d t t!|}tj"j#|||j|d d}|Stj"$|}tj ||d |j}tj%&||}|S)aG Returns a quantized version of the given ONNX initializer. :param weight: The ONNX initializer to quantize. :param quant_type: The final quantized data type. :param zero_point: The zero-point value to use for quantization. :param scale: The scale value to use for quantization. :param axis: The quantization axis if quantizing per-channel. Defaults to None. :param quant_weight_name: The name of the quantized initializer. If not specified, the quantized name is generated. :return: The quantized ONNX initializer. Nr%zThe initializer of shape z! could not be created, expecting z, got z and shape=z raw=rFzQuantized weights for z. must be 8-bit before packing as 4-bit values.T)rawr_)'tensor_proto_to_arrayrrshapelistrtakerprlrreshape concatenater)TENSOR_NAME_QUANT_SUFFIXrrrP data_typedimsextendflattencopytobytesraw_datarrstrrTrRr`r[r\bytespack_bytes_to_4bitrrtensor_dtype_to_np_dtype numpy_helper from_array)rrrrrr weight_data q_weight_data channel_count channel_dimsquantized_channel_data_listrz channel_data channel_scalechannel_zero_pointquantized_channel_data q_weight_nameq_weight_initializercheck packed_dataquant_np_dtyperrr!quantize_onnx_initializersb         r c Cs|tjjkr tdd}|rt|}n|r|tvrt|}nt|}|s.td|d|\}}|dks:|dkrQtd|d|d|j d |d |d | |S) z Return qmin and qmax, the minimum and maximum value representable by the given qType :parameter qType: onnx.onnx_pb.TensorProto.UINT8 or onnx.onnx_pb.TensorProto.UINT8 :return: qmin, qmax z;This function is not implemented for float 8 as not needed.Nr}r~rrrz, dtype=z, reduce_range=z , symmetric=z, qType=) rrrPrONNX_INT_TYPE_REDUCED_RANGEgetONNX_INT_TYPE_SYMMETRIC_RANGEONNX_INT_TYPE_RANGEr0r`)rrrqrangerrrrr!r)s8     rcCst|||d\}}||S)z Helper function to get the quantization range for a type. parameter qType: quantization type. return: quantization range. r)r)rrrrrrrr!get_qrange_for_qTypeIsrrranktuple[bool, int]cCs,|dkr||n|}|dko||k}||fS)z Helper function that tries to return a normalized axis in the range [0, rank - 1]. :parameter axis: The axis to normalize. :parameter rank: The tensor rank (number of dimensions). :return (is_valid, axis_norm) rr)rr axis_normis_validrrr!normalize_axisSsrsrc_8bitr bytearraycCst|}|dkr tS|dd}t|}d}d}||dkr?||dd@d>||d@B||<|d7}|d7}||dks||krK||d@||<|S)aB Copies a source array of 8-bit values into a destination bytearray of packed 4-bit values. Assumes that the source values are already in the appropriate int4 range. :parameter src_8bit: The 8-bit element values to pack. :return A bytearray with every two 8-bit src elements packed into a single byte. rr%r@rcrB)rvr)r num_elemsdst_sizedstsrc_idst_irrr!r_s  $ rc@s eZdZdZggdfddZdS)QuantizedInitializerzJ Represents a linearly quantized weight input from ONNX operators Nc C:||_||_||_||_||_||_||_||_| |_dSr') r) initializerrminsrmaxs zero_pointsscalesrrr) r+r)rr r!r"r#rrrrrr!__init__s  zQuantizedInitializer.__init__r4r5r6__doc__r$rrrr!r}s  rc@s"eZdZdZ    dddZdS)QuantizedValuezI Represents a linearly quantized value (input\output\intializer) Nc Crr') original_nameq_name scale_namezp_name value_typer node_type node_qtype scale_type) r+r)new_quantized_namer*zero_point_namequantized_value_typerr-r.r/rrr!r$s  zQuantizedValue.__init__)NNNNr%rrrr!r's r'c@seZdZdZddZdS)BiasToQuantizez+ Represents a bias to be quantized cCs||_||_||_dSr') bias_name input_name weight_name)r+r4r5r6rrr!r$s zBiasToQuantize.__init__Nr%rrrr!r3s r3cCs|jdkrtd|jd|jdkr|j}n^|jdkr |j}nU|jdkr)|j}nL|jdkr2|j}nC|jdkr;|j}n:|jd krD|j}n1|jd krM|j }n(|jd krV|j }n|jd kr_|j }n|jd krh|j }n td|jd|jd|j|iS)z Convert attribute to kwarg format for use with onnx.helper.make_node. :parameter attribute: attribute in AttributeProto format. :return: attribute in {key: value} format. rz attribute z does not have type specified.r%r@rArBrCrDrd rz has unsupported type rF) rnr0r)rrzsrEgfloatsintsstringstensorsgraphs) attributerrrr!attribute_to_kwargs0            rAcs*fdd|D}t|dkr|dSdS)z Helper function to find item by name in a list. parameter item_name: name of the item. parameter item_list: list of items. return: item if found. None otherwise. csg|] }|jkr|qSrr()ritem item_namerr!rsz find_by_name..rN)rv)rD item_listitemsrrCr! find_by_namesrGcCs*d}tt|D] }|||kr|}q|S)zC Helper function to return index of an item in a node list rh)rrv) elem_name elem_listelem_idxrzrrr!get_elem_indexs  rKcCstjd||g|S)z Helper function to create a Mul node. parameter inputs: list of input names. parameter output: output name. parameter name: name of the node. return: Mul node in NodeProto format. Mul)rrr )inputsoutputr)rrr! get_mul_nodesrOfilenamer identifierrcCs|j|j||jS)zp Helper function to generate a identifiable filepath by concatenating the given identifier as a suffix. )parentjoinpathstemsuffix)rPrQrrr!generate_identified_filename srVcCsddl}ddlm}ddl}|j|jdtdt|tdt||j||dd|d| d| d | dS) Nr) thresholdz Histogram:zHistogram Edges:T)fillz Tensor valueCountszTensor value V.S. Counts) sysmatplotlib.pyplotpyplotrlset_printoptionsmaxsizeprintstairsxlabelylabeltitleshow)hist hist_edgesrZpltrlrrr! apply_plots     rhrFcs<ddlddl}ddlddlmmm}ddlmmm}ddl m m m t d|Gfdddj}j||d}ttj|dd  }||Wdn1sfwYd}|d } g} t|D]I} || } | } t| d |t| d |g}tt|}| | }| |}|!| |"| ||#| ||$| }| %|q}|&| t'| | D]}| (|q| )}|*| |+| ||,| }| -|| .}ttj|d d }||Wdn 1swYtj/dddvrF|j0|d}|1}t2|D]}|3|}t |4t |5q/ttj|dd C}t|D]3} || } | } t| d |t| d |g}| dtt|}|||dqWWddS1swYdS)z> Helper function to write calibration table to files. rN)CalibrationMethod TensorData TensorsDatazcalibration cache: cs eZdZfddZdS)z*write_calibration_table..MyEncodercsbt|fr |St|jr|t|jddSt|r*|jjt|dSj ||S)Nz numpy.array)rr`CLS)rlr) rto_dictrrtolistrr` __class__r4 JSONEncoderdefault)r+objrirjrkjsonnprr!rq3s  z2write_calibration_table..MyEncoder.defaultN)r4r5r6rqrrsrr! MyEncoder2srv)clszcalibration.jsonwihighestlowestzcalibration.flatbufferswbQUANTIZATION_DEBUG)r%1zcalibration.cache  )6rt flatbuffersrl5onnxruntime.quantization.CalTableFlatBuffers.KeyValue quantizationCalTableFlatBuffersKeyValue5onnxruntime.quantization.CalTableFlatBuffers.TrtTableTrtTable"onnxruntime.quantization.calibraterirjrklogginginforpdumpsopenospathjoinwriterqBuildersortedkeysrmfloatr rBrr CreateString KeyValueStartKeyValueAddKeyKeyValueAddValue KeyValueEndrpTrtTableStartDictVectorrvPrependUOffsetTRelative EndVector TrtTableStartTrtTableAddDict TrtTableEndFinishOutputenvironGetRootAsTrtTable DictLengthrDictKeyValue)calibration_cachedirrrrrv json_datafilerbuilderkey_value_listkeyrd_valuesr;rflat_key flat_value key_value main_dict cal_tablebufdict_lenrzrrsr!write_calibration_table"sx                     $r-C6?cCs|dktj}|dktj}|}|j|}|sdS|t|t|}|dks8Jd|d|d||tj}|||| |7}|dkdksSJ|S)a~Given a discrete distribution (may have not been normalized to 1), smooth it by replacing zeros with eps multiplied by a scaling factor and taking the corresponding amount off the non-zero values. Ref: http://web.engr.illinois.edu/~hanj/cs412/bk3/KL-divergence.pdf https://github.com//apache/incubator-mxnet/blob/master/python/mxnet/contrib/quantization.py rNrzn_zeros=z , n_nonzeros=z, eps1=)rrlrtsumsizer)pepsis_zeros is_nonzerosn_zeros n_nonzeroseps1rerrr!smooth_distribution~s " r model_pathcCs(tj|dd}tdd|jjDS)NF)load_external_datacss|]}t|VqdSr')ruses_external_data)r intializerrrr! sz*model_has_external_data..)rloadas_posixrgraphr)rmodelrrr!model_has_external_datasropt_model_pathcCsFt}||_tj|_i}dg|d<t||fddgi|}dS)z Generate model that applies graph optimization (constant folding, etc.) parameter model_path: path to the original onnx model parameter opt_model_path: path to the optimized onnx model :return: optimized onnx model ConstantSharingdisabled_optimizers providersCPUExecutionProviderN)rroptimized_model_filepathrORT_ENABLE_BASICgraph_optimization_levelr)rr sess_optionkwargs_rrr!optimize_models    rrrcCs>ddi}|jr|jD] }||j|jiq tj||dS)z>Tag the model that it went through quantization pre-processingonnx.quant.pre_processonnxruntime.quantNmetadata_propsupdaterrrrset_model_props)rrproprrr!add_pre_process_metadatas  rcC0|jr|jD]}|jdkr|jdkrdSqdS)zCCheck the model whether it went through quantization pre-processingrrTFrrr)rrrrr!model_has_pre_process_metadatas  rcCs>ddi}|jr|jD] }||j|jiq tj||dS)N onnx.inferrr)rrrrrr!add_infer_metadatas  rcCr)NrrTFr)rrrrr!model_has_infer_metadatas  rcCsBt|d}tjt|t|t|}t|||S)Nz -inferred) rVrshape_inferenceinfer_shapes_pathrrrrunlink)rinferred_model_pathrrrr!load_model_with_shape_infers rcCsdtjdd"}t|}t|d}tj||ddt |WdS1s+wYdS)Nz ort.quant.)prefixz model.onnxT)save_as_external_data) tempfileTemporaryDirectoryrdeepcopyrrSr save_modelrr)r quant_tmp_dir model_copyrrrr!&save_and_reload_model_with_shape_infers  $rrrcCs>|jtjjtjjfvrtj|Std|j dt |j)Nz&Only float type is supported. Weights z is ) rrrrrrrto_arrayr0r) type_to_name)rrrr!rs  r tensor_namecC|dS)N_QuantizeLinearrrrrr!add_quant_suffixrcC|tSr')QUANT_INPUT_SUFFIXrrrr!add_quant_input_suffixrrcCr)N_QuantizeLinear_Outputrrrrr!add_quant_output_suffixrrcCr)N_DequantizeLinearrrrrr!add_dequant_suffixrrcCr)N_DequantizeLinear_Inputrrrrr!add_dequant_input_suffixrrcCrr')DEQUANT_OUTPUT_SUFFIXrrrr!add_dequant_output_suffixrr)NN)FN)FNNN)rrrrrrrrrrrrrrrr)rr)rrrrrrrrrrrrrr)FF)rrrrrr)rrrr)rPrrQrrr)rF)r)rr)rrrr)rr)rrrr)rrrr)rrrr)rrrr)rrrr)rr)j __future__rrrrrenumrpathlibrrlrrrrrr onnx.helperr r r r onnx.referencer onnxruntimerrrrr ImportErrorrronnx.reference.op_runr __producer__ __version__ onnx_domain ms_domain QUANT_OP_NAMErDEQUANT_OP_NAMErrMODEL_SIZE_THRESHOLDrrrr$r;r?rWrHr`rJrNrLrPrTrRrrqr\r[r^r]rr r r|rrrrrr rrrrrr'r3rArGrKrOrVrhrrrrrrrrrrrrrrrrrrrrr!s        &$ $$$  $ $$$$$$$    4?) ? B O  %    \