o vi@sddlmZddlZddlZddlZddlZddlmZddlm Z ddl Z ddl Z ddl m Z m Z mZddl mZddlmZmZmZmZddlmZdd lmZmZmZzdd lmZWn eyidZYnwz dd lmZm Z Wn eydZdZ Ynwzdd l!m"Z"Wn eydZ"Ynwd Z#dZ$dZ%dZ&dZ'dZ(dZ)dZ*dZ+dZ,iZ-dde.e DZ/GdddeZ0GdddeZ1GdddeZ2Gdd d eZ3ej j4e 5d!ej j6e 5d"ej j7e 5d#ej j8e 5d$ej j9eej j:eej j;e iZd%e j=d&e j>d%fej j4e j=d'e j?d%e j=d(e j?d%fej j8e j=de j@d%e j=d)e j@d%fej j7e j=d*e jAd%e j=d+e jAd%fej j;e j=de d%e j=d,e d%fej j:e j=d-ed%e j=d.ed%fiZBej j6e j=de j>d%e j=d/e j>d%fej j4e j=d0e j?d%e j=d(e j?d%fej j8e j=de j@d%e j=d1e j@d%fej j7e j=d2e jAd%e j=d+e jAd%fiZCej j6e j=de j>d%e j=d(e j>d%fej j4e j=d3e j?d%e j=d4e j?d%fej j8e j=de j@d%e j=d+e j@d%fej j7e j=d5e jAd%e j=d6e jAd%fej j;e j=ded%e j=d.ed%fej j:e j=d7ed%e j=d8ed%fiZDd9d:d;d<ZEdd=d>ZFdd@dAZGdBdCZH ?   dddQdRZI dddTdUZJ  ddd^d_ZKdd`daZLddbdcZMddgdhZNddldmZOGdndodoZPGdpdqdqZQGdrdsdsZRdtduZSdvdwZTdxdyZUdzd{ZVdddZWddZXdddZYdddZZdddZ[dddZ\dddZ]dddZ^dddZ_dddZ`dddZadddZbdddZcdddZddddZedddZfdddZgdddZhdddZidddZjdddZkdS)) annotationsN)Enum)Path) ModelProto TensorProtoexternal_data_helper)onnx_pb) make_graph make_model make_nodemake_tensor_value_info)ReferenceEvaluator)GraphOptimizationLevelInferenceSessionSessionOptions float8e4m3fn)int4uint4)to_array_extendedz onnx.quantizez0.1.0ai.onnxz com.microsoftQuantizeLinear_QuantizeLinear_InputDequantizeLinear_DequantizeLinear_Output _quantizedlcCs(i|]}ttt|trtt||qS) isinstancegetattrrint).0krrc/home/air/biblejyuku/back/venv/lib/python3.10/site-packages/onnxruntime/quantization/quant_utils.py 9s(r#c@(eZdZdZdZddZeddZdS)QuantizationModercC|jSNnameselfrrr"__str__DzQuantizationMode.__str__cC zt|WStytwr()r%KeyError ValueError)moderrr" from_stringG   zQuantizationMode.from_stringN)__name__ __module__ __qualname__ IntegerOps QLinearOpsr- staticmethodr3rrrr"r%@ r%c@r$)QuantizedValueTyperr&cCr'r(r)r+rrr"r-Sr.zQuantizedValueType.__str__cCr/r()r<r0r1)vrrr"r3Vr4zQuantizedValueType.from_stringN)r5r6r7Input Initializerr-r:r3rrrr"r<Or;r<c@sHeZdZdZdZdZdZdZdZdZ dd Z e d d Z e d d ZdS) QuantTyperr&cCr'r(r)r+rrr"r-gr.zQuantType.__str__cCr/r()r@r0r1)trrr"r3jr4zQuantType.from_stringcCs|tjkrtjS|tjkrtjS|tjkrtjS|tjkr tj S|tj kr(tj S|tj kr0tj S|tjkr8tjStd|d)NzUnexpected value qtype=.)r@QInt8rINT8QUInt8UINT8QUInt16UINT16QInt16INT16 QFLOAT8E4M3FN FLOAT8E4M3FNQUInt4UINT4QInt4INT4r1r+rrr" tensor_typeqs       zQuantType.tensor_typeN)r5r6r7rHrJrPrNrLrTrRr-r:r3propertyrVrrrr"r@^s r@c@r$) QuantFormatrr&cCr'r(r)r+rrr"r-r.zQuantFormat.__str__cCr/r()rXr0r1)formatrrr"r3r4zQuantFormat.from_stringN)r5r6r7 QOperatorQDQr-r:r3rrrr"rXr;rXint8uint8int16uint16dtypeiiiiiiiii@ii@rBzero_point_indexcGsg}t|D]H\}}tt|tjr|t|nt|tjr(||n t d|d|||krN|d}|j tj ksF|j tj krNt d|j qt |dkrYt|S|dS)Nzarg z is not an array: rizzero_point cannot be r&r) enumeratenumpy issubdtypetypenumberappendarrayrndarray TypeErrorrafloat32float16lentuple)rkargsnew_argsiar=rrr" _check_types  r}c Cs|tvs Jd|d|tjjtjjtjjtjjfvr|dkr(td|d|jt j kr2tj }n|jt j kr requested. Only INT8, UINT8, INT16, and UINT16 are supported.rz2zero_point is expected to be null for float 8 not rGzUnexpected dtype Constant zero_point)valuer)XscalerYqurr)rrF) reduce_range symmetric)out) ONNX_TYPE_TO_NP_TYPE onnx_protorrQFLOAT8E4M3FNUZ FLOAT8E5M2FLOAT8E5M2FNUZNotImplementedErrorrarmruFLOATrvFLOAT16r1r r r onnxhelper make_tensorr r r}runget_qmin_qmax_for_qTypemaxminasarrayastyperoundclip)qTypearrrrlowhigh onnx_type onnx_modelrefraqminqmaxcliplowcliphigharr_fp32rrr"quantize_nparraysN       rFc Cs|dks|dkrtd|d|t|tjd|jd}t|tjd|jd}|dur;t||tj||jd}|rOtt|t|}| }| }||ks]Jd|d|tj||tj d}tj|tj dtj|tj d}t||} | dksJd| t |jj krtjd |jd} tjd|jd} | | gS|rtjt ||tjd tj d|jd} ntjt ||| |jd} | |j} | | gS) aCalculate the scale s and zero point z for the quantization relation r = s(q-z), where r are the original values and q are the corresponding quantized values. r and z are calculated such that every value within [rmin,rmax] has an approximate representation within [qmin,qmax]. In addition, qmin <= z <= qmax is enforced. If the symmetric flag is set to True, the interval [rmin,rmax] is symmetrized to [-absmax, +absmax], where absmax = max(abs(rmin), abs(rmax)). :parameter rmin: minimum value of r :parameter rmax: maximum value of r :parameter qmin: minimum value representable by the target quantization data type :parameter qmax: maximum value representable by the target quantization data type :parameter symmetric: True if the floating-point range should be made symmetric. Defaults to False. :parameter min_real_range: Minimum floating-point range (i.e., rmax - rmin) to enforce. Defaults to None. :return: zero and scale [z, s] rBqmin and qmax must meet requirement: qmin <= 0 <= qmax while qmin:, qmmax:r`Nzqmin=z > qmax=z scale issue?g@)r1rmminimumrrramaximumrrabsfloat64finfotinyrr) rminrmaxrrrmin_real_rangeabsmaxdrdqrrrrr"compute_scale_zps4   rc sd}|tvr?|tjkr2ddlmddlm}|}fddtdD}tj dd|Dtj d }nt d |d |t|<n |tjkrLddlm}|}|durXt d |d t t|}tj d|d }tj |||jd }||gS)arCalculate the scale s for a float8 type (E4M3FN). The function assumes the coefficient distribution and the float 8 distribution are similar to two gaussian laws. :return: zero and scale [z, s] More details in notebook `quantization_fp8.ipynb `_. Nrfloat8e4m3_to_float32rcsg|]}|qSrr)r r{rrr" Ksz+compute_scale_zp_float8..cSs$g|]}t|st|s|qSr)rmisnanisinf)r frrr"rMs$r`zQuantization to element_type=z not implemented.zUnexpected element_type rG)FLOAT8_DISTRIBUTIONSrrQonnx.numpy_helperr#onnx.reference.custom_element_typesrrangermrrrur1rtstdra) element_typerzp_dtyper all_valuesvaluesstd_f8zerorrrr"compute_scale_zp_float8:s*       rdata numpy.ndarray quant_typeonnx.TensorProto.DataTyperboolrr float | None rmin_override rmax_overridereturn#tuple[numpy.ndarray, numpy.ndarray]cCsPt|tjstdt|d|dur|}n t|r|nd}|dur(|}n t|r0|nd}tj||j d}tj||j d}tjd|j d} |t j krh|rUt dt |} t|| \} } t| | dd S|t jt jt jt jt jt jfvrt|||d \} } t|rt||| | ||\} } ntjd| j d} t| | dd Std |d) a Returns the zero_point and scale for the given data. :param data: The data for which to compute quantization parameters. :param quant_type: The quantization data type. :param symmetric: whether symmetric quantization is used or not. :parameter reduce_range: True if the quantization range should be reduced. Defaults to False. :parameter min_real_range: Minimum floating-point range (i.e., rmax - rmin) to enforce. Defaults to None. :parameter rmin_override: The value of rmin to use if not None. Otherwise, uses min(data). :parameter rmax_override: The value of rmax to use if not None. Otherwise, uses max(data). :return: zero point and scale z%Weight must be given as an array not rGNgr`rz1Unsupported option reduce_range=True for float 8.rrjrz Unexpected value for quant_type=)rrmrsrtrorwrrrrrarrQ RuntimeErrorrrr}rIrKrOrMrUrSrrr1)rrrrrrrrrrrrrrrrr"compute_data_quant_params_s>   r2tuple[numpy.ndarray, numpy.ndarray, numpy.ndarray]c Cst|||||||\}}|tjkrIt||||} t| tjd@dkrDt |} t d| d| d| d| d ||| fS|tj tjtjtjtjtjfvret||||} ||| fStd|d)al :param data: data to quantize :param qType: data type to quantize to. :param symmetric: whether symmetric quantization is used or not. :parameter reduce_range: True if the quantization range should be reduced. Defaults to False. :parameter min_real_range: Minimum floating-point range (i.e., rmax - rmin) to enforce. Defaults to None. :parameter rmin_override: The value of rmin to use if not None. Otherwise, uses min(data). :parameter rmax_override: The value of rmax to use if not None. Otherwise, uses max(data). :return: minimum, maximum, zero point, scale, and quantized weights To pack weights, we compute a linear transformation - when data `type == uint8` mode, from `[rmin, rmax]` -> :math:`[0, 2^{b-1}]` and - when data `type == int8`, from `[-m , m]` -> :math:`[-(2^{b-1}-1), 2^{b-1}-1]` where `m = max(abs(rmin), abs(rmax))` and add necessary intermediate nodes to transform quantized weight to full weight using the equation :math:`r = S(q-z)`, where - *r*: real original value - *q*: quantized value - *S*: scale - *z*: zero point rcz+One of the quantized value is NaN data in [z, z], quantized_data in [z].zUnexpected value for qType=rG)rrrQranyrrmr]ravelrrrrrIrKrOrMrUrSr1) rrrrrrrrrquantized_datanp_datarrr" quantize_datas@    rweightonnx.TensorProtorraxis int | Nonequant_weight_name str | Nonec Cst|}d}|durt||||}n?|j|}t|j} d| |<g} t|D]$} || |} || } || }t|| | |}| t | | q(t | |}|rW|n|j t }|tjjkrt}||_|j|j||_ ||_tdurt|}|j|jks||krtd|jd|ddd|ddd|jdt|dd d |S|tjjtjjfvr|jtjtjfvrtd |d t t!|}tj"j#|||j|d d}|Stj"$|}tj ||d |j}tj%&||}|S)aG Returns a quantized version of the given ONNX initializer. :param weight: The ONNX initializer to quantize. :param quant_type: The final quantized data type. :param zero_point: The zero-point value to use for quantization. :param scale: The scale value to use for quantization. :param axis: The quantization axis if quantizing per-channel. Defaults to None. :param quant_weight_name: The name of the quantized initializer. If not specified, the quantized name is generated. :return: The quantized ONNX initializer. Nr&zThe initializer of shape z! could not be created, expecting z, got z and shape=z raw=rGzQuantized weights for z. must be 8-bit before packing as 4-bit values.T)rawr`)'tensor_proto_to_arrayrrshapelistrtakerqrmrreshape concatenater*TENSOR_NAME_QUANT_SUFFIXrrrQ data_typedimsextendflattencopytobytesraw_datarrstrrUrSrar\r]bytespack_bytes_to_4bitrrtensor_dtype_to_np_dtype numpy_helper from_array)rrrrrr weight_data q_weight_data channel_count channel_dimsquantized_channel_data_listr{ channel_data channel_scalechannel_zero_pointquantized_channel_data q_weight_nameq_weight_initializercheck packed_dataquant_np_dtyperrr"quantize_onnx_initializersb         r c Cs|tjjkr tdd}|rt|}n|r|tvrt|}nt|}|s.td|d|\}}|dks:|dkrQtd|d|d|j d |d |d | |S) z Return qmin and qmax, the minimum and maximum value representable by the given qType :parameter qType: onnx.onnx_pb.TensorProto.UINT8 or onnx.onnx_pb.TensorProto.UINT8 :return: qmin, qmax z;This function is not implemented for float 8 as not needed.Nr~rrrrz, dtype=z, reduce_range=z , symmetric=z, qType=) rrrQrONNX_INT_TYPE_REDUCED_RANGEgetONNX_INT_TYPE_SYMMETRIC_RANGEONNX_INT_TYPE_RANGEr1ra)rrrqrangerrrrr"r)s8     rcCst|||d\}}||S)z Helper function to get the quantization range for a type. parameter qType: quantization type. return: quantization range. r)r)rrrrrrrr"get_qrange_for_qTypeIsrrranktuple[bool, int]cCs,|dkr||n|}|dko||k}||fS)z Helper function that tries to return a normalized axis in the range [0, rank - 1]. :parameter axis: The axis to normalize. :parameter rank: The tensor rank (number of dimensions). :return (is_valid, axis_norm) rr)rr axis_normis_validrrr"normalize_axisSsrsrc_8bitr bytearraycCst|}|dkr tS|dd}t|}d}d}||dkr?||dd@d>||d@B||<|d7}|d7}||dks||krK||d@||<|S)aB Copies a source array of 8-bit values into a destination bytearray of packed 4-bit values. Assumes that the source values are already in the appropriate int4 range. :parameter src_8bit: The 8-bit element values to pack. :return A bytearray with every two 8-bit src elements packed into a single byte. rr&rArdrC)rwr)r num_elemsdst_sizedstsrc_idst_irrr"r_s  $ rc@s eZdZdZggdfddZdS)QuantizedInitializerzJ Represents a linearly quantized weight input from ONNX operators Nc C:||_||_||_||_||_||_||_||_| |_dSr() r* initializerrminsrmaxs zero_pointsscalesrrr) r,r*r r!r"r#r$rrrrrr"__init__s  zQuantizedInitializer.__init__r5r6r7__doc__r%rrrr"r}s  rc@s"eZdZdZ    dddZdS)QuantizedValuezI Represents a linearly quantized value (input\output\intializer) Nc Crr() original_nameq_name scale_namezp_name value_typer node_type node_qtype scale_type) r,r*new_quantized_namer+zero_point_namequantized_value_typerr.r/r0rrr"r%s  zQuantizedValue.__init__)NNNNr&rrrr"r(s r(c@seZdZdZddZdS)BiasToQuantizez+ Represents a bias to be quantized cCs||_||_||_dSr() bias_name input_name weight_name)r,r5r6r7rrr"r%s zBiasToQuantize.__init__Nr&rrrr"r4s r4cCs|jdkrtd|jd|jdkr|j}n^|jdkr |j}nU|jdkr)|j}nL|jdkr2|j}nC|jdkr;|j}n:|jd krD|j}n1|jd krM|j }n(|jd krV|j }n|jd kr_|j }n|jd krh|j }n td|jd|jd|j|iS)z Convert attribute to kwarg format for use with onnx.helper.make_node. :parameter attribute: attribute in AttributeProto format. :return: attribute in {key: value} format. rz attribute z does not have type specified.r&rArBrCrDrEre rz has unsupported type rG) ror1r*rr{srFgfloatsintsstringstensorsgraphs) attributerrrr"attribute_to_kwargs0            rBcs*fdd|D}t|dkr|dSdS)z Helper function to find item by name in a list. parameter item_name: name of the item. parameter item_list: list of items. return: item if found. None otherwise. csg|] }|jkr|qSrr))r item item_namerr"rsz find_by_name..rN)rw)rE item_listitemsrrDr" find_by_namesrHcCs*d}tt|D] }|||kr|}q|S)zC Helper function to return index of an item in a node list ri)rrw) elem_name elem_listelem_idxr{rrr"get_elem_indexs  rLcCstjd||g|S)z Helper function to create a Mul node. parameter inputs: list of input names. parameter output: output name. parameter name: name of the node. return: Mul node in NodeProto format. Mul)rrr )inputsoutputr*rrr" get_mul_nodesrPfilenamer identifierrcCs|j|j||jS)zp Helper function to generate a identifiable filepath by concatenating the given identifier as a suffix. )parentjoinpathstemsuffix)rQrRrrr"generate_identified_filename srWcCsddl}ddlm}ddl}|j|jdtdt|tdt||j||dd|d| d| d | dS) Nr) thresholdz Histogram:zHistogram Edges:T)fillz Tensor valueCountszTensor value V.S. Counts) sysmatplotlib.pyplotpyplotrmset_printoptionsmaxsizeprintstairsxlabelylabeltitleshow)hist hist_edgesr[pltrmrrr" apply_plots     rirGcs<ddlddl}ddlddlmmm}ddlmmm}ddl m m m t d|Gfdddj}j||d}ttj|dd  }||Wdn1sfwYd}|d } g} t|D]I} || } | } t| d |t| d |g}tt|}| | }| |}|!| |"| ||#| ||$| }| %|q}|&| t'| | D]}| (|q| )}|*| |+| ||,| }| -|| .}ttj|d d }||Wdn 1swYtj/dddvrF|j0|d}|1}t2|D]}|3|}t |4t |5q/ttj|dd C}t|D]3} || } | } t| d |t| d |g}| dtt|}|||dqWWddS1swYdS)z> Helper function to write calibration table to files. rN)CalibrationMethod TensorData TensorsDatazcalibration cache: cs eZdZfddZdS)z*write_calibration_table..MyEncodercsbt|fr |St|jr|t|jddSt|r*|jjt|dSj ||S)Nz numpy.array)rraCLS)rmr) rto_dictrstolistrra __class__r5 JSONEncoderdefault)r,objrjrkrljsonnprr"rr3s  z2write_calibration_table..MyEncoder.defaultN)r5r6r7rrrrtrr" MyEncoder2srw)clszcalibration.jsonwihighestlowestzcalibration.flatbufferswbQUANTIZATION_DEBUG0)r&1zcalibration.cache  )6ru flatbuffersrm5onnxruntime.quantization.CalTableFlatBuffers.KeyValue quantizationCalTableFlatBuffersKeyValue5onnxruntime.quantization.CalTableFlatBuffers.TrtTableTrtTable"onnxruntime.quantization.calibraterjrkrllogginginforqdumpsopenospathjoinwriterrBuildersortedkeysrnfloatr rCrr CreateString KeyValueStartKeyValueAddKeyKeyValueAddValue KeyValueEndrqTrtTableStartDictVectorrwPrependUOffsetTRelative EndVector TrtTableStartTrtTableAddDict TrtTableEndFinishOutputenvironGetRootAsTrtTable DictLengthrDictKeyValue)calibration_cachedirrrrrw json_datafilerbuilderkey_value_listkeyrd_valuesr<rflat_key flat_value key_value main_dict cal_tablebufdict_lenr{rrtr"write_calibration_table"sx                     $r-C6?cCs|dktj}|dktj}|}|j|}|sdS|t|t|}|dks8Jd|d|d||tj}|||| |7}|dkdksSJ|S)a~Given a discrete distribution (may have not been normalized to 1), smooth it by replacing zeros with eps multiplied by a scaling factor and taking the corresponding amount off the non-zero values. Ref: http://web.engr.illinois.edu/~hanj/cs412/bk3/KL-divergence.pdf https://github.com//apache/incubator-mxnet/blob/master/python/mxnet/contrib/quantization.py rNrzn_zeros=z , n_nonzeros=z, eps1=)rrmrusumsizer)pepsis_zeros is_nonzerosn_zeros n_nonzeroseps1rfrrr"smooth_distribution~s " r model_pathcCs(tj|dd}tdd|jjDS)NF)load_external_datacss|]}t|VqdSr()ruses_external_data)r intializerrrr" sz*model_has_external_data..)rloadas_posixrgraphr )rmodelrrr"model_has_external_datasropt_model_pathcCsFt}||_tj|_i}dg|d<t||fddgi|}dS)z Generate model that applies graph optimization (constant folding, etc.) parameter model_path: path to the original onnx model parameter opt_model_path: path to the optimized onnx model :return: optimized onnx model ConstantSharingdisabled_optimizers providersCPUExecutionProviderN)rroptimized_model_filepathrORT_ENABLE_BASICgraph_optimization_levelr)rr sess_optionkwargs_rrr"optimize_models    rrrcCs>ddi}|jr|jD] }||j|jiq tj||dS)z>Tag the model that it went through quantization pre-processingonnx.quant.pre_processonnxruntime.quantNmetadata_propsupdaterrrrset_model_props)rrproprrr"add_pre_process_metadatas  rcC0|jr|jD]}|jdkr|jdkrdSqdS)zCCheck the model whether it went through quantization pre-processingrrTFrrr)rrrrr"model_has_pre_process_metadatas  rcCs>ddi}|jr|jD] }||j|jiq tj||dS)N onnx.inferrr)rrrrrr"add_infer_metadatas  rcCr)NrrTFr)rrrrr"model_has_infer_metadatas  rcCs2dd|jD}t|dkrtd|dj}|S)NcSs g|] }|jr |jdkr|qS)r)domain)r opsetrrr"rs z%get_opset_version..r&z$Failed to find proper ai.onnx domainr) opset_importrwr1version)rai_onnx_domain opset_versionrrr"get_opset_versions   r weight_typecCst|}|}t|d|}|dkr"|tjjkr"td|dd}n|dkr0td|dn|dkr?td|dd}||krNtj||}t |}|S) NrVz$The original model opset version is z, which does not support quantization to float 8. Please update the model to opset >= 19. Automatically update the model to opset 19. Please verify the quantized model.rze, which does not support node fusions. Please update the model to opset >= 11 for better performance.z, which does not support quantization. Please update the model to opset >= 11. Automatically update the model to opset 11. Please verify the quantized model. ) rrrrrQrwarningversion_converterconvert_version&save_and_reload_model_with_shape_infer)rrrtarget_opset_versionweight_quant_typerrr"update_opset_versions*    rcCsBt|d}tjt|t|t|}t|||S)Nz -inferred) rWrshape_inferenceinfer_shapes_pathrrrrunlink)rinferred_model_pathrrrr"load_model_with_shape_infers rcCsdtjdd"}t|}t|d}tj||ddt |WdS1s+wYdS)Nz ort.quant.)prefixz model.onnxT)save_as_external_data) tempfileTemporaryDirectoryrdeepcopyrrTr save_modelrr)r quant_tmp_dir model_copyrrrr"rs  $rr rcCs>|jtjjtjjfvrtj|Std|j dt |j)Nz&Only float type is supported. Weights z is ) rrrrrrrto_arrayr1r* type_to_name)r rrr"r s  r tensor_namecC|dS)N_QuantizeLinearrrrrr"add_quant_suffixr cC|tSr()QUANT_INPUT_SUFFIXr rrr"add_quant_input_suffixr rcCr )N_QuantizeLinear_Outputrr rrr"add_quant_output_suffixr rcCr )N_DequantizeLinearrr rrr"add_dequant_suffix!r rcCr )N_DequantizeLinear_Inputrr rrr"add_dequant_input_suffix%r rcCrr()DEQUANT_OUTPUT_SUFFIXr rrr"add_dequant_output_suffix)r r)NN)FN)FNNN)rrrrrrrrrrrrrrrr)rr)rrrrrrrrrrrrrr)FF)rrrrrr)rrrr)rQrrRrrr)rG)r)rr)rrrr)rr)rrrr)rrrr)rrrr@rr)rrrr)rrrr)r rrr)rrrr)rr)l __future__rrrrrenumrpathlibrrmrrrrrr onnx.helperr r r r onnx.referencer onnxruntimerrrrr ImportErrorrronnx.reference.op_runr __producer__ __version__ onnx_domain ms_domain QUANT_OP_NAMErDEQUANT_OP_NAMErrMODEL_SIZE_THRESHOLDrrrr%r<r@rXrIrarKrOrMrQrUrSrrrr]r\r_r^rrr r}rrrrrr rrrrrr(r4rBrHrLrPrWrirrrrrrrrrrrrrr rrrrrrrrr"s        &$ $$$  $ $$$$$$$    4?) ? B O  %    \      $