o 3Ih3+@sddlZddlZddlZddlZddlZddlZddlmZm Z m Z m Z ddl m Z mZGddde ZddZdd Zd d Zed krIedSdS) N) QuantFormat QuantTypeStaticQuantConfigquantize)CalibrationDataReaderCalibrationMethodc@s*eZdZddZdefddZddZdS) OnnxModelCalibrationDataReaderc stj|_fddtjD}t|}g}|D]1i}fddttD}fdd|D}t ||ddD] \}} | ||j <qA| |qt |t |ks[Jt |dt |ksgJt |_dS)Ncs&g|]}|drtjj|qS)test_data_set_) startswithospathjoin model_dir.0aselfo/home/air/sanwanet/gpt-api/venv/lib/python3.10/site-packages/onnxruntime/quantization/static_quantize_runner.py s  z;OnnxModelCalibrationDataReader.__init__..csg|] }tj|qSr)r r r r)data_dirrrrscsg|]}|qSr)read_onnx_pb_data)r data_pathrrrrsF)strictr)r r dirnamerlistdir onnxruntimeInferenceSession get_inputssortedzipnameappendlenitercalibration_data) r model_path data_dirs model_inputs name2tensors name2tensor data_paths data_ndarrays model_input data_ndarrayr)rrr__init__s     z'OnnxModelCalibrationDataReader.__init__returncCs t|jdS)z9generate the input data dict for ONNXinferenceSession runN)nextr&rrrrget_next!s z'OnnxModelCalibrationDataReader.get_nextcCsPt}t|d}||Wdn1swYtj|}|S)Nrb)onnx TensorProtoopenParseFromStringread numpy_helperto_array)rfile_pbtensorfretrrrr%s   z0OnnxModelCalibrationDataReader.read_onnx_pb_dataN)__name__ __module__ __qualname__r0dictr3rrrrrr s rcCstjdd}|jddddd|jdd dd d|jd gd d dd|jdgd ddd|jdddd|jdddd|jdddd|jdddd|jdddd|jdd gd!d"|jd#d$gd%d&d'|jd(d)d)d*gd+d'|jd,dd-d|jd.dd/d|jd0dd1d|jd2dd3d|jd4td5d6d7|jd8dd9d|jd:dd;d|jdd dd?d"|jd@d ddAd"|jdBdCdDdEgdFdG|jdHdIdJ|S)KNz%The arguments for static quantization) descriptionz-iz--input_model_pathTzPath to the input onnx model)requiredhelpz-oz--output_quantized_model_pathz'Path to the output quantized onnx modelz--activation_typeqint8quint8qint16quint16qint4quint4 qfloat8e4m3fnrIz!Activation quantization type used)choicesdefaultrFz --weight_typerHzWeight quantization type usedz--enable_subgraph store_truez#If set, subgraph will be quantized.)actionrFz--force_quantize_no_input_checka By default, some latent operators like maxpool, transpose, do not quantize if their input is not quantized already. Setting to True to force such operator always quantize input and so generate quantized output. Also the True behavior could be disabled per node using the nodes_to_exclude.z--matmul_const_b_onlyz3If set, only MatMul with const B will be quantized.z--add_qdq_pair_to_weightzjIf set, it remains floating-point weight and inserts both QuantizeLinear/DeQuantizeLinear nodes to weight.z--dedicated_qdq_pairzFIf set, it will create identical and dedicated QDQ pair for each node.z)--op_types_to_exclude_output_quantization+z]If any op type is specified, it won't quantize the output of ops with this specific op types.)nargsrPrFz--calibration_methodminmaxrUentropy percentile distributionzCalibration method used)rPrOrFz--quant_formatqdq qoperatorzQuantization format usedz--calib_tensor_range_symmetriczoIf enabled, the final range of tensor during calibration will be explicitly set to symmetric to central point 0z--calib_moving_averagezIf enabled, the moving average of the minimum and maximum values will be computed when the calibration method selected is MinMax.z--disable_quantize_biaszWhether to quantize floating-point biases by solely inserting a DeQuantizeLinear node If not set, it remains floating-point bias and does not insert any quantization nodes associated with biases.z--use_qdq_contrib_opszIf set, the inserted QuantizeLinear and DequantizeLinear ops will have the com.microsoft domain, which forces use of ONNX Runtime's QuantizeLinear and DequantizeLinear contrib op implementations.z--minimum_real_rangeg-C6?aIf set to a floating-point value, the calculation of the quantization parameters (i.e., scale and zero point) will enforce a minimum range between rmin and rmax. If (rmax-rmin) is less than the specified minimum range, rmax will be set to rmin + MinimumRealRange. This is necessary for EPs like QNN that require a minimum floating-point range when determining quantization parameters.)typerPrFz --qdq_keep_removable_activationsz|If set, removable activations (e.g., Clip or Relu) will not be removed, and will be explicitly represented in the QDQ model.z*--qdq_disable_weight_adjust_for_int32_biaszIf set, QDQ quantizer will not adjust the weight's scale when the bias has a scale (input_scale * weight_scale) that is too small.z --per_channelz&Whether using per-channel quantizationz--nodes_to_quantizezfList of nodes names to quantize. When this list is not None only the nodes in this list are quantized.z--nodes_to_excludeznList of nodes names to exclude. The nodes in this list will be excluded from quantization when it is not None.z--op_per_channel_axisr#)OP_TYPEPER_CHANNEL_AXISa8Set channel axis for specific op type, for example: --op_per_channel_axis MatMul 1, and it's effective only when per channel quantization is supported and per_channel is True. If specific op type supports per channel quantization but not explicitly specified with channel axis, default channel axis will be used.)rTrRmetavarrPrFz--tensor_quant_overridesz4Set the json file for tensor quantization overrides.)rF)argparseArgumentParser add_argumentfloat parse_args)parserrrrparse_arguments-s     rgcCs|siSt| }t|}Wdn1swY|D]}||D]}tj|dtjd|d<t|d|d<q%q|S)Nscale)dtype zero_point)r7jsonloadnparrayfloat32)filer>quant_override_dictr=enc_dictrrrget_tensor_quant_overridess   rsc Cst}t|jd}tjtjtjtjtjtj tj d}||j }||j }t |j}|j|j|j|j|j|j||j|j|j |j|j|j|jt|jd}tjtj tj!tj"d}t#j$t#j%d}t&|||j'||j(||d|j)|j*|j+ddd|d } t,|j|j-| ddS) N)r'rG)EnableSubgraphForceQuantizeNoInputCheckMatMulConstBOnlyAddQDQPairToWeight"OpTypesToExcludeOutputQuantizationDedicatedQDQPair QDQOpTypePerChannelSupportToAxisCalibTensorRangeSymmetricCalibMovingAverage QuantizeBiasUseQDQContribOpsMinimumRealRangeQDQKeepRemovableActivations"QDQDisableWeightAdjustForInt32BiasTensorQuantOverridesrV)rZr[F) calibration_data_readercalibrate_method quant_formatactivation_type weight_typeop_types_to_quantizenodes_to_quantizenodes_to_exclude per_channel reduce_rangeuse_external_data_formatcalibration_providers extra_options)r. model_output quant_config).rgrinput_model_pathrQInt8QUInt8QInt16QUInt16QInt4QUInt4 QFLOAT8E4M3FNrrrCop_per_channel_axisenable_subgraphforce_quantize_no_input_checkmatmul_const_b_onlyadd_qdq_pair_to_weight'op_types_to_exclude_output_quantizationdedicated_qdq_paircalib_tensor_range_symmetriccalib_moving_averagedisable_quantize_biasuse_qdq_contrib_opsminimum_real_rangeqdq_keep_removable_activations(qdq_disable_weight_adjust_for_int32_biasrstensor_quant_overridesrMinMaxEntropy Percentile DistributionrQDQ QOperatorrcalibration_methodrrrrroutput_quantized_model_path) args data_readerarg2quant_typerr'qdq_op_type_per_channel_support_to_axisrarg2calib_methodarg2quant_formatsqcrrrmainsj   r__main__)rarkr numpyrmr5ronnxruntime.quantizationrrrr"onnxruntime.quantization.calibraterrrrgrsrr@rrrrs    =