o 3Ihz^@sddlZddlZddlZddlZddlZddlZddlmZmZm Z m Z m Z m Z m Z ddlmZmZddlmZeeZddZdd d Zdd d Zd dZgdZdgddgddgdZGdddZ         dddZdddZdS)N)AttributeProto GraphProto ModelProto NodeProto TensorProtohelper numpy_helper) infer_shapesinfer_shapes_path)versioncCsdd|DS)z| Convert numpy float16 to python int. :param np_list: numpy float16 list :return int_list: python int list cSs.g|]}tt|dddddqS)HN)intbinviewzfill).0_r`/home/air/sanwanet/gpt-api/venv/lib/python3.10/site-packages/onnxruntime/transformers/float16.py $s.z%_npfloat16_to_int..r)np_listrrr_npfloat16_to_intsr"\o>@cCsdd}|t|dkjddkrF|t|dk}|t|dk}||kr7td|d|||krFtd|d||t|dkjddkr|t|dk}|t|dk}|| kr{td|d| || krtd|d| t|d||||}t|| |d| |}t|||td||}t|td|| | |}t|S)a? Convert float32 numpy array to float16 without changing sign or finiteness. Positive values less than min_positive_val are mapped to min_positive_val. Positive finite values greater than max_finite_val are mapped to max_finite_val. Similar for negative values. NaN, 0, inf, and -inf are unchanged. cSst||k||kSN)np logical_and)abcrrrbetween/sz&convert_np_to_float16..betweenrzthe float32 number z will be truncated to infz-inf) rwhereshapemaxminloggerdebugfloatfloat16)np_arraymin_positive_valmax_finite_valr" positive_max positive_min negative_max negative_minrrrconvert_np_to_float16's(   r3cCst|tstdt||jtjkrMtj|_|jr7tt |j||}t |}||j dd<g|jdd<|j rMt j|j dd}t|||}||_ |S)aConvert tensor float to float16. Args: tensor (TensorProto): the tensor to convert. min_positive_val (float, optional): minimal positive value. Defaults to 1e-7. max_finite_val (float, optional): maximal finite value. Defaults to 1e4. Raises: ValueError: input type is not TensorProto. Returns: TensorProto: the converted tensor. 3Expected input type is an ONNX TensorProto but got Nfloat32dtype) isinstancer ValueErrortype data_typeFLOATFLOAT16 float_datar3rarrayr int32_dataraw_data frombuffertobytes)tensorr-r. float16_dataint_list float32_list float16_listrrrconvert_tensor_float_to_float16Is    rIcCst|j}t|j|j|Sr)rto_arrayr%rmake_tensor_value_infonamer;)rDr%rrrmake_value_info_from_tensorns rM)ArrayFeatureExtractor BinarizerCastMapCategoryMapperDictVectorizerFeatureVectorizerImputer LabelEncoderLinearClassifierLinearRegressor Normalizer OneHotEncoderRandomUniformLike SVMClassifier SVMRegressorScalerTreeEnsembleClassifierTreeEnsembleRegressor TreeEnsembleZipMapNonMaxSuppressionTopKRoiAlignRangeCumSumMinMaxUpsampler )Resize GroupNorm SkipGroupNormc@s,eZdZdZdefddZdefddZdS) InitializerTrackerz'Class for keeping track of initializer. initializercCs||_g|_g|_dSr)ro fp32_nodes fp16_nodes)selfrorrr__init__s zInitializerTracker.__init__nodecCs$|r |j|dS|j|dSr)rpappendrq)rrrtis_node_blockedrrradd_nodeszInitializerTracker.add_nodeN)__name__ __module__ __qualname____doc__rrsrrwrrrrrnsrnFc +s |dksJd|tttjjksJd|durin|} t|trb|} tt j tdkr]|s]t j t j| d} | j} t| | t | }d}Wdn1sWwYnt | }t|tsptdt|d}|stt j td krzt}Wnw|durt}|durg}t|}t|}td |d |d d |d|d|d|g}g}g}g}|dur||}||i}t}t}dd|jjD}dd|jjD}ttrfdd|D}fdd|D}nsg}g}t |jjD]V\}}|j|vr\dt|}|||j<|!|jdt|}|jj"!}|#|||_t$j%|jj&_'t(j)d|jg|gt$j%|dg}|jj*+||||!|qt |jjD]U\}}|j|vrdt|}|||j<|!|jdt|}|jj"!}|#|||_t$j%|jj&_'t(j)d|g|jgd|dg}|jj*+||||!|qdi}|rg} |D]}!t|!tr| |!jt|!t,r|!j-D]}|j.t$j/kr|j|vsJt0|||j<q|!j*D]}|j|vrqt1t2|jD]}|j||vr||j||j|<q t1t2|jD]}|j||vr:||j||j|<q&|j3|vpF|j|v}"t |jD](\}}||vrs|"pj|t45|j3gvoj|| 5|j3gv}#||6||#qL|"r||q|j3dkr|j7D]}$|$jdkr|$j8t$j/krt$j%|$_8nq|j3dvrd}%|j7D]}$|$jd krd}%|$j8t$j/krt$j%|$_8q|j3d!vr|%s|j7+t(9d t$j%g|j3t4vs|j3| vr|j7D]}$| |$qq||qt|!t:r'| |!j;|!j|!j=|||!j?D] }t>|||}qt|!t,rzt@A|!j|!j|!j"D]B}|jj&j't$j/krS|j|vrSt$j%|jj&_'|||jBd"rx|jjCj'j&j't$j/krx|j|vrxt$j%|jjCj'j&_'||q7q| }|s|DD]*}&|s|&jErt>|&j-|||&_-|tF|&j-|&jGr|stHd#|&jEq|D]o}'t |'jD]f\}}|t4|'j3vs|| 5|'j3gvrѐq|D]J}(||(jkr|jj"!}|#|(|'jd$t|}||_t$j/|jj&_'|'jd%t|}t(j)d|g|gd|dg}|jj*+|||'j|<nqӐqq| r(t$jInt$j/})|D]}'t1t2|'jD]T}|'j|}|D]I}(||(jkr|jj"!}|#|(|'jd$t|}||_|)|jj&_'|'jd%t|}t(j)d|g|g|)|dg}|jj*+|||'j|<nq?q6t1t2|'jD]T}|'j|}*|D]I}(|*|(jkr|jj"!}|#|(|'jd&t|}||_|)|jj&_'|'jd't|}t(j)d|g|*gd(|dg}|jj*+|||'j|<nqqq-|S))aConvert tensor float type in the input ONNX model to tensor float16. Args: model (ModelProto or str): The ONNX model or path of the model to convert. min_positive_val (float, optional): minimal positive value. Defaults to 5.96e-08. max_finite_val (float, optional): maximal finite value of float16. Defaults to 65504. keep_io_types (Union[bool, List[str]], optional): It could be boolean or a list of float32 input/output names. If True, model inputs/outputs should be left as float32. Defaults to False. disable_shape_infer (bool, optional): Skips running onnx shape/type inference. Useful if shape inference has been done. Defaults to False. op_block_list (List[str], optional): List of op types to leave as float32. Defaults to None, which will use `float16.DEFAULT_OP_BLOCK_LIST`. node_block_list (List[str], optional): List of node names to leave as float32. Defaults to None. force_fp16_initializers(bool): force converting all float initializers to float16. Default to false, which will convert only the one needed to avoid precision loss. force_fp16_inputs(Dict[str, List[int]]): Force the conversion of the inputs of some operators to float16, even if this script's preference it to keep them in float32. Raises: ValueError: input type is not ModelProto. Returns: ModelProto: converted model. rzginvalid min_positive_val. smallest positive float16 value: subnormal 5.96e-08, and normalized 6.104e-05z4invalid max_finite_val. largest float16 value: 65504Nz1.8.0)dirTz$Expected an ONNX ModelProto but got z1.2.0z"fp16 parameters: min_positive_val=z max_finite_val=z keep_io_types=z disable_shape_infer=z op_block_list=z node_block_list=z force_fp16_initializers=cS"g|] }|jjjtjkr|jqSrr: tensor_type elem_typerr<rLrnrrrr"z,convert_float_to_float16..cSr}rr~rrrrrrcg|]}|vr|qSrrr keep_io_typesrrr crrrrrrrr rgraph_input_cast_graph_input_castCast)torLgraph_output_cast_graph_output_castrjr)EyeLike Multinomial RandomNormalRandomNormalLike RandomUniformrZ SequenceEmpty BernoulliFr7)rrr sequence_typezXinitializer is used by both fp32 and fp16 nodes. Consider add these nodes to block list: _input_cast_ _input_cast _output_cast_ _output_cast )Jr*rfinfor+r&r8strr parseonnx __version__tempfileNamedTemporaryFileospathdirnamerLr loadrr9r:r DEFAULT_OP_BLOCK_LISTsetr(r)rugraphinputoutputlist enumerateadd value_infoCopyFromrr=rrr make_nodertextendrror;r<rnrangelenop_typeALWAYS_FLOAT_INPUTSgetrw attributeimake_attributerggraphstrItensors itertoolschainHasFieldrvaluesrqrMrpinfoBFLOAT16)+modelr-r.rdisable_shape_infer op_block_listnode_block_listforce_fp16_initializersforce_fp16_inputs#use_bfloat16_as_blocked_nodes_dtypeforce_fp16_inputs_dict model_pathtmpfileshape_infer_model_pathfunc_infer_shapequeuevalue_info_list node_listmixed_float_type_node_list name_mappinggraph_io_to_skipio_casts fp32_inputs fp32_outputsrr output_name node_namenew_value_infonew_node input_namefp32_initializers next_levelqrvuse_fp32_weightattr has_dtypevaluertr accuracy_typerrrrconvert_float_to_float16s $     ,                                               d $                 rcCst|tstdt||jtjkrtdd}|jr#t|j}|j r.tj |j dd}|dur6t dt |||}t t|t|S)zSMeasure the maximum absolute difference after converting a float tensor to float16.r4z#Expected tensor data type is float.Nr5r6zexternal data not loaded!)r8rr9r:r;r<r>rr?rArB RuntimeErrorr3amaxabsr5)rDr-r. float32_datarErrrfloat_to_float16_max_diffs    r)rr) rrFFNNFNF)rloggingrrnumpyrrrrrrrrronnx.shape_inferencer r packagingr getLoggerrxr(rr3rIrMrrrnrrrrrrs<$   "%" >