o 3Ih3@sddlZddlZddlZddlZddlmZ dddZddZddZd d Z d d Z   dd dZ ddZ ddZ ddZedkr}eZededdlmZeejejsfejsaJdeeZnejZddlmZeeeZeD]ZeeqvdSdS)N) TensorProtocCst}|jdddtdd|jdddtdd|jd d dtd d d |jdddtddd |jddtd dd |jddtd dd |jddtddd |jddtddd |jddtddd |jddtdd d |jd!dtdd"d |jd#dtdd$d |jd%dd&gd'd(d)|jd*d+dd,d-d.|jdd/|jd0dtd1d2d |jd3dd,d4d.|jdd5|jd6dd,d7d.|jdd8|jd9d:dd,d;|jdd<||S)=Nz-iz--inputFz2Set the input file for reading the profile results)requiredtypehelpz-mz--modelzIonnx model path to run profiling. Required when --input is not specified.z-bz --batch_sizezbatch size of input)rrdefaultrz-sz--sequence_length zsequence length of inputz--past_sequence_lengthzpast sequence length for gpt2z--global_lengthz&number of global tokens for longformerz --samplesiz\number of samples to test. Set it large enough to reduce the variance of performance result.z --thresholdg{Gz?zfThreshold of run time ratio among all nodes. Nodes with larger ratio will show in top expensive nodes.z --thread_numznumber of threads to usez--input_ids_namez"input name for input IDs, for bertz--segment_ids_namez$input name for segment IDs, for bertz--input_mask_namez'input name for attention mask, for bertz--dummy_inputsr)bertgpt2 longformerrzEType of model inputs. The default will create dummy inputs with ones.)rrchoicesrz-gz --use_gpu store_truezuse GPU)ractionr)use_gpuz --providercudazExecution provider to usez--basic_optimizationz_Enable only basic graph optimizations. By default, all optimizations are enabled in OnnxRuntime)basic_optimizationz--kernel_time_onlyz.Only include the kernel time and no fence time)kernel_time_onlyz-vz --verbose)rr)verbose)argparseArgumentParser add_argumentstrintfloat set_defaults parse_args)argvparserra/home/air/sanwanet/gpt-api/venv/lib/python3.10/site-packages/onnxruntime/transformers/profiler.pyparse_argumentss       r!c CsDddlm}||||| |dd}|D]}|d|} q|} | S)Nr)create_onnxruntime_sessionT)enable_all_optimization num_threadsenable_profiling)benchmark_helperr"run end_profiling) onnx_model_pathrproviderr thread_num all_inputsr"sessioninputs_ profile_filerrr run_profiles  r1cCs&t|dtkrt||dSdS)Nvalue)r WhichOneofrgetattr)dimrrr get_dim_from_type_protos&r6cCsdd|jjjDS)NcSsg|]}t|qSr)r6).0drrr sz-get_shape_from_type_proto..) tensor_typeshaper5) type_protorrr get_shape_from_type_protosr=c si|D]m}t|j}g}t|D]\}}t|tr!||qt|dkr+dSt|dkr7|||d<t|dkrC|||d<|jjj } | t j t j t j fvsTJ| t j kr\tjn | t j krdtjntj} tj|| d} | |j<qfddt|D} | S)a Create dummy inputs for ONNX model. Args: onnx_model (OnnxModel): ONNX model batch_size (int): batch size sequence_length (int): sequence length samples (int): number of samples Returns: List[Dict]: list of inputs Nrrdtypecg|]}qSrrr7r/ dummy_inputsrr r9z'create_dummy_inputs..)'get_graph_inputs_excluding_initializersr=r enumerate isinstancerappendlenr: elem_typerFLOATINT32INT64numpyfloat32int64int32onesnamerange) onnx_model batch_sizesequence_lengthsamples graph_inputr; symbol_dimsir5rK data_typedatar,rrCr create_dummy_inputss0            r_c CsBddlm}m}|||||\} } } ||||dd| | | dd } | S)a-Create dummy inputs for BERT model. Args: onnx_model (OnnxModel): ONNX model batch_size (int): batch size sequence_length (int): sequence length samples (int): number of samples input_ids_name (str, optional): Name of graph input for input IDs. Defaults to None. segment_ids_name (str, optional): Name of graph input for segment IDs. Defaults to None. input_mask_name (str, optional): Name of graph input for attention mask. Defaults to None. Returns: List[Dict]: list of inputs r)find_bert_inputsgenerate_test_data{F) test_casesseedr input_ids segment_ids input_maskrandom_mask_length)bert_test_datar`ra) rVrWrXrYinput_ids_namesegment_ids_nameinput_mask_namer`rarerfrgr,rrr create_bert_inputss rmcs|||||d}i|D]V}t|j}t|D]\}} t| tr4| |vr.td| || ||<q|jjj} | t j t j t j fvsFJ| t j krNt jn | t j krVt jnt j} t j|| d} | |j<qfddt|D} | S)aCreate dummy inputs for GPT-2 model. Args: onnx_model (OnnxModel): ONNX model batch_size (int): batch size sequence_length (int): sequence length past_sequence_length (int): past sequence length samples (int): number of samples Raises: RuntimeError: symbolic is not supported. Use the tool convert_to_onnx.py to export ONNX model instead. Returns: List[Dict]: list of inputs )rWseq_len past_seq_len total_seq_lensymbol is not supported: r?crArrrBrCrr r97rEz&create_gpt2_inputs..)rFr=rrGrHr RuntimeErrorr:rKrrLrMrNrOrPrQrRrSrTrU)rVrWrXpast_sequence_lengthrYsymbolsrZr;r\r5rKr]r^r,rrCr create_gpt2_inputs s0       rucs ||d}i|D]m}t|j}t|D]\}} t| tr0| |vr*td| || ||<q|jjj} | t j t j t j fvsBJ| t j krJt jn | t j krRt jnt j} d|jvrlt j|| d} d| ddd|f<nt j|| d} | |j<q fddt|D} | S) aCreate dummy inputs for Longformer model. Args: onnx_model (OnnxModel): ONNX model batch_size (int): batch size sequence_length (int): sequence length global_length (int): number of global tokens samples (int): number of samples Raises: RuntimeError: symbolic is not supported. Use the tool convert_longformer_to_onnx.py to export ONNX model instead. Returns: List[Dict]: list of inputs )rWrXrqglobalr?rNcrArrrBrCrr r9frEz,create_longformer_inputs..)rFr=rrGrHrrrr:rKrrLrMrNrOrPrQrRrTzerosrSrU)rVrWrX global_lengthrYrtrZr;r\r5rKr]r^r,rrCr create_longformer_inputs;s.         rycCs|jdkr|jntjdd}dtjvrt|tjd<ddlm}ddlm }|||j }d}|j dkrEt ||j |j|j|j|j|j}n.|j dkrWt||j |j|j|j}n|j d krit||j |j|j|j}n t||j |j|j}t|j |j|j|j|j|}|S) NrF)logicalOMP_NUM_THREADS)load) OnnxModelr r r )r+psutil cpu_countosenvironronnxr|rVr}modelrDrmrWrXrYrjrkrlrursryrxr_r1rr*r)argsr$r|r}rVr,r0rrr r'jsV       r'__main__ Arguments) setup_loggerzMrequires either --model to run profiling or --input to read profiling results)process_results)N)NNN)rrrOr~rrr!r1r6r=r_rmruryr'__name__ argumentsprintr&rrinputrr0profile_result_processorrresultslinerrrr sD   / )0/6