o 3Ih@spddlmZddlmZddlmZddlmZmZm Z ddl m Z ee Z GdddeZGdd d eZd S) ) getLogger)Fusion) FusionUtils) NodeProto TensorProtohelper) OnnxModelcseZdZdZd2dedeffdd Zdedd eeefBfd d Z d ed e ee efde de fddZ ddZddZddZddZddZdedeed eBffddZ  d3ded ed ed!ed"d eBd#ed Bf d$d%Zd&d'Zd(d)Z d4d*d+Zd,d-Zd.d/Zd0d1ZZS)5FusionEmbedLayerNoMaskz Fuse embedding layer into one node (EmbedLayerNormalization). It supports the following model types: BERT, DistilBert, ALBert. no maskmodel descriptioncs<t|dddg|t||_d|_d|_d|_d|_dS)NEmbedLayerNormalizationLayerNormalizationSkipLayerNormalizationF)super__init__rutils shape_infershape_infer_done attention embed_node)selfr r  __class__j/home/air/sanwanet/gpt-api/venv/lib/python3.10/site-packages/onnxruntime/transformers/fusion_embedlayer.pyrs  zFusionEmbedLayerNoMask.__init__addreturnNcCsP|j|dgdg}|durdS|j|dgdg}|dur dS|d|dfS)NGatherr)r match_parent_path)rr gather_0_path gather_1_pathrrrmatch_two_gather%sz'FusionEmbedLayerNoMask.match_two_gather layernorminput_name_to_nodesis_distil_bertc Cs|jj|d|dd|_|jdurdS|jd|vrdS||jd}tdd|D}|gd kr_|D]+}|jd kr^|j|gd gd }|dur^|d jd|jdkr^|d|_dSq3t |dkr|djdkr|djd|vr||djd}t |dkr|djdkr|djd|vr||djd} | D] }|jdkr||_dSqtdd| D}|r|gdkr|gdkr|gdkrt ddSdS|gdkr|gd krt ddSdS)aCheck that LayerNormalization has a child of Attention node or subgraph like Attention. Args: layernorm (NodeProto): LayerNormalization node input_name_to_nodes (Dict[str, List[NodeProto]]): map from input name to nodes is_distil_bert (bool): whether it is DistilBert or not Returns: bool: whether there is Attention node or subgraph like Attention AttentionF) recursiveNTrcSg|]}|jqSrop_type.0childrrr JzCFusionEmbedLayerNoMask.check_attention_subgraph..)MatMulr1r1rr)Addr1MultiHeadAttentionr1)NNrrrr1r2cSr)rr*r,rrrr/gr0)r1r1r1Shaper)r2r1r1r1r6r6)r2r1r1r1r6z Equal(B=?) -- Not -- Cast(to=6) -- CumSum(axis=1) -- Mul -- Cast(to=7) -- Add(B=1) -- Cast(to=7)* --> Gather | ^ V | +------------------------------+ Roberta new pattern from transformers v4.9: (input_ids) --> Equal(B=?) -- Not -- Cast(to=6) -- CumSum(axis=1) -- Add(B=0) -- Mul -- Cast(to=7) -- Add(B=1) --> Gather | ^ V | +-------------------------------------------+ start_node = position_embedding_gather start_index = 1 # match optional Cast node. parent = self.model.get_parent(start_node, start_index, output_name_to_node) if parent is None: return if parent.op_type == "Cast": if OnnxModel.get_node_attribute(parent, "to") != 7: return start_node = parent start_index = 0 i, path, return_indices = self.model.match_parent_paths( start_node, [ (['Add', 'Cast', 'Mul', 'CumSum', 'Cast', 'Not', 'Equal'], [start_index, 0, 0, 0, 0, 0, 0]), (['Add', 'Cast', 'Mul', 'Add', 'CumSum', 'Cast', 'Not', 'Equal'], [start_index, 0, 0, 0, 0, 0, 0, 0])], output_name_to_node) if path is not None: # constant input of Add shall be 1. i, value = self.model.get_constant_input(path[0]) if value != 1: return False _, self.padding_word_id = self.model.get_constant_input(path[-1]) return input_ids == path[-1].input[0] FrrrOrPrQrrr match_position_embedding_robertas-z7FusionEmbedLayerNoMask.match_position_embedding_robertac Cs|j|ddgddg|}|durdS|\}}|j|jd}|durTt|jdkrT|jddkrT|j|ddgrT|j|ddgrTt|jd ksV|j|d dgsVdS|j}|d krjt |d dgsidSn |j|ddgsudS|j |d|} | durdS| j d kr|j| ddsdS|j | d|} n| } | dus| j d krdS|j| ddsdS|j | d|} | dus| j dkrdS|| jdkS)a  Match position embedding path from input_ids to Gather for BERT. BERT Embedding Layer Pattern: (input_ids) / / Shape / | / Gather (indices=1) / | / Add (optional, B=0) / | Gather (segment_ids) Unsqueeze (axes=0) \ | | \ Gather Slice (data[1,512], starts=0, ends=*, axes=1, steps=1) \ / | Add Gather \ / Add | LayerNormalization SlicerIrr5NFr axesr2rr6) r r get_constant_valuer:r<rSrrNget_opset_versionrcheck_node_attribute get_parentr+) rrOrPrQpathslice unsqueeze slice_weight opset_versionrAgatherrSrrrmatch_position_embedding_bertsT   z4FusionEmbedLayerNoMask.match_position_embedding_bertcCs(||||r dS||||rdSdS)NTF)rkrYrZrrrmatch_position_embedding9s z/FusionEmbedLayerNoMask.match_position_embeddingc Cs|jd}|r |jdnd}|jd}|js!|jjdd|_d|_|jdurs|j|}|j|}|r6|s8Jt|dkrLt|dkrL|d|dksYtd|d|dS|rs|j ||sstd |d |j|dS|j |jd } | dust| j dkrtd dS|j |jd } | dust| j dks| j d| j dkrtd dS|r|j |jd } | dust| j dks| j d| j dkrtddS| j d | j d krt d|jd d| j d d|jd d| j d |rU| j d | j d kr-t d|jd d| j d d|jd d| j d | j d | j d krUt d|jd d| j d d|jd d| j d dS)zXSanity check of embedding weights, and match hidden_size of weights and shape of inputs.rNT)updater5z^Cannot fuse EmbedLayerNormalization: input_ids and position_ids not matched in 2nd dimension: z vs FzYCannot fuse EmbedLayerNormalization: input_ids and segment_ids does not have same shape: z != rzICannot fuse EmbedLayerNormalization: word embedding table is not expectedzMCannot fuse EmbedLayerNormalization: position embedding table is not expectedzLCannot fuse EmbedLayerNormalization: segment embedding table is not expectedzword_embedding_table (z) size z <= position_embedding_table (z <= segment_embedding_table (zposition_embedding_table () r:rr infer_runtime_shaperget_edge_shaper<r=info compare_shaperarSwarning) rword_embedding_gathersegment_embedding_gatherrOrP segment_ids position_idsinput_ids_shapeposition_ids_shapeword_embedding_tableposition_embedding_tablesegment_embedding_tablerrrcheck_embeddingGsj           222z&FusionEmbedLayerNoMask.check_embedding input_namecCsdd}|j|}|dur&|jjjtjkr |j|\}}||fS|}||fS|j|\}}||fS)aCast a graph input or node input to int32. Args: input_name (str): name of graph input or node input Returns: A tuple of casted input name and the cast node. int32_output (str): If input is int32, it is the input name, Otherwise it is output name of Cast node. input_cast_node (Union[None, NodeProto]): Cast node. It could be None if input is int32. N) r find_graph_inputtype tensor_type elem_typerINT32rcast_input_to_int32)rr}input_cast_node graph_input int32_outputrrr cast_to_int32s z$FusionEmbedLayerNoMask.cast_to_int32FrPrsrOrtrvc Csg} ||\}} |jd} |jdkr|jd} |jd} n |jd} |jd} d}|durL||jd\}} |||jd|jd|jd| | g}n|d|jd|jdd| | g}|durp|d||\}} ||| d | d g}|r|dur|n| d }||tjd||| d }d |_|j D]}|j dkr|j |gqt |j dkr|j t ddg| || D] }|j|j|j <q|j | ||_|S)agCreate an EmbedLayerNormalization node. Note that segment embedding is optional. Args: input_ids (str): input_ids for word embeddings layernorm (NodeProto): LayerNormalization or SkipLayerNormalization node. word_embedding_gather (NodeProto): the Gather node for word embedding position_embedding_gather (NodeProto): the Gather node for position embedding segment_embedding_gather (Union[None, NodeProto]): the Gather node for segment embedding, or None. Returns: NodeProto: the EmbedLayerNormalization node created. r rrr5r]Nr_output_dummy_mask_index_embedding_sum)outputsnamez com.microsoftepsilong-q=)rr create_node_namer+r:appendr make_nodedomain attributerextendr<make_attributethis_graph_namenode_name_to_graph_name nodes_to_addr)rrPr$rsrOrtrvembedding_sum_outputembedding_sum_namerrT node_namegammabetaembed_node_inputsruembed_node_outputsrrattrArrrcreate_fused_nodesl              z(FusionEmbedLayerNoMask.create_fused_nodecCs$|j|jd|jdd|_dS)NrT)r replace_input_of_all_nodesr8 prune_graph)rr$rrrr finish_fusion s z$FusionEmbedLayerNoMask.finish_fusioncCs*|jdkot|jdkot|jddkS)Nrr]r)r+r<r8)rrArrr"is_skip_layer_norm_with_sum_outputs*z9FusionEmbedLayerNoMask.is_skip_layer_norm_with_sum_outputc Csx||}|dur dS|\}}|jd} |jd} |j||dds#dS||d|s,dS|jdkrP||} d} |} | rA|jdnd}|duoN|j|du}n@|} |jdkrYdnd} t |j| krg|j| nd}|duot|j|du}|o||vot ||dk}|duo|jdkp|p|} |j | ||||| | |r|ndd}| rd | j| <|s|j ||jd | ||d S) NFrr&rr]r2r)rr_no_use__to_be_removed_r5T) r#r:rEr|r+rr8r find_graph_outputr<rrr)rr$add_before_layernormr%rQoptional_segment_gather two_gatherrsrOrPrvneed_embedding_sum_outputsum_output_indexnode_with_sum_output sum_outputis_sum_graph_outputis_sum_used_by_multiple_nodesrrrr fuse_gpt2sX         z FusionEmbedLayerNoMask.fuse_gpt2c Cs||}|dur dS|\}}|jd}|j||ddsdS||||s'dS||d|s0dS|||||d} ||| dS)aFuse embedding layer for DistilBert Args: layernorm (NodeProto): node of LayerNormalization or SkipLayerNormalization add_before_layernorm (NodeProto): the Add node before LayerNormalization, or the SkipLayerNormalization itself input_name_to_nodes (Dict[str, List[NodeProto]]): map from input name to nodes output_name_to_node (Dict[str, List[NodeProto]]): map from output name to nodes NFrTr)r#r:rErlr|rr) rr$rr%rQrrsrOrPrrrrfuse_distilbertcs    z&FusionEmbedLayerNoMask.fuse_distilbertcCs|j|dgdg}|durdS||d}|durdS|\}}|jd} |j||dds0dS|j|dgdg} | dur@dS| d} || | |sZ||| |sTdS|} | }| } |||| scdS|| ||| |} ||| dS) aFuse embedding layer for Bert Args: layernorm (NodeProto): node of LayerNormalization or SkipLayerNormalization add_before_layernorm (NodeProto): the Add node before LayerNormalization, or the SkipLayerNormalization itself input_name_to_nodes (Dict[str, List[NodeProto]]): map from input name to nodes output_name_to_node (Dict[str, List[NodeProto]]): map from output name to nodes r2rNFrrrT) r r r#r:rErlr|rr)rr$rr%rQ add_2_gatherrrsrtrPposition_embedding_pathrOtemprrrr fuse_berts>   z FusionEmbedLayerNoMask.fuse_bertc Cs|j|dgdg}|jdkr|durdS|d}d}nP|j|dgdg}|j|dgdg}|durG|durG|dur>dS|d}|d}n%|durh|durh|j|dgdg}|dur_dS|d}|d}n|}d}||||||rwdS|||||rdS|||||rdSdS)Nr2rrrr)r r r+rrr) rrAr%rQfirst_add_pathrrr!r"rrrfuses<    zFusionEmbedLayerNoMask.fuse)r )NFN)N)__name__ __module__ __qualname____doc__rstrrrtupler#dictlistboolrErYr[rkrlr|rrrrrrrr __classcell__rrrrr sT  T>/HJ b Q)2r cs8eZdZd deffdd ZddZfddZZS) FusionEmbedLayerNormalizationFr cst|d||_dS)Nz with mask)rruse_mask_index)rr rrrrrs z&FusionEmbedLayerNormalization.__init__cCs|j}t|jdkr|j|td|jn"t|jdkr1|jds1||jd<td|jn td|jdS|D]$}td|j|jdkrS|jd|jd<q<|jd kr`|jd|jd <qrr+r8)r mask_int32attention_nodesrattention_noderrr replace_masks"    z*FusionEmbedLayerNormalization.replace_maskcsfd|_d|_d|_t||||jdurdS|js'td|ddS|jdur=|jdur=td|ddS|jrG|jj d}n|jj d}||}|j |rkdd|D}| |||ddS||vr|td ||ddS||}|j d vrd d|D}|j d kr|j d }t|t|kr|j|| |||ddSdS)NzG--use_mask_index is not set: EmbedLayerNormalization will not have maskz EmbedLayerNormalization(no mask)zLEmbedLayerNormalization will not have mask since attention node is not foundr]r^cSg|] }|jdvr|qS)r'r3r*r-rArrrr/z6FusionEmbedLayerNormalization.fuse..z"EmbedLayerNormalization(with mask)zHEmbedLayerNormalization will not have mask since %s is not a node output) ReduceSumrKcSrrr*rrrrr/$rrr)rr;rrrrr=r>increase_counterr:r r~rr+r<nodes_to_remover)rrAr%rQrchildren_nodesrrrrrsJ                z"FusionEmbedLayerNormalization.fuse)F)rrrrrrrrrrrrrsrN)loggingr fusion_baser fusion_utilsronnxrrr onnx_modelrrr=r rrrrrs    X