o XxiD< @sddlZddlmZmZddlmZddlZddlZddlm Z e e Z e ejeZe ejedZeee edZdZdZd Zd Zd ZeZd d ZGdddeZddZ GdddeZ!d'ddZ"ddZ#GdddeZ$e dkrd%ddfd%ddfd %dd%dfd!%dd"%dfd!%dd#%dfd$%dd%dfd%%dd%dfd&%dd%dfgZ&e"e&\Z'Z(e#e(Z)e$e)gZ*e+e*,d%de+e*,d%ddSdS)(N)packunpack OrderedDict) lru_cachez0%(asctime)s %(name)s - %(levelname)s %(message)s cCst|t|dSN)loggersetLevelhandler)levelrI/home/air/biblejyuku/back/venv/lib/python3.10/site-packages/janome/fst.pyset_fst_log_level+s rc@seZdZdZgdZdddZddZdd Zd d Zd d Z ddZ ddZ ddZ ddZ ddZddZddZddZdS)Statez State Class )idfinal trans_map final_outputNcCs||_d|_i|_t|_dSNF)rrrsetr)selfrrrr__init__6s zState.__init__cC|jSr rrrrris_final<zState.is_finalcCs ||_dSr r)rrrrr set_final? zState.set_finalcCs||jvr |j|dSdS)Nstate)rrcharrrr transitionBs zState.transitioncCs.|||jvr tn|j|dd|j|<dS)Noutput)r%r)rbytes)rr'r%rrrset_transitionHszState.set_transitioncCrr )rr rrr state_outputLr"zState.state_outputcCstdd|D|_dS)NcSsg|]}t|qSr)r+.0errr Psz*State.set_state_output..rr)rr)rrrset_state_outputOszState.set_state_outputcC t|_dSr r2r rrrclear_state_outputRs zState.clear_state_outputcCs||jvr |j|dStSNr)r*r&rrrr)Us z State.outputcCs$||jvrt||j|d<dSdSr6r*)rr'outrrr set_output[s zState.set_outputcCsd|_i|_t|_dSr)rrrrr rrrclear_s z State.clearcCs:|dus t|ts dS|j|jko|j|jko|j|jkSr) isinstancerrrr)rotherrrr__eq__ds   z State.__eq__cCs"tt|jt|jt|jSr )hashstrrrrr rrr__hash__ms"zState.__hash__r )__name__ __module__ __qualname____doc__ __slots__rr!r#r(r,r-r3r5r)r8r9r<r?rrrrr0s   rcCs\t|}|j|_|jD]\}}||t|d|||dq t|j|_|S)Nr%r))rrritemsr,copyr8r)srcrr%ctrrr copy_stateqsrJc@sLeZdZdZdZddZddZddZd d Zd d Z d dZ ddZ dS)FSTz& FST (final dictionary) class icCr4r )r dictionaryr rrrrs z FST.__init__cCs t|jSr )lenrLr rrrsizer$zFST.sizecCs|jt|Sr )rLgetr=rr%rrrmemberz FST.membercCs||jt|<dSr rLr=rPrrrinsertsz FST.insertcCs|jt|=dSr rSrPrrrremoverRz FST.removecCst|jtjkSr )rMrLrKMAX_SIZEr rrrexceed_max_sizerRzFST.exceed_max_sizec Cs|jD];}|jD]\}}tdt|jt|t|djt|dgq |r@tdt|jtdt|j gqdS)N r%r)r) rLvaluesrrEprintjoinr>rr!r)rsrHvrrrprint_dictionarys4$zFST.print_dictionaryN) r@rArBrCrVrrNrQrTrUrWr^rrrrrK{s rKcsvt|}td|tg}t}fdd}dd}t}t}d} |D]`\}}||ks2J|||} |D]} t| q9t|t|krW|tt|t|ksIt t|| dD]} || d || d||| q_t | dt|dD]} ||  || d || d|| q||kr|t| d |t| ttgt d| dD]} g}|| d|| d}d}|t|kr|t|kr||||kr||||d7}|t|kr|t|kr||||ks|t|d}|| d|| d|tD]} || | dur4||| | }|| | |q|| rXt}|| D] }|||qF||  ||t|d}q||krr|t||n || || ||}| d7} |r|q't t|ddD]} || d || d||| q||dtd | fS) Nz(partial) input size: %dcs.|}|durt|}||Sr )rQrJrNrT)r%r\fstDictrrfind_minimizeds  z1create_minimum_transducer..find_minimizedcSs`d}|t|kr.|t|kr.||||kr.|d7}|t|kr.|t|kr.||||ks|S)NrrrM)s1s2irrr prefix_lens ((z-create_minimum_transducer..prefix_lenrrTznum of state: )rMrinforKr+CHARSaddappendrranger,r9r#r3rr)r8r(r!r-debugrN)inputs on_progress inputs_sizebuffer prev_wordrarf current_wordcurrent_output processedpref_lenrHrej common_prefixr)k word_suffix new_outputtmp_settmp_strrr_rcreate_minimum_transducersx      & "(( ( r~cCsg}i}d}t|jD]\}}tt|jddD]z\}\}}t} d} dt} } |dkr5| t7} |drG| t 7} t |d} |d} | t d| 7} | t d|7} | dkrd| t d| 7} | | 7} | |dj } | d usrJ|t | d | }|dksJ| t d|7} |t| |t | 7}q|rt} t} d}|jrtd d |jDr| t7} t |j}|js| t7} | t d| 7} |r| t d|7} |jD]}t |} | t d| 7} | r| |7} q|t| |t | 7}|||j <q td t ||d|S)z5 convert FST to byte array representing arcs rT)reverser)bBIr%Nr css|] }t|dkVqdS)rNrbr.rrr -szcompileFST..zcompiled arcs size: ) enumeraterLrYsortedrrE bytearrayr+ FLAG_LAST_ARCFLAG_ARC_HAS_OUTPUTrMrrOrrkr!FLAG_FINAL_ARCranyFLAG_ARC_HAS_FINAL_OUTPUTrrmrr[)fstarcsaddressposnumr\rerHr]baryflag output_sizer) next_addrtarget output_countr7rrr compileFSTsd"         rc@s:eZdZdddZdddZdd Zed d d d ZdS)Matcherr cCsJ|r#||_t||_ddtt|D|_||_||_t|_ dSdS)NcSsg|]}tqSrr)r/_rrrr1Msz$Matcher.__init__..) dict_datarMdict_lenrlcachemax_cache_sizemax_cached_word_len threadingLocklock)rrrrrrrrHs zMatcher.__init__TcCs4t}t|jD] }|||||O}qt||fSr )rrlr_runbool)rwordcommon_prefix_matchr)rerrrrunRs z Matcher.runc Cs6t}d}d}}|j|}t|t|} } tt| |jddD]L} |d| |j|vrm|j||d| \}}}|j"|j||d| =|t||f|j||d| <Wdn1sdwY| }nq!|| kr|||} | dt @r|s|| kr| dD] } | || q| dt @s|| kr |S|| d7}||jkr|j+|t||f|j||d|<t|j||j kr|j|j ddWdn1swYn3|| kr||| d kr|| d7}|d 7}|| d 7}n| dt @r |S|| d7}n |S|| kss|S) NrrrrgF)lastrr )rrrMrlminrrrnext_arcrrjrrpopitem)rrdata_numroutputsbufrerdataword_lendata_lenrwarcr7rrrrXs\ "        "z Matcher._runi)maxsizec CsJ|dksJ|}d}t}dg}d}||}|d7}|t@ra|t@r`td|||dd} |d7}g} t| D]$} td|||dd} |d7}| r]| |||| || 7}q9| }n:||}|d7}|t@rtd|||dd} |d7}|||| }|| 7}td|||dd}|d7}|||||||fS)Nrrrrr )r+rrrrlrkr) rraddrrlabelr)rrrfinal_output_countrrrrrrrs@  zMatcher.next_arcN)rr )T)r@rArBrrrrrrrrrrGs   8r__main__aprutf830aug31decfeb2829janjuljunr )-rFstructrr collectionsrloggingr functoolsr getLoggerr@rrWARN StreamHandlerr Formatter formatter setFormatter addHandlerrrFLAG_TARGET_NEXTFLAG_STOP_NODErrrrirobjectrrJrKr~rrencodeinputs1rurrmrZrrrrrsV        A #i@s