o *Tha! @s<ddlZddlmZmZmZddlZddlZddlmZm Z m Z m Z m Z m Z mZddlZddlmZddlmmmZddlmZddlmZddlmZddlmZd e eeffd d Zd ed ed efddZ ded ed eded df ddZ!ded ed eded df ddZ"ded dfddZ#ded dfddZ$dS)N)FutureThreadPoolExecutorwait)AnyDictListOptionalSetTuplecast) ClientAPI) SegmentAPI) RecordSet)test_hnsw_config)MetadatareturncCs tdd}tdd}||fS)N i')randomrandint)NDra/home/air/segue/gemini/back/venv/lib/python3.10/site-packages/chromadb/test/test_multithreaded.pygenerate_data_shapes  rrrcCsZddt|D}ddt|D}ddt|D}tj||}||||d}|S)NcSsg|]}t|qSr)str.0irrr z'generate_record_set..cSsg|]}||iqSrrrrrrrcSsg|]}d|qS)zdoc rrrrrrr!ids embeddings metadatas documents)rangenprrandtolist)rrr#r%r&r$normalized_record_setrrrgenerate_record_setsr,client num_workersc Cst||}|d}|d}|d}|d}td|d|d|d||jd td } t|d x} g} d } | t|krtd |d} t | t|| }| d }| |d }|durht|||dkrhn<| j | j ||||dur{|||nd|dur|||nd|dur|||ndd}| || |7} | t|ks>Wdn1swYt | | D] }|}|dur|qt| |t| |t| |t| tddt|Dd}d}tj| |||ddS)Nr#r$r%r&Adding  records with  dimensions on  workerstestnamemetadata max_workersrrr"cSg|]}|qSrrrrrrr[z+_test_multithreaded_add.. n_results query_indices)r,printresetcreate_collectionrrlenrrminsubmitaddappendr exception invariantscount ids_matchmetadatas_match no_duplicatessampler' ann_accuracy)r-rrr. records_setr#r$r%r&collexecutorfutures total_sent batch_sizeto_sendstartendfuturerIr@r?rrr_test_multithreaded_add)s`            r[c sH||jdtdt||}ttt|dttt|dttt|d|dt t t d|d|d |d  dd t d tttdd ffdd }t|de}g}d}|tdkrtdd} | dkrtd|d} t| t|} |d} || d} ||| | | }|||| 7}n| dkr||| }|||tdksoWd n1swYt||D] }|}|d ur|qt|tr|jdurt|t|t|t |t!t"ddt#|Dd}d}tj$|||dd S)zeTest that will use multiple threads to interleave operations on the db and verify they work correctlyr3r4r#r$r%r&r/r0r1r2N operation ids_to_modifyrcsv|dkre|dus Jfdd|D}t|dkrdSj|dur+fdd|Dnddur9fdd|DnddurGfdd|Dndd|WddS1s^wYdS|d krg}d }t}Wdn1swYfd d|D}t|dkst||krdSt|td t|}fd d|D}j||ddSdS)z,Perform a random operation on the collectionrNcg|]}|qSrindexridr#rrr~r!zJ_test_interleaved_add_query..perform_operation..cg|]}|qSrrrr$rrrr crdrrr)r%rrrr crdrrr)r&rrrr r"r:r=cr^rr_rarcrrrr!rcrdrrrrerrrr )r?) rDrGupdatelistcopyrrOrEquery)r\r]indices_to_modifycurrently_added_idsr?currently_added_indicesr@ query_vectors added_idsrRr&r$r#lockr%rrperform_operationxsP   "    z6_test_interleaved_add_query..perform_operationr7r9r:rrTcSr;rrrrrrrr<z/_test_interleaved_add_query..r=r>)N)%rBrCrr,r rrfloatrset threadingLockrAintrrrDrrrErFrHrrI isinstancer get_settings is_persistentrJ fd_not_exceeding_threadpool_sizerKrLrMrNrOr'rP)r-rrr.rQrqrSrTrUr\rVrWrXrYrZrIr@r?rrnr_test_interleaved_add_queryes|   2           r|cC>tdD]}tdtd}t\}}t||||qdSNrr)r'rrmultiprocessing cpu_countrr[r-rr.rrrrrtest_multithreaded_add  rcCr}r~)r'rrrrrr|rrrrtest_interleaved_add_queryrr)%rconcurrent.futuresrrrrrutypingrrrrr r r numpyr( chromadb.apir !chromadb.test.property.invariantsr3propertyrJchromadb.api.segmentr !chromadb.test.property.strategiesrrchromadb.typesrrwrr,r[r|rrrrrrsL$      < u