U ‰d§Dã@s<ddlmZmZmZmZmZmZddlmZddl m Z ddl m Z ddl mZmZmZddlmZddlmZmZmZddlmZdd lmZdd lmZmZmZdd lm Z dd l!Z"dd l#m$Z%dd l&Z&dd l'Z'dd l(Z(dd l)Z)ddd„Z*ddd„Z+Gdd„deƒZ,Gdd„deƒZ-Gdd„deƒZ.Gdd„deƒZ/d S)é)ÚCompositeReaderÚCompositeReaderBuilderÚ ReaderBuilderÚReaderWithDelayÚReaderWithLimitÚReaderWithTimeLimit)ÚDataset)Ú DBFileReader)Úpipe)ÚStructÚ NewRecordÚ FeedRecord)Ú LocalSession)Ú TaskGroupÚ final_outputÚ WorkspaceType)ÚTestCase)Ú CachedReader)ÚcoreÚ workspaceÚschema)ÚopsNédc Csz|pd}t d |¡¡}t |¡Dtdt t|||ƒ¡fƒ}t||ƒ}t ||d}t |||ƒW5QRX|  |¡|S)NÚsrcú{}_initÚlabel©Úname) rÚNetÚformatÚ NameScoper ÚnpÚarrayÚranger rr Úrun)ÚwsÚsizeÚoffsetrZsrc_initZ src_valuesZ src_blobsÚsrc_ds©r)ú=/tmp/pip-unpacked-wheel-ua33x9lu/caffe2/python/dataio_test.pyÚmake_source_dataset!s    r+c CsR|pd}t d |¡¡}t |¡t||d}| |¡W5QRX| |¡|S)NÚdstrr)rrrr rZ init_emptyr$)r%rrZdst_initÚdst_dsr)r)r*Úmake_destination_dataset-s   r.c@s,eZdZdd„Zdd„Zdd„Zdd„Zd S) ÚTestReaderBuildercCs0t dt ¡f¡|_||_||_||_d|_dS)Nr)rr ZScalarÚ_schemaÚ_nameÚ_sizeÚ_offsetÚ_src_ds)Úselfrr&r'r)r)r*Ú__init__8s ÿzTestReaderBuilder.__init__cCs|jS©N)r0©r5r)r)r*rAszTestReaderBuilder.schemacCst||j|j|jd|_iS)N©r'r&r)r+r3r2r1r4)r5r%r)r)r*ÚsetupDs ÿzTestReaderBuilder.setupcKs|jSr7)r4)r5Úkwargsr)r)r*Ú new_readerIszTestReaderBuilder.new_readerN)Ú__name__Ú __module__Ú __qualname__r6rr:r<r)r)r)r*r/7s r/c@sDeZdZe ej d¡d¡dd„ƒZe ej d¡d¡dd„ƒZ dS)ÚTestCompositeReaderZ JENKINS_URLzFlaky test on Jenkinsc sLtj ¡‰tˆƒ}d}dd„t|ƒDƒ}d‰‡fdd„t|ƒDƒ}‡‡fdd„t||ƒDƒ}‡fdd„|Dƒ}t||ƒD]\}}t |t||ˆƒ¡qxtj dd„t||ƒDƒŽ} t ˆ| ƒ} t ƒ,} t |d d„|Dƒƒ} t | |  ¡dd W5QRX| | ¡t|ƒD]@} tˆ t|  ¡||  ¡ƒ¡ƒ}t || |d  | ¡¡qdS) NécSsg|]}d |¡‘qS©zsrc_{}©r©Ú.0Úir)r)r*Ú Ssz=TestCompositeReader.test_composite_reader..rcsg|] }|ˆ‘qSr)r)rD©r&r)r*rGUscs g|]\}}tˆ|ˆ|d‘qS©r9)r+©rErr'©r&r%r)r*rGVsÿcs g|]}ˆ t|jdƒ¡‘qS)r)Ú fetch_blobÚstrZ field_blobs)rEr©r%r)r*rGYscSs g|]\}}|| ¡ ¡f‘qSr))ÚcontentÚ clone_schema)rErr(r)r)r*rG`sÿcSsg|] }| ¡‘qSr))Úreader)rEr(r)r)r*rGis©Únum_runtime_threadszi: {})rÚCÚ Workspacerr#ÚzipÚnptÚassert_array_equalrr r.rrr Úwriterr$ÚsortedrLrMrOrr)r5ÚsessionÚnum_srcsÚnamesÚoffsetsZsrc_dsesÚdataÚdr'Ú dst_ds_schemar-ÚtgrQrFÚ written_datar)rKr*Útest_composite_readerNs:  ÿþÿ  ÿ  ÿz)TestCompositeReader.test_composite_readerc stj ¡}t|ƒ}d}dd„t|ƒDƒ}d‰‡fdd„t|ƒDƒ}‡fdd„t||ƒDƒ}tjdd„t||ƒDƒŽ}t||ƒ}t ƒ2} t ||ƒ} | j |dt |   ¡| ¡dd W5QRX| | ¡t||ƒD]D\} } t| t| ¡|  ¡ƒ¡ƒ} t t| | ˆƒ| d  | ¡¡qÐdS) NrAcSsg|]}d |¡‘qSrBrCrDr)r)r*rGwszETestCompositeReader.test_composite_reader_builder..rcsg|] }|ˆ‘qSr)r)rDrHr)r*rGyscsg|]\}}t|ˆ|d‘qSrI)r/rJrHr)r*rGzsÿcSsg|]\}}|| ¡f‘qSr))r)rErZsrc_ds_builderr)r)r*rGsÿrNrRzname: {})rrTrUrr#rVrr r.rrr:r r<rYr$rZrLrMrOrrWrXr)r5r%r[r\r]r^Zsrc_ds_buildersrar-rbZreader_builderrr'rcr)rHr*Útest_composite_reader_builderrs@  þþÿ ÿ ÿ ÿÿz1TestCompositeReader.test_composite_reader_builderN) r=r>r?ÚunittestÚskipIfÚosÚenvironÚgetrdrer)r)r)r*r@Ms #r@c@s\eZdZdd„Zdd„Zdd„Zdd„Zd d „Zd d „Zd d„Z dd„Z dd„Z dd„Z dS)ÚTestReaderWithLimitc s,tj ¡}t|ƒ}t|ƒ}dgd‰‡fdd„}tƒ}t| ¡d|dW5QRX| |¡|  ˆd  ¡d¡|  ˆd  ¡d¡|  ˆd   ¡d¡tƒ<}t| ¡d d }tt | ¡d d dd }t||d dW5QRX| |¡|  ˆd  ¡d ¡|  ˆd  ¡d ¡|  ˆd   ¡d ¡dS)NrAc st ¡0t gdg¡}t gdg¡}t gdg¡}W5QRXt ¡t gdg¡}W5QRXt |¡t |¡t ¡6t t |¡¡t |¡W5QRXt |¡W5QRXt ¡<t t |¡ƒˆd<t t |¡ƒˆd<t t |¡ƒˆd<W5QRX|S)NZglobal_counterZglobal_counter2Zglobal_counter3Ú task_counterréé) rZ task_initZ CreateCounterZtask_instance_initZCountUpZtask_instance_exitZloopZ RetrieveCountZ task_exitr)ZrecZcounter1Zcounter2Zcounter3rl©Ztotalsr)r*Úprocžs"      z6TestReaderWithLimit.test_runtime_threads..procé)rSÚ processorrrrmrnrRé)Únum_iteré)rrrS) rrTrUrr+rr rQr$Ú assertEqualÚfetchr)r5r%r[r(rprbZq1Zq2r)ror*Útest_runtime_threads˜s,    þ z(TestReaderWithLimit.test_runtime_threadscCs<tj ¡}t|ƒ}t||d}t|| ¡ ¡ƒ}||||fS)NrH)rrTrUrr+r.rOrP)r5r&r%r[r(r-r)r)r*Ú_test_limit_reader_init_sharedÊs   z2TestReaderWithLimit._test_limit_reader_init_sharedc Ks| |¡\} } } } ttjdH} |dkr@|t|  ¡|ƒf|Ž}n||  ¡f|Ž}t||  ¡|dW5QRX|  | ¡t t | j t |   ¡ ¡ƒ ¡ƒƒ}| |||¡| |||¡| t | j t |   ¡ ¡ƒ ¡ƒtt|ƒƒ¡| | j t | ¡ƒ ¡|¡dS)N)Zworkspace_typerrR)ryrrÚGLOBALrrQr rYr$ÚlenrZÚblobsrMrOrrwZassertGreaterEqualZassertLessEqualrvÚlistr#Z data_finished)r5Z reader_classr&Úexpected_read_lenÚexpected_read_len_thresholdÚexpected_finishÚ num_threadsÚ read_delayZ limiter_argsr%r[r(r-rbrQÚread_lenr)r)r*Ú_test_limit_reader_sharedÖs8ÿ  ÿþ "þþ þÿz-TestReaderWithLimit._test_limit_reader_sharedc Cs|jtdddddddddS)NrrTrq©r&r~rr€rr‚rt©r„rr8r)r)r*Ú%test_count_limit_reader_without_limitúsùz9TestReaderWithLimit.test_count_limit_reader_without_limitc Cs|jtdddddddddS)NrrFrqr…r†r8r)r)r*Ú'test_count_limit_reader_with_zero_limitsùz;TestReaderWithLimit.test_count_limit_reader_with_zero_limitc Cs|jtdddddddddS)Nré rFrqr…r†r8r)r)r*Ú&test_count_limit_reader_with_low_limitsùz:TestReaderWithLimit.test_count_limit_reader_with_low_limitc Cs|jtdddddddddS)NrrTrqénr…r†r8r)r)r*Ú'test_count_limit_reader_with_high_limitsùz;TestReaderWithLimit.test_count_limit_reader_with_high_limitc Cs|jtdddddddddS)NrrTrqgš™™™™™¹?©r&r~rr€rr‚Úduration©r„rr8r)r)r*Ú$test_time_limit_reader_without_limit&sùz8TestReaderWithLimit.test_time_limit_reader_without_limitc CsVd}d}d}d}tt|||ƒƒ}|d|}|jt||d|dd|||ddS)Né2égÐ?rmrnFr)ÚintÚroundr„r)r5r&rZsleep_durationrŽr~r)r)r*Ú'test_time_limit_reader_with_short_limit1s ùz;TestReaderWithLimit.test_time_limit_reader_with_short_limitc Cs|jtdddddddddS)Nr‘rTr’gš™™™™™É?r‰rrr8r)r)r*Ú&test_time_limit_reader_with_long_limitJsùz:TestReaderWithLimit.test_time_limit_reader_with_long_limitN) r=r>r?rxryr„r‡rˆrŠrŒrr•r–r)r)r)r*rk—s2 $     rkc@s„eZdZdd„Zdd„Zedd„ƒZdd„Zed d „ƒZed d „ƒZ e   d e j  ¡kd¡dd„ƒZe   d e j  ¡kd¡dd„ƒZdS)ÚTestDBFileReadercCs g|_dSr7)Ú temp_pathsr8r)r)r*ÚsetUpYszTestDBFileReader.setUpcCs|jD]}| |¡qdSr7)r˜Ú _delete_path)r5Úpathr)r)r*ÚtearDown\s zTestDBFileReader.tearDowncCs2tj |¡rt |¡ntj |¡r.t |¡dSr7)rhr›ÚisfileÚremoveÚisdirÚshutilÚrmtree)r›r)r)r*ršas   zTestDBFileReader._delete_pathc Cs*t ¡ }|j}W5QRX|j |¡|Sr7)ÚtempfileÚNamedTemporaryFilerr˜Úappend)r5ÚfZ temp_pathr)r)r*Ú_make_temp_pathhs  z TestDBFileReader._make_temp_pathcCst||ƒ}| ¡Sr7)r+rQ)r%r&r(r)r)r*Ú_build_source_readeros z%TestDBFileReader._build_source_readerc CsZt|| ¡ ¡ƒ}tƒ}t|| ¡ddW5QRX| |¡|jt|  ¡  ¡ƒ  ¡S)NrqrR) r.rrPrr rYr$r|rMrOrrw)r%rQr[r-rbr)r)r*Ú_read_all_datats  zTestDBFileReader._read_all_dataÚLevelDBz Need LevelDBc Cstj ¡}t|ƒ}| ¡}t| |d¡|dd}| ¡}| |¡|  |||¡}|  t |ƒt t dƒƒ¡t| |d¡|ƒ}| ¡}| |¡|  |||¡}|  t |ƒt t dƒƒ¡| |¡t| |d¡|ƒ}| ¡}| |¡|  |||¡}|  t |ƒt t dƒƒ¡| |¡dS)NrF)Z loop_overéÈi,)rrTrUrr¦rr§Úbuild_cache_stepr$r¨rvrZr}r#rš) r5r%r[Údb_pathZcached_reader1r«r_Zcached_reader2Zcached_reader3r)r)r*Útest_cached_reader~s<  ÿ  ÿ   ÿ z#TestDBFileReader.test_cached_readercCs‚tj ¡}t|ƒ}| ¡}t| |d¡|dd}| ¡}| |¡t |dd}|  |||¡}|  t |ƒt tdƒƒ¡| |¡dS)Nrr©)r¬Zdb_type)rrTrUrr¦rr§r«r$r r¨rvrZr}r#rš)r5r%r[r¬Z cached_readerr«Zdb_file_readerr_r)r)r*Útest_db_file_reader¦s"  ý þz$TestDBFileReader.test_db_file_readerN)r=r>r?r™rœÚ staticmethodršr¦r§r¨rfrgrrTZregistered_dbsr­r®r)r)r)r*r—Xs    'r—)rrN)N)0Zcaffe2.python.dataiorrrrrrZcaffe2.python.datasetrZcaffe2.python.db_file_readerr Zcaffe2.python.pipeliner Zcaffe2.python.schemar r r Zcaffe2.python.sessionrZcaffe2.python.taskrrrZcaffe2.python.test_utilrZcaffe2.python.cached_readerrZ caffe2.pythonrrrZcaffe2.python.net_builderrZnumpyr!Z numpy.testingZtestingrWrhr rfr¢r+r.r/r@rkr—r)r)r)r*Ús.          JB