U 9%eE1@sddlZddlZddlmZmZddlmZGdddeZGdddeZ Gd d d e Z Gd d d e Z Gd ddeZ GdddeZ dS)N)DatasetIterableDataset) ModelOutputc@s$eZdZddZddZddZdS)PipelineDatasetcCs||_||_||_dSNdatasetprocessparams)selfr r r r ^/var/www/html/Darija-Ai-API/env/lib/python3.8/site-packages/transformers/pipelines/pt_utils.py__init__ szPipelineDataset.__init__cCs t|jSrlenr r r r r__len__szPipelineDataset.__len__cCs|j|}|j|f|j}|Srr)r iitem processedr r r __getitem__s zPipelineDataset.__getitem__N)__name__ __module__ __qualname__rrrr r r rrsrc@s6eZdZd ddZddZddZdd Zd d ZdS) PipelineIteratorNcCs4||_||_||_|dkrd}||_d|_d|_dS)a Roughly equivalent to ``` for item in loader: yield infer(item, **params) ``` Arguments: loader (`torch.utils.data.DataLoader` or any iterator): The iterator that will be used to apply `infer` on. infer (any function): The function to apply of each element of `loader`. params (`dict`): The parameters passed to `infer` along with every item loader_batch_size (`int`, *optional*): If specified, the items of `loader` are supposed to come as batch, and are loader_batched here making it roughly behave as ``` for items in loader: for i in loader_batch_size: item = items[i] yield infer(item, **params) ```N)loaderinferr loader_batch_size_loader_batch_index_loader_batch_datar rrr rr r rrszPipelineIterator.__init__cCs t|jSr)rrrr r rr?szPipelineIterator.__len__cCst|j|_|Sriterriteratorrr r r__iter__Bs zPipelineIterator.__iter__cstjtjrjj}nvi}jD]X\}}t|tr|}t|dtjrttfdd|D||<q,t|dt j r,tfdd|D||<q,|dkrt|trt|dtjrtfdd|D||<q,t|dt j r,tfdd|D||<q,|dkr"d||<q,t|jtjrL|j d||<q,t|jt j rxt |jd||<q,|j||<q,j |}jd 7_|S) ze Return item located at `loader_batch_index` within the current `loader_batch_data`. rc3s|]}|jdVqdSrNr unsqueeze.0elrr r Usz5PipelineIterator.loader_batch_item..c3s |]}t|jdVqdSr'np expand_dimsr r*rr rr-Ws> attentions hidden_statespast_key_valuesc3s|]}|jdVqdSr'r(r*rr rr-\sc3s |]}t|jdVqdSr'r.r*rr rr-^sNr) isinstancer!torchTensorr itemsrZto_tupletupler/Zndarrayr)r0 __class__)r resultZloader_batchedkelementr rrloader_batch_itemFs6    z"PipelineIterator.loader_batch_itemcCs|jdk r|j|jkr|St|j}|j|f|j}|jdk rt|tj rT|}nt | d}||}t|t rt |}n |j d}d|kr|jkrnn||_||_d|_|S|SdS)Nr)r rr=nextr%rr r4r5r6listkeysrshaper!)r rr first_tensorkeyobserved_batch_sizer r r__next__us$      zPipelineIterator.__next__)N)rrrrrr&r=rEr r r rrs  '/rcs.eZdZdfdd ZddZddZZS) PipelineChunkIteratorNcst|||dS)aK Roughly equivalent to ``` for iterator in loader: for item in iterator: yield infer(item, **params) ``` Arguments: loader (`torch.utils.data.DataLoader` or any iterator): The iterator that will be used to apply `infer` on. infer (any function): The function to apply of each element of `loader`. params (`dict`): The parameters passed to `infer` along with every item N)superrr"r9r rrszPipelineChunkIterator.__init__cCst|j|_d|_|Sr)r$rr% subiteratorrr r rr&s zPipelineChunkIterator.__iter__cCsl|jdkr"|jt|jf|j|_zt|j}Wn6tk rf|jt|jf|j|_t|j}YnX|Sr)rIrr>r%r StopIteration)r rr r rrEs zPipelineChunkIterator.__next__)N)rrrrr&rE __classcell__r r rHrrFsrFc@s eZdZdZddZddZdS)PipelinePackIteratora Roughly equivalent to ``` packed = [] for item in loader: packed.append(item) if item["is_last"]: yield packed packed = [] ``` but it also handles cases where `item` are batched (meaning it's a dict of Tensor with first dimension > 1. In that case it does ``` packed = [] for batch in loader: # item is batched for item in batch: packed.append(item) if item["is_last"]: yield packed packed = [] ``` Arguments: loader (`torch.utils.data.DataLoader` or any iterator): The iterator that will be used to apply `infer` on. infer (any function): The function to apply of each element of `loader`. params (`dict`): The parameters passed to `infer` along with every item loader_batch_size (`int`, *optional*): If specified, the items of `loader` are supposed to come as batch, and are loader_batched here making it roughly behave as ``` for items in loader: for i in loader_batch_size: item = items[i] yield infer(item, **params) ```cCst|j|_|Srr#rr r rr&s zPipelinePackIterator.__iter__cCs@d}g}|jdk rP|j|jkrP|j|jkrP|}|d}|||r|Sq|s<|jt|jf|j}|jdk r"t |t j r|}nt | d}||}t |t rt|}n |jd}d|kr|jkrnn||_||_d|_|j|jkr:|}|d}|||r|SqqP|}|d}||qP|S)NFis_lastr)r rr=popappendrr>r%r r4r5r6r?r@rrAr!)r rMZ accumulatorrrrBrCrDr r rrEsB             zPipelinePackIterator.__next__N)rrr__doc__r&rEr r r rrLs-rLc@s,eZdZeedddZddZddZdS) KeyDatasetr rCcCs||_||_dSrrR)r r rCr r rr)szKeyDataset.__init__cCs t|jSrrrr r rr-szKeyDataset.__len__cCs|j||jSrrRr rr r rr0szKeyDataset.__getitem__Nrrrrstrrrrr r r rrQ(srQc@s.eZdZeeedddZddZddZdS) KeyPairDatasetr key1key2cCs||_||_||_dSrrW)r r rXrYr r rr5szKeyPairDataset.__init__cCs t|jSrrrr r rr:szKeyPairDataset.__len__cCs"|j||j|j||jdS)N)textZ text_pairrWrSr r rr=szKeyPairDataset.__getitem__NrTr r r rrV4srV)numpyr/r5Ztorch.utils.datarrZ utils.genericrrrrFrLrQrVr r r rs -d