U -e@sddZddlZddlmZeeeefedddZGdddeZ Gd d d eZ Gd d d Z dS) a Hashing function for dataset keys using `hashlib.md5` Requirements for the hash function: - Provides a uniformly distributed hash from random space - Adequately fast speed - Working with multiple input types (in this case, `str`, `int` or `bytes`) - Should be platform independent (generates same hash on different OS and systems) The hashing function provides a unique 128-bit integer hash of the key provided. The split name is being used here as the hash salt to avoid having same hashes in different splits due to same keys N)Union) hash_datareturncCsLt|tr|St|tr&|dd}nt|tr:t|}nt||dS)z| Returns the input hash_data in its bytes form Args: hash_data: the hash salt/key to be converted to bytes \/zutf-8) isinstancebytesstrreplaceintInvalidKeyErrorencode)rrQ/var/www/html/Darija-Ai-Train/env/lib/python3.8/site-packages/datasets/keyhash.py _as_bytes%s    rcs eZdZdZfddZZS)r z6Raises an error when given key is of invalid datatype.csDd|_d|dt||_d|_t|j|j|jdS)Nz7 FAILURE TO GENERATE DATASET: Invalid key type detectedz Found Key z of type z- Keys should be either str, int or bytes type)prefixtypeerr_msgsuffixsuper__init__)selfr __class__rrr?szInvalidKeyError.__init____name__ __module__ __qualname____doc__r __classcell__rrrrr <sr cs"eZdZdZdfdd ZZS)DuplicatedKeysErrorz(Raise an error when duplicate key found.cs||_||_||_d|_t|dkr>dd|d||_n.dd|dddt|dd||_|rxd|nd |_t |j|j|jdS) Nz3Found multiple examples generated with the same keyz The examples at index z, z have the key z... (z more) have the key  r!) keyduplicate_key_indicesfix_msgrlenjoinrrrr)rr$r%r&rrrrIs .zDuplicatedKeysError.__init__)r!rrrrrr Fsr c@s8eZdZdZedddZeeeefedddZ dS) KeyHasherz,KeyHasher class for providing hash using md5) hash_saltcCstt||_dS)N)hashlibmd5r _split_md5)rr*rrrrYszKeyHasher.__init__)r$rcCs*|j}t|}||t|dS)zReturns 128-bits unique hash of input key Args: key: the input key to be hashed (should be str, int or bytes) Returns: 128-bit int hash key)r-copyrupdater hexdigest)rr$r,Zbyte_keyrrrhash\s  zKeyHasher.hashN) rrrrr rrr rr2rrrrr)Vsr)) rr+typingrr r rr Exceptionr r r)rrrrs