U Ȉd$P@sddlZddlZddlmZddlmZddlmZddlm Z m Z m Z ddl m Z mZddlZddlZddlZddlmZddlmZddlZddlmZddlZd d Zd d Zd dZddZddZd/ddZddZddZ ddZ!e e e"e fe e e"e fe e e"e fdddZ#e e e"e fe e e"e fe e e"e fddd Z$d!d"Z%d#d$Z&d%d&Z'd'd(Z(d)d*Z)d+d,Z*d-d.Z+dS)0N)detect)Counter stopwords)AnyDictList)medianstdev) STOP_WORDS)tqdmc Csi}|D]F}|d}tj|d}||kr||kr>d||<||d7<qg}|D],}|d}||d}|||gdqXdd|DS) N Timestampz%Y-%m-%d %H:%M:%Srz%A)namedatacSsg|]}|ddqS)rr).0itemrr,sz%count_ads_by_date..)datetimestrptimedatestrftimegetappend) ads_dataZ week_datesZ date_countadZdate_strdate_obj chart_dataday_namecountrrrcount_ads_by_dates"   r"csVtt|d\}}tj|d|ddfddtdD}dd|D}|S)Nz-Wz-1z %Y-W%W-%wcsg|]}tj|dqS))days)r timedelta)riZ first_dayrrr1sz"get_week_dates..cSs$g|]}tj|ddqS)%Y-%m-%d)rrrr)rrrrrr2s)mapintsplitrrrange)Z week_stringyearZweek_numdatesZ date_objectsrr&rget_week_dates.s r/cCs*|D] }|d|kr|dgSqgS)Nidbrands)r)categories_datar0categoryrrrget_brands_by_id5s r4cCsj|d}dd|D}|D]$}|d}||kr||d7<qdd|D}dd|D}|S) N,cSsi|] }|dqSrr)rbrandrrr @sz&count_ads_by_brand..BrandrcSsi|]\}}|dkr||qSr6rrr7r!rrrr8JscSsg|]\}}||dqS))ryrr:rrrrMsz&count_ads_by_brand..)r+items)r brand_strr1Z brand_countrr7rrrrcount_ads_by_brand;s r>cCs|dd}|dkr,tj|dgd}n$|dkrHtj|dgdd}ntd |jd d gd d }|jdd}|D]}t|d|d<d|d<qrt |}t |}|S)N.Zxlsxr ) parse_datesZcsvz%m/%d/%Y %I:%M:%S %p)rA date_formatzFile type not supported.Z Ad_IframeZ Day_Iframer)Zaxisrecords)Zorientrr!) r+lowerpd read_excelZread_csv ValueErrordropZto_dictstrjsondumpsloads) file_pathZ file_typedfrdatum json_data json_objectrrr get_excel_adsUs    rRFc st|d}t|}W5QRXg}d}tt|dd}|D]}|d|dD]} | d|krR| d| d} tfd d |Dd} | dkrggd d d d d } || | d D]&} | d| dkr| d| dq| dD]&} | d| dkr| d| dq| dd7<| d| 7<qRq<|D]8} | dd kr8| d| d| d<| d|| d<q8g}|D]:} | ddkrzd| dkrzd| dkrz|| qz|t|d}tj ||ddW5QRX|S)NrzProcessing frames)totaldescrresponseZ ConfidenceNamec3s|]}|dkr|VqdS)objectNr)rdrrr s z'generate_static_json..r)rZparents categoriesr!total_confidenceavg_confidencesecondsZParentsr^Z Categoriesr_r!r`rarbzEvents and AttractionszWeapons and Militaryw)indent) openrJloadr lenupdatenextrclosedump)Zoriginal_json_file_pathZstatic_json_file_pathconfidence_valuefZ original_jsonresultsZfpsZpbarframeobjZ confidenceobj_dictparentr3Z final_resultrr\rgenerate_static_jsonpsT       * rucCstt|SN)setrwords)langrrr get_stopwordssrzcCstj||d}g}tt|D]n}tj|d|ddtj|d|ddtj|d|dd|d|dd }||qt |}t |}|S) Reads an Excel file and converts its data to a JSON object. Args: filepath (str): The file path of the Excel file. Returns: A JSON object containing the data from the Excel file. ) sheet_nameZ StartDater(Z StartTime%H:%M:%SZEndTimez Event Namer)r time starttime endshowviews) rErFr,rirrrrrJrKrL)filepathr|rNrr%rOrPrQrrrexcel_to_json_epgs     rcCst|}g}tt|D]`}dtj|d|ddtj|d|ddtt|d|dd}| |qt |}t |}|S) r{z2M Marocdayr(timer}%I:%M %pi@)Zchannelrrviewers) rErFr,rirrrr*roundrrJrKrL)rrNrr%rOrPrQrrrexcel_to_json_viewss    r)json1json2returnc Csg}|D]}|}tj|dd}|D]\}tj|dd}tj|dd}|d} ||krz|kr.nq.| |d<qq.||q|S)a^ Adds a 'show' key to each dictionary in json1 based on the time range in json2. Args: json1: A list of dictionaries representing the first JSON. json2: A list of dictionaries representing the second JSON. Returns: A list of dictionaries representing the updated JSON1, with a 'show' key added to each item. rrr~r}rr)copyrrrr) rr updated_jsonitem1 updated_itemritem2 start_timeend_timerrrradd_show_to_viewss  rcsg}|D]}|}tj|ddtj|dd|d}fdd|D}t||d<|rtt|nd|d <|rt|nd|d <|rt|t|nd|d <|rt|nd|d <t|d krt |nd|d<| |q|S)a Calculates some statistics based on the views in json1 within the time range of each show in json2. Adds attributes 'total', 'max', 'min', 'mean', 'median', 'ecart-type views' to each show in json2. Args: json1: A list of dictionaries representing the first JSON. json2: A list of dictionaries representing the second JSON. Returns: A list of dictionaries representing the updated JSON2, with added attributes to each show. r~r}rrcs>g|]6}tj|ddkr.krnq|dqS)rrr)rrr)rrrrrrr6s  z%add_calcul_to_epg..rVrmaxminmeanr rz ecart-type) rrrrsumrrrir r r)rrrrrrZviews_in_rangerrradd_calcul_to_epgs   rcst|}|dkr td}tn|dkr8td}tngS|dsP|d||}fdd|D}t|}t| t | dd d d }fd d|DS) a Returns a list of words in the input text, in descending order of frequency rank. Parameters: text (str): The input text to extract words from. Returns: list: A list of dictionaries containing each word in the text, its frequency rank, and count. enZen_core_web_smfrZfr_core_news_smZ sentencizercs,g|]$}|js|js|jkr|jqSr)Zis_stopZis_puncttextrD)rtokenrrrras z speech_words..cSs|dSNrrxrrrjzspeech_words..Tkeyreversecs g|]\}}|||dqS)wordrankr!rrrr! total_wordsrrrks) rspacyrhr FR_STOP_WORDSZhas_pipeZadd_piperrvaluessortedr<)rryZnlpdocrx word_counts sorted_wordsrrrr speech_wordsFs      rcst|}|dkrtdn|dkr,tdng|}t|}fdd|D}dd|D}t|}t|t| dd d d }fd d|D}|S) Nrenglishrfrenchcsg|]}|kr|qSrrrrrrrrsz#speech_words_v1..cSs"g|]}tdd|Ds|qS)css|]}|tjkVqdSrv)string punctuation)rcharrrrr]sz-speech_words_v1...)allrrrrrscSs|dSrrrrrrrrz!speech_words_v1..Trcs g|]\}}|||dqSrrrrrrrs) rrzrDnltk word_tokenizerrrrr<)rryrxfiltered_wordsZ cleaned_wordsrr ranked_wordsrrrspeech_words_v1ws    rc-sDt|}|dkrjdddddddd d d d d ddddddddddddddddddd d!d"d#d$d%d&d'd(d)d*d+g*nl|d,krd-d.d/d0d1d2d3d4d5d6d7d8d9d:d;dd?d@dAdBdCdDdEdFddGdHdIdJdKdLdMdNdOdPdOdQdRdSdMdTdUdVg-ng|tdWdWtj}|}t|}fdXdY|D}t |}t | dZd[d\d]}d^dYt |D}|S)_Nraantheandorbutifthenitisarewaswereofforonintothatwiththisatfrombybenotasyouyourtheytheirr%weourusheshehimherhishersitsrununelelalesetoumaissialorsestsontétaitétaientdedudespoursurdansàaveccecetcettecesparpascommevousvotreilsleurjenousnotreilelleluisonsasescsg|]}|kr|qSrrrrrrrsz(speech_words_copy_v4..cSs|dSrrrrrrrrz&speech_words_copy_v4..TrcSs"g|]\}\}}||ddqS)r)rrr)rrrr!rrrrs ) r translaterI maketransrrrDrrrrr< enumerate)rryrxrrrrrrrspeech_words_copy_v4sZ` rc-s<t|}|dkrjdddddddd d d d d ddddddddddddddddddd d!d"d#d$d%d&d'd(d)d*d+g*nl|d,krd-d.d/d0d1d2d3d4d5d6d7d8d9d:d;dd?d@dAdBdCdDdEdFddGdHdIdJdKdLdMdNdOdPdOdQdRdSdMdTdUdVg-ng|tdWdWtj}|}t|}fdXdY|D}t |}t | dZd[d\d]}d^dY|DS)_Nrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr%rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr r r r r rrrrrrrrcsg|]}|kr|qSrrrrrrrsz(speech_words_copy_v3..cSs|dSrrrrrrrrz&speech_words_copy_v3..TrcSsg|]\}}||dqS)rr!rrrrrrs) rrrIrrrrDrrrrr<rryrxrrrrrrspeech_words_copy_v3sZ` rcsptdt|}tdtt||}fdd|D}t|}t| dddd}dd|DS) Nrcsg|]}|kr|qSr)rDrZ stop_wordsrrrs z%speech_words_copy..cSs|dSrrrrrrrrz#speech_words_copy..TrcSsg|]\}}||dqSrrrrrrrs) rZdownloadrrwrrxr+rrr<rrrrspeech_words_copys  rcCs8|}t|}dd|D}t|dddd}|S)NcSsg|]\}}||dqSrrrrrrrsz(speech_words_copy_v1..cSs|dS)Nr!rrrrrrrz&speech_words_copy_v1..Tr)r+rr<r)rrxrresultrrrspeech_words_copy_v1s r cCs`|}t|}t|dddd}dd|D}g}|D]\}}||kr<|||fq<|S)NcSs|dSrrrrrrrrz&speech_words_copy_v2..TrcSsg|]\}}||fqSrrrrrrrsz(speech_words_copy_v2..)r+rrr<r)rrxrrZ unique_wordsrr!rrrspeech_words_copy_v2 s r!)rS),rrZ langdetectr collectionsrZ nltk.corpusrtypingrrr statisticsr r ZpandasrErJrZspacy.lang.en.stop_wordsr Zspacy.lang.fr.stop_wordsrr rr"r/r4r>rRrurzrrrIrrrrrrrr r!rrrrs@       F"6$6)1%$!