U }fR@sddlZddlZddlmZddlmZddlmZddlm Z m Z m Z ddl m Z mZddlZddlZddlZddlmZddlmZddlZddlmZddlZddlZd d Zd d Zd dZddZddZddZd1ddZddZ ddZ!ddZ"e e e#e fe e e#e fe e e#e fddd Z$e e e#e fe e e#e fe e e#e fdd!d"Z%d#d$Z&d%d&Z'd'd(Z(d)d*Z)d+d,Z*d-d.Z+d/d0Z,dS)2N)detect)Counter stopwords)AnyDictList)medianstdev) STOP_WORDS)tqdmc Cst|d}|}W5QRXg}|D]f}t|}|d|d|d|d|d|d|d|d |d d }||q&|S) Nrcidtextauthorchannelvotesphotoheartreply time_parsed) rrrrrrrrr)open readlinesjsonloadsgetappend) file_pathfiledataresultline json_dataZ attributesr#P/var/www/html/Presentation_Sanoa_Al_oula_2024_Ramdan/src/apps/dashboard/utils.pyread_json_files"   r%c Csi}|D]F}|d}tj|d}||kr||kr>d||<||d7<qg}|D],}|d}||d}|||gdqXdd|DS) N Timestampz%Y-%m-%d %H:%M:%Srz%A)namercSsg|]}|ddqS)rrr#).0itemr#r#r$ Esz%count_ads_by_date..)datetimestrptimedatestrftimerr) ads_dataZ week_datesZ date_countadZdate_strdate_obj chart_dataday_namecountr#r#r$count_ads_by_date-s"   r6csVtt|d\}}tj|d|ddfddtdD}dd|D}|S)Nz-Wz-1z %Y-W%W-%wcsg|]}tj|dqS))days)r, timedelta)r)iZ first_dayr#r$r+Jsz"get_week_dates..cSs$g|]}tj|ddqS)%Y-%m-%d)r,r-r/r.)r)r.r#r#r$r+Ks)mapintsplitr,r-range)Z week_stringyearZweek_numdatesZ date_objectsr#r:r$get_week_datesGs rCcCs*|D] }|d|kr|dgSqgS)Nidbrands)r)categories_datarDcategoryr#r#r$get_brands_by_idNs rHcCsj|d}dd|D}|D]$}|d}||kr||d7<qdd|D}dd|D}|S) N,cSsi|] }|dqSrr#)r)brandr#r#r$ Ysz&count_ads_by_brand..Brandr'cSsi|]\}}|dkr||qSrJr#r)rKr5r#r#r$rLcscSsg|]\}}||dqS))r(yr#rNr#r#r$r+fsz&count_ads_by_brand..)r?items)r0 brand_strrEZ brand_countr1rKrr#r#r$count_ads_by_brandTs rRcCs|dd}|dkr,tj|dgd}n$|dkrHtj|dgdd}ntd |jd d gd d }|jdd}|D]}t|d|d<d|d<qrt |}t |}|S)N.Zxlsxr&) parse_datesZcsvz%m/%d/%Y %I:%M:%S %p)rU date_formatzFile type not supported.Z Ad_IframeZ Day_Iframer')Zaxisrecords)Zorientrr5) r?lowerpd read_excelZread_csv ValueErrordropZto_dictstrrdumpsr)rZ file_typedfrdatumr" json_objectr#r#r$ get_excel_adsns    rbFc st|d}t|}W5QRXg}d}tt|dd}|D]}|d|dD]} | d|krR| d| d} tfd d |Dd} | dkrggd d d d d } || | d D]&} | d| dkr| d| dq| dD]&} | d| dkr| d| dq| dd7<| d| 7<qRq<|D]8} | dd kr8| d| d| d<| d|| d<q8g}|D]:} | ddkrzd| dkrzd| dkrz|| qz||S)Nr zProcessing frames)totaldescr'responseZ ConfidenceNamec3s|]}|dkr|VqdS)objectNr#)r)dr(r#r$ s z'generate_static_json..r)riparents categoriesr5total_confidenceavg_confidencesecondsZParentsrmZ Categoriesrnr5rorprqzEvents and AttractionszWeapons and Military) rrloadr lenupdatenextrclose)Zoriginal_json_file_pathZstatic_json_file_pathconfidence_valuefZ original_jsonresultsZfpsZpbarframeobjZ confidenceobj_dictparentrGZ final_resultr#rkr$generate_static_jsonsP       *rcCstt|SN)setrwords)langr#r#r$ get_stopwordssrcCstj||d}g}tt|D]n}tj|d|ddtj|d|ddtj|d|dd|d|dd }||qt |}t |}|S) Reads an Excel file and converts its data to a JSON object. Args: filepath (str): The file path of the Excel file. Returns: A JSON object containing the data from the Excel file. ) sheet_nameZ StartDater<Z StartTime%H:%M:%SZEndTimez Event Namer)r. time starttime endshowviews) rYrZr@rtr,r-r/rrr^r)filepathrr_rr9r`r"rar#r#r$excel_to_json_epgs     rcCst|}g}tt|D]`}dtj|d|ddtj|d|ddtt|d|dd }| |qt |}t |}|S) rzAl-Aouladayr<minuter%I:%M %pZpurcenti@)rr.timeviewers) rYrZr@rtr,r-r/r>roundrrr^r)rr_rr9r`r"rar#r#r$excel_to_json_viewss    r)json1json2returnc Csg}|D]}|}tj|dd}|D]\}tj|dd}tj|dd}|d} ||krz|kr.nq.| |d<qq.||q|S)a^ Adds a 'show' key to each dictionary in json1 based on the time range in json2. Args: json1: A list of dictionaries representing the first JSON. json2: A list of dictionaries representing the second JSON. Returns: A list of dictionaries representing the updated JSON1, with a 'show' key added to each item. rrrrrr)copyr,r-rr) rr updated_jsonitem1 updated_itemritem2 start_timeend_timerr#r#r$add_show_to_viewss  rcsg}|D]}|}tj|ddtj|dd|d}fdd|D}t||d<|rtt|nd|d <|rt|nd|d <|rt|t|nd|d <|rt|nd|d <t|d krt |nd|d<| |q|S)a Calculates some statistics based on the views in json1 within the time range of each show in json2. Adds attributes 'total', 'max', 'min', 'mean', 'median', 'ecart-type views' to each show in json2. Args: json1: A list of dictionaries representing the first JSON. json2: A list of dictionaries representing the second JSON. Returns: A list of dictionaries representing the updated JSON2, with added attributes to each show. rrrrcs>g|]6}tj|ddkr.krnq|dqS)rrr)r,r-r)r)rrrr#r$r+Os  z%add_calcul_to_epg..rermaxminmeanr r'z ecart-type) rr,r-rsumrrrtr r r)rrrrrrZviews_in_ranger#rr$add_calcul_to_epg6s   rcst|}|dkr td}tn|dkr8td}tngS|dsP|d||}fdd|D}t|}t| t | dd d d }fd d|DS) a Returns a list of words in the input text, in descending order of frequency rank. Parameters: text (str): The input text to extract words from. Returns: list: A list of dictionaries containing each word in the text, its frequency rank, and count. enZen_core_web_smfrZfr_core_news_smZ sentencizercs,g|]$}|js|js|jkr|jqSr#)Zis_stopZis_punctrrX)r)tokenrr#r$r+zs z speech_words..cSs|dSNr'r#xr#r#r$zspeech_words..Tkeyreversecs g|]\}}|||dqS)wordrankr5r#r)rr5 total_wordsr#r$r+s) rspacyrsr FR_STOP_WORDSZhas_pipeZadd_piperrvaluessortedrP)rrZnlpdocr word_counts sorted_wordsr#rrr$ speech_words_s      rcst|}|dkrtdn|dkr,tdng|}t|}fdd|D}dd|D}t|}t|t| dd d d }fd d|D}|S) Nrenglishrfrenchcsg|]}|kr|qSr#r#r)rrr#r$r+sz#speech_words_v1..cSs"g|]}tdd|Ds|qS)css|]}|tjkVqdSr)string punctuation)r)charr#r#r$rlsz-speech_words_v1...)allrr#r#r$r+scSs|dSrr#rr#r#r$rrz!speech_words_v1..Trcs g|]\}}|||dqSrr#rrr#r$r+s) rrrXnltk word_tokenizerrrrrP)rrrfiltered_wordsZ cleaned_wordsrr ranked_wordsr#rr$speech_words_v1s    rc-sDt|}|dkrjdddddddd d d d d ddddddddddddddddddd d!d"d#d$d%d&d'd(d)d*d+g*nl|d,krd-d.d/d0d1d2d3d4d5d6d7d8d9d:d;dd?d@dAdBdCdDdEdFddGdHdIdJdKdLdMdNdOdPdOdQdRdSdMdTdUdVg-ng|tdWdWtj}|}t|}fdXdY|D}t |}t | dZd[d\d]}d^dYt |D}|S)_Nraantheandorbutifthenitisarewaswereofforonintothatwiththisatfrombybenotasyouyourtheytheirr9weourusheshehimherhishersitsrununelelalesetoumaissialorsestsontétaitétaientdedudespoursurdansàaveccecetcettecesparpascommevousvotreilsleurjenousnotreilelleluisonsasescsg|]}|kr|qSr#r#rrr#r$r+sz(speech_words_copy_v4..cSs|dSrr#rr#r#r$rrz&speech_words_copy_v4..TrcSs"g|]\}\}}||ddqS)r')rrr#)r)rrr5r#r#r$r+s ) r translater] maketransrrrXrrrrrP enumerate)rrrrrrrr#rr$speech_words_copy_v4sZ` r#c-s<t|}|dkrjdddddddd d d d d ddddddddddddddddddd d!d"d#d$d%d&d'd(d)d*d+g*nl|d,krd-d.d/d0d1d2d3d4d5d6d7d8d9d:d;dd?d@dAdBdCdDdEdFddGdHdIdJdKdLdMdNdOdPdOdQdRdSdMdTdUdVg-ng|tdWdWtj}|}t|}fdXdY|D}t |}t | dZd[d\d]}d^dY|DS)_Nrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr9rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr r r r r rrrrrrrrrrrrrrrrrrcsg|]}|kr|qSr#r#rrr#r$r+sz(speech_words_copy_v3..cSs|dSrr#rr#r#r$rrz&speech_words_copy_v3..TrcSsg|]\}}||dqS)rr5r#rr#r#r$r+s) rr r]r!rrrXrrrrrPrrrrrrr#rr$speech_words_copy_v3sZ` r&csptdt|}tdtt||}fdd|D}t|}t| dddd}dd|DS) Nrcsg|]}|kr|qSr#)rXrZ stop_wordsr#r$r+s z%speech_words_copy..cSs|dSrr#rr#r#r$rrz#speech_words_copy..TrcSsg|]\}}||dqSr$r#rr#r#r$r+s) rZdownloadrrrrr?rrrPr%r#r'r$speech_words_copys  r(cCs8|}t|}dd|D}t|dddd}|S)NcSsg|]\}}||dqSr$r#rr#r#r$r+sz(speech_words_copy_v1..cSs|dS)Nr5r#rr#r#r$rrz&speech_words_copy_v1..Tr)r?rrPr)rrrr r#r#r$speech_words_copy_v1s r)cCs`|}t|}t|dddd}dd|D}g}|D]\}}||kr<|||fq<|S)NcSs|dSrr#rr#r#r$r*rz&speech_words_copy_v2..TrcSsg|]\}}||fqSr#r#rr#r#r$r+-sz(speech_words_copy_v2..)r?rrrPr)rrrrZ unique_wordsrr5r#r#r$speech_words_copy_v2"s r*)rc)-rrZ langdetectr collectionsrZ nltk.corpusrtypingrrr statisticsr r ZpandasrYrrZspacy.lang.en.stop_wordsr Zspacy.lang.fr.stop_wordsrr r,r%r6rCrHrRrbrrrrr]rrrrr#r&r(r)r*r#r#r#r$sD       F"6$6)1%$!