U Ç-eÿ<ã@sÔddlZddlmZmZmZddlmZeddd„iƒeejejejejejdœd d „ƒƒZddd„Z d d„Z eejejejejejdœdd„ƒZddd„Zdd„Z ddd„ZGdd„dejjƒZGdd„dƒZdS)éNé)ÚcdivÚ heuristicsÚjit)ÚlanguageÚEVEN_KcCs|d|ddkS)NÚKÚTILE_Kr©)Únargsr r ú^/var/www/html/Darija-Ai-Train/env/lib/python3.8/site-packages/triton/ops/blocksparse/matmul.pyÚór )ÚTILE_MÚTILE_Nr ÚBLOCKrc*Cs@t d¡|}||d7}t d¡}t |d¡}t |d¡}||t d|¡|}t d|¡}||||||dd…df||ddd…f|}t |d¡}||t d|¡|}t d|¡} ||||||ddd…f| | dd…df| }!tj||ftjd}"t|d|ƒD]}#|rPt |¡}$t |!¡}%n@tj||ddd…f|#kdd}$tj|!| dd…df|#kdd}%|"tj|$|%tjd7}"|||7}|!|| 7}!q0|" |j j ¡}&t d|¡|}'t d|¡|}(||||||'dd…df| |(ddd…f|})tj|)|&d d dS)Nrréé©Údtypeg)ÚmaskÚother©Z out_dtypeT©r)ÚtlÚ program_idÚloadÚarangeÚzerosÚfloat32ÚrangeÚdotÚtorÚ element_tyÚstore)*ÚAÚBÚCZ stride_zaÚ stride_haZ stride_maÚ stride_akÚ stride_zbÚ stride_hbÚ stride_bkZ stride_nbÚ stride_zcÚ stride_hcZ stride_mcZ stride_ncrZgrid_offsetÚlutrrr rrÚblock_idZoff_zÚoff_hZstart_amÚoffs_amÚoffs_akZa_ptrsZstart_bnÚoffs_bnÚoffs_bkZb_ptrsÚaccÚkÚaÚbÚcÚoffs_cmÚoffs_cnÚpcr r rÚ_sdd_kernelsf ÿþýüÿþýü ÿþýür>c CsÐ| d¡dkr$| d¡dkr$| ¡}| d¡dkrH| d¡dkrH| ¡}|rd||}}||}}|rldnd} |rxdnd}|j| |j|}} || kr°td|›d| ›dƒ‚| dkrâtj|jd |jd ||f|j|jd }n(| j|jd |jd ||fkst‚| }|jdd|jd g}t ||||| d ¡| d¡| |rJdnd¡| |r\dnd¡| d ¡| d¡| |r~dnd¡| |rdnd¡| d ¡| d¡| d¡| d¡|d |||d|ddd |S)NrrréþÿÿÿéÿÿÿÿzInner dimension mismatch (A: z vs B: ú)r©rÚdeviceé é)rrr rÚ num_stagesÚ num_warps) ÚstrideÚ contiguousÚshapeÚ ValueErrorÚtorchÚemptyrrCÚAssertionErrorr>)r8r9Útrans_aÚtrans_bÚtrans_cÚspdimsÚblockr/ÚwidthsÚoutZa_dimZb_dimZKaZKbr:Úgridr r rÚ sdd_matmulTsX *$ù rWcCs&|jdd |¡ ¡}| ¡}|dfS)NF©Úas_tuple)Únonzeror"ÚintrI)ÚlayoutrSrCr/r r rÚsdd_lutvsr])rrr ÚGROUP_SIZE_Mrc4Csþt d¡}t d¡}t d¡}t d¡}t |||||¡\}}t d¡}||d}t |d¡}t |d¡}t |d¡}t |d¡} ||}!t |!d¡}"t |"d¡}"t d|¡}#t d|¡}$||||"||#dd…df||$ddd…f|}%||t d|¡}&t t |&||¡|¡}&t |!¡}'t |'d¡}'|'t d|¡}(|||| ||&ddd…f| |(dd…df| })tj||ftj d}*|!d7}!t |!d¡}+t |+d¡}+t |!¡},t |,d¡},t |d|ƒD]|}-t |%¡}.t |)¡}/|*tj|.|/tj d7}*|%|+7}%|)|,| 7})|!d7}!t |!d¡}+t |+d¡}+t |!¡},t |,d¡},qê|* |j j¡}0||t d|¡}1||t d|¡}2|| ||||1dd…df| |2ddd…f|}3tj|3|0|2ddd…f|kd dS) NrrrrErérrr)rrZnum_programsZ swizzle2drZmultiple_ofrZmax_contiguousrrr r!r"rr#r$)4r%r&r'Z stride_azr(Z stride_amr)r*r+r,Z stride_bnr-r.Z stride_cmZ stride_cnZDS0ZDS1r/rrr r^rZpid_mZpid_nZ num_pid_mZ num_pid_nZpidzÚheaderÚoffsetrÚcolumnr1Zpincr0r2r3Úpar4Zstart_bkr5Zpbr6Zinc_aZinc_br7r8r9r:r;r<r=r r rÚ_dsd_kernel‚s‚ ÿþý ÿþý ÿþýürdc sÆ| d¡dkr$| d¡dkr$| ¡}| d¡dkrH| d¡dkrH| ¡}|||rTdnd} | d¡‰| d¡}| |r|dnd¡‰|j}ˆ} |}|r˜ˆn| }|r¤| nˆ}| dkrÌtj| |||f||jd}n| j| |||fksât‚| }d}‡‡‡fdd„}t ||||| d¡| d¡| |r$dnd¡| |r6dnd¡| d¡| d¡| |rXdnd¡| |rjdnd¡| d¡| d¡| |rŒdnd¡| |rždnd¡ˆ| |||t |d ƒ|d d d d|S)NrrrrrBé€cstˆ|dƒˆˆgS)Nr)r)Úmeta©ÚBS0ZBS3Úwidthr rr ærzdsd_matmul..rDrE)rrr rrFrGr^)rHrIÚsizerrLrMrCrJrNrdÚmin)r8r9rOrPrQrRrSr/rirUZAS1ÚBS1rZCS0ZCS1ZCS2ZCS3r:rrVr rgrÚ dsd_matmulÎs^ ù rmcCs¢t ||rdnd¡}t |¡jdd\}}| ¡}||}|rL|jdd} n| dd¡jdd} | d¡} t |¡}tj|dd…dd |dd…<t || dt |¡¡}| dd…df|}| ¡} | dd…|dd…8<||}| dd¡ d|¡} || dd…dd…f<| dd…df|d|8<|||dk| ||dkdf<| d¡} |rvtj | |jd }nštjgtj|jd}d}t| d¡ƒD]r}||dd…dd…f ¡ ¡}| ¡}dtj ||jd ||dk<t |||j|jdkdf¡}||7}qœ|||}|dd…|dd…||8<| dd¡ d|¡}|r||dd…dd…f<|dd…df|d|8<n<|||dd…dd…f<|dd…df|d||8<|||dk|||dkdf<| d¡}| d¡}|d|d|}||}tj||||fdd d¡ ¡}tj| |fdd d¡ ¡}tjd |j|jd}t ||f¡}t ||f¡}| tj¡ |¡}||fS)a Generates the look-up table for incrementing pointers in the DSD/DDS matmul. Example (BLOCK=32, STEP=16) [[1, 0, 0, 1, 0], [0, 1, 1, 0, 1], [1, 0, 1, 0, 0]] Then the offsets for A are [0 , 16, 32, 48] <- row 0 \----/ \----/ col=0 col=3 [64, 80, 96, 112, 128, 144] <- row 1 \----/ \----/ \------/ col=1 col=2 col=3 [160, 176, 192, 208] which leads to increments table [0, 16, 16, 16, || 64, 16, 16, 16, 16, 16, || 160, 16, 16, 16] Because B is dense, the offsets are [0, 16, 96, 112] <- row 0 [32, 48, 64, 80] <- row 1 [0, 16, 64, 80] <- row 2 rrTrXFrNr@)Údim)rCrBrEé)rCr)rLÚsumZ ones_likerZÚflattenZ transposerjZ zeros_likeZcumsumrkÚcloneÚviewÚrepeatrrCZtensorZint64r ÚlongÚcatÚTÚstackrIrrÚtypeZint32r")r\rSÚstepZtransrCÚsizesZhead_idZcol_idÚsegmentsZnnzZ num_blocksÚoffsetsZB_idxZB_incsÚdivZA_idxÚcurrent_offsetÚzZlayoutwZmsumZA_incsrir`ZincsÚpadr/r r rÚdsd_lutôsd "$"$ r‚c Cs"t|||||||||| d S©N)rU)rm) r8r9rOrPrQrRrSr/rirUr r rÚ dds_matmulZsr„c@s0eZdZeeedœZedd„ƒZedd„ƒZ dS)Ú_matmul©ÚsddÚdsdÚddscCsxtj||||||||| | |d }| ||¡||_||_| |_||_||_||_||_ ||_ ||_||_|dk |_ |Srƒ)r…ÚfnZsave_for_backwardÚda_lutÚda_widthÚdb_lutÚdb_widthÚmoderRrSrOrPrQÚhas_out)Úctxr8r9rOrPrQrrRrSÚc_lutÚc_widthr‹rŒrrŽrUr:r r rÚforwardfs" z_matmul.forwardc Csè|j\}}d\}}|j}|jdrh|d|d|d}tj||||j|j|j|j|j |j |jƒ }|jdr¸|d|d|d}tj||||j|j|j|j|j |j|j ƒ }|jrÂ|nd} ||dddddddddddd| fS)N)NNrrr)Z saved_tensorsrZneeds_input_gradr…rŠrQrPrOrRrSr‹rŒrrŽr) r‘Zdcr8r9ÚdaÚdbrZmode_daZmode_dbZdoutr r rÚbackward{sT ÿ ÿ þz_matmul.backwardN) Ú__name__Ú __module__Ú__qualname__rWrmr„rŠÚstaticmethodr”r—r r r rr…bs r…c@s eZdZddd„Zddd„ZdS) ÚmatmulFc CsB|dkrtdƒ‚||_||_||_||_||_||_|j|_t |dƒ}|jdkr”t |||ƒ\|_|_t |||d|ƒ\|_|_t |||d|ƒ\|_|_|jdkrèt ||||j|ƒ\|_|_t |||ƒ\|_|_t ||||j|ƒ\|_|_|jdkr>t ||||j|ƒ\|_|_t ||||j|ƒ\|_|_t |||ƒ\|_|_dS) Nr†z"Supported modes are: sdd, dsd, ddsrDr‡TFrˆr‰)ÚNotImplementedErrorrSrrOrPrQr\rJrRrkr]r’r“r‚r‹rŒrrŽ) Úselfr\rSrrCrOrPrQrzr r rÚ__init__•s, zmatmul.__init__NcCsBt |||j|j|j|j|j|j|j|j |j |j|j|j |¡}|S)N)r…ÚapplyrOrPrQrrRrSr’r“r‹rŒrrŽ)ržr8r9rUr:r r rÚ__call__s$ûzmatmul.__call__)FFF)N)r˜r™ršrŸr¡r r r rrœ“s rœ)N)N)N)rLÚrrrrrZ constexprr>rWr]rdrmr‚r„ZautogradÚFunctionr…rœr r r rÚs6ÿù@ "ùK &f 1