U *-e0@sddlmZddlmZmZmZmZddlmZm Z GdddZ GdddeZ Gdd d eZ Gd d d eZ Gd d d eZGdddeZeeeedddZe eee fe dddZee eee fee e fdddZe e edddZee ee e fedddZdS) )Enum) NamedTupleDictListSet)Nodemap_argc@s@eZdZdZeddddZddZdd Zd d Zd d Z dS) PartitionzPartition class contains all the information about an individual partition. It also provides necessary methods for manipulation the partition. N) partition_idreturncCs4t|_||_t|_t|_d|_d|_g|_dS)Nr)setnodesr parentschildrenZ bfs_levelused_mem_byteslogical_device_ids)selfr rh/var/www/html/Darija-Ai-Train/env/lib/python3.8/site-packages/torch/fx/experimental/partitioner_utils.py__init__ szPartition.__init__cCs t|jSN)strr rrrr__str__szPartition.__str__cCs,d|_|jD]}|jt||j7_q dS)Nr)rrget_extra_size_of)rnoderrrrecalculate_mem_sizes zPartition.recalculate_mem_sizecsdit|jfddt|jfddD]}|jdkr0|j|q0|j||dS)Ncs |Sr setdefaultn input_nodesrrz$Partition.add_node..cs |Srrr r"rrr$ r%>get_attr placeholder)rargskwargsopraddr)rrr!rr"radd_nodes  zPartition.add_nodecs|jkrj|it|jfddt|jfddD]2}tfdd|jDrF|jdkrFj|qFdS)Ncs |Srrr r"rrr$.r%z'Partition.remove_node..cs |Srrr r"rrr$/r%c3s|]}|jkVqdSr)r.0r!rrr 4sz(Partition.remove_node..>r&r') rremoverr(r)allusersr*r)rrZ input_noder)r#rr remove_node(s   zPartition.remove_node) __name__ __module__ __qualname____doc__intrrrr,r3rrrrr s   r c@s&eZdZUeed<eed<eed<dS)DevicenameZavailable_mem_bytesZ logical_idN)r4r5r6r__annotations__r8rrrrr9;s r9c@seZdZUeed<eed<dS) NodeLatencymem_latency_seccomputer_latency_secNr4r5r6floatr;rrrrr<As r<c@s&eZdZUeed<eed<eed<dS)PartitionLatencyr=r>overall_latency_secNr?rrrrrAHs rAc@s eZdZdZdZdZdZdZdS) PartitionModerN)r4r5r6 size_basedZ sparse_nnZ cost_awareZkl_basedZ aot_basedrrrrrCQs rCc@seZdZUeeed<ejZeed<dZ e ed<iZ e e efed<iZe e efed<iZe eeefed<dZeed <d S) PartitionerConfigZdevicesmodetransfer_rate_bytes_per_secnode_to_latency_mappingnode_to_partition_mapping#partition_to_logical_device_mappingF saturate_hostN)r4r5r6rr9r;rCrHrJrLr@rMrrr<rNr8rOrPboolrrrrrIYs   rI)rrr csit|jfddt|jfddd}D]0}||kr4t|dd}|r\||j7}q4tdq4t|dd}|r||j7}ntd|S)zGiven a node and a set of nodes, this function return the extra size that needed if this node is included in this set. cs |Srrr r"rrr$kr%z#get_extra_size_of..cs |Srrr r"rrr$lr%r size_bytesNznode has no size_bytes attr)rr(r)getattr output_size RuntimeError total_size)rrZtotal_size_of_input_nodesr!rRrr"rrds     r) partitionrMr csttttddd}ttdfdd |}tdddd}|D](}|tdddd}|j|jkrF|}qF|S) zVGiven a partition and its nodes' latency, return a PartitionLatency for this partition)rWr cslg}jD]\}|jdkrq it|jfddt|jfddtfddDs ||q |S)z>Given a partition, return a list of nodes on the top bfs level>r&r'cs |Srrr r"rrr$r%zEget_latency_of_one_partition..get_top_nodes..cs |Srrr r"rrr$r%c3s"|]}|jko|jdkVqdS)>r&r'N)rr*r-)rWrrr/szFget_latency_of_one_partition..get_top_nodes..)rr*rr(r)anyappend)rW top_nodesrr)r#rWr get_top_nodess    z3get_latency_of_one_partition..get_top_nodes)rr c s|}|jt|j|j}|j|j}|j|j}t|jj}|rtdddd}|D]&}|t|||} | j|jkr\| }q\|St|||S)zyGiven a top node of a partition, this function returns the latency of the critical path in the partition rKr=r>rB) rBmaxr>r=r r2 intersectionrrA) rpartition_latencyZ node_latencyrBr=r>r2Z max_latencyr!Znew_partition_latency dfs_helperrMrWrrrasH  z0get_latency_of_one_partition..dfs_helperrKr\)r rrrArB)rWrMr[rZZcritical_path_latencyrr_rr`rget_latency_of_one_partitions.,rb) partitionsrMr cCs$i}|D]}t||}|||<q|S)zGiven all the partitions and node_to_latency_mapping dictionary, return a mapping dictionary of each partition to its overall latency )rb)rcrMpartition_to_latency_mappingrWr_rrr get_partition_to_latency_mappings re)parent_partitionchild_partitionrLcs|jgkr$|jgkr$|j|jkr$dSd}t}|jD]t}it|jfddt|jfddD]>}||jkrh||krht|dd}|dk r||j7}||qhq4||S)zfGiven two partitions (parent and child), calculate the communication latency between the two. rKrcs |Srrr r"rrr$r%z*get_comm_latency_between..cs |Srrr r"rrr$r%rRN) rr rrr(r)rSrTr+)rfrgrLZ comm_sizeZ visited_nodesrr!rRrr"rget_comm_latency_betweens(     rh)rcrdrLcsbtttdfdd ttttddd}||}d}|D]}|d}||krB|}qB|S)zGiven all partitions in a graph, find the critical path among all partitions and return its latency as the latency of the whole graph )rWlatency_so_far_secr csX||j7}|j}|jrTd}|jD]*}t||}|||}||kr$|}q$|S|S)zJThis function helps to recursively get the latency of a path of partitionsrK)rBrrh)rWrirZmax_latency_secchildZcomm_latency_secZnew_latency_secrardrLrrras( z4get_latency_of_partitioned_graph..dfs_helper)rcr cSs*g}|D]}t|jdkr||q|S)zvThis function is to return all the partitions without parents as the starting points of all the paths r)lenrrY)rctop_partitionsrWrrrget_top_partitions,s  z.get_top_partitionsrK)r r@r)rcrdrLrnrmZcritical_path_latency_secrWZ latency_secrrkr get_latency_of_partitioned_graph s   roN)enumrtypingrrrrZ torch.fx.noderrr r9r<rArCrIr8rrbrer@rhrorrrrs2 4    Z   %