U *-eс@s~ddlZddlmZddlmZddlmZmZddlm Z m Z m Z ddl m Z ddlZddlmZddlZddlmZmZdd lmZd gZGd d d eZd dZedddZeddGdddejjZddZddZej j!j"j#ej j!j$j#ej j!j%j#ej j!j&j'ej j!j(j#ej j!j)j*ej j!j+j#ej j!j,j#iZ-eee'dddZ.eeeeeedddZ/ed dd!d Z0dS)"N)Node) compatibility)FakeTensorMode FakeTensor)tree_map tree_flatten tree_map_only)StorageWeakRef)Enum)SetDict) defaultdict reinplacec@seZdZdZdZdZdS) _ViewTyperN)__name__ __module__ __qualname__NonViewSingleOutputViewMultiOutputViewrrZ/var/www/html/Darija-Ai-Train/env/lib/python3.8/site-packages/torch/fx/passes/reinplace.pyrsrcCsL|dk rHt|tjjrH|j}t|jdkrH|jd}|jdk oF|jj SdS)Nr) isinstancetorch_ops OpOverload_schemalen arguments alias_infois_writeZtgtZschemaZ first_argrrr _is_view_ops  r$)returncCsd|dk r^t|tjjr^|j}t|jdkr^|jd}|jdk r^|jjs^d|jj krXt j St j St j S)Nr*)rrrrrrr r!r"Z after_setrrrrr#rrr_get_view_types  r'F)Zis_backward_compatiblecs.eZdZedfdd ZfddZZS)_FunctionalizationMetadataPropnodec sr|jd7_t|}||jd<|j|jd<|j}|jtjjj j krT|dd}|j dkrt |j}|t jkrt|jdtst|jd|jd<n`|t jkr|jd|j|<nD|jtjkr|jd}|j|d}|dk rt|tst||jd<d|jkrnt|jdtstt|jdjdts6tt|jd}t|jdjd}||ksnt|S)Nr fake_resultnode_idx call_functionrview_of) node_countersuperrun_nodemetaargstargetropsatencopy_defaultopr'rrrrAssertionErrorrmulti_output_view_nodes _operatorgetitemgetrr _typed_storage) selfr*resultZ node_argsZ view_typeZlist_argZmaybe_base_of_viewZ view_storageZ base_storage __class__rrr18s6            z'_FunctionalizationMetadataProp.run_nodec sHi|_d|_t,fdd|D}tj|W5QRSQRXdS)Ncsg|]}|qSr)Z from_tensor.0amoderr ssz<_FunctionalizationMetadataProp.propagate..)r;r/rr0run)r@r3Z fake_argsrBrHr propagatens z(_FunctionalizationMetadataProp.propagate)rrrrr1rL __classcell__rrrBrr(5s6r(cCs|jdo|jdd|jk}t|jt|jkoNtddt|j|jD}|jdjdk rn|jdjjsrttdd|jddDst|o|S)N_rDcss|]\}}|j|jkVqdSN)type)rFZa1Za2rrr xsz!_schemas_match..rcss|]}|jdkVqdSrO)r!rErrrrQ}sr) nameendswithrr allzipr!r"r:)Zfunctional_schemaZinplace_schemaZ names_matchZarg_types_matchrrr_schemas_matchvs  " rVcsttjjsdStrdSjdd}jj}t tj |}|dkrNdnt ||dddkrldSfdd D}fdd|D}t |dkrdSt |dkst |d}|S) N.rDrNcsg|]}t|qSr)getattr)rFZ overload_name)maybe_inplace_oprrrJsz)_maybe_get_inplace_op..csg|]}tj|jr|qSr)rVr)rFf)r9rrrJsrr)rrrrr$rsplitZoverloadpacketrrXr5Z overloadsrr:)r9Z op_namespaceZ op_base_nameZmaybe_namespace_moduleZinplace_overloadsZ'inplace_overloads_with_matching_schemasZ inplace_opr)rYr9r_maybe_get_inplace_ops*    r\)tensor_aliasesop_indexcCsvdd}t}|D]^}|j}|D]N}d|jks |jd|kr>q ||krdt|jtjjs |jtj krdq | |q q|S)NcSs t|tr|t|dSrO)rraddr r?)xset_rrr_add_if_tensors z2_get_all_later_node_usages.._add_if_tensorr,) setusersr2rr4rrrr<r=r_)r]r^rbZnodes_used_aftertZ usage_nodesnrrr_get_all_later_node_usagessrg)later_node_usages self_aliasesr%c Cs2dd}t}t|dddD] }|jtkr0q|jd}|jd}t|tsRtt|jdt sftt|tsttt|jdt stt|j}|D]}d |jkrq|jd } z^|| jdf|jd d|j } |jd} || jd|jdr || | r | |Wqt k r(YqYqXqq|S) NcSs0||ko.||ko.||kSrO)sizeZstrideZstorage_offset)rGbrrrmatching_view_metadatas z=_get_view_inverse_node_usages..matching_view_metadatacSs |jdS)Nr,r2r`rrrz/_get_view_inverse_node_usages..)keyrrr+r.r) rcsortedr4_VIEW_INVERSE_MAPr3rrr:r2rkwargsr_ Exception) rhrirlZview_inverse_nodesrfbaseZ mutated_viewZ original_viewZ self_aliasZself_alias_baseZview_replay_metadataZexpected_metadatarrr_get_view_inverse_node_usagess4      " rwTc( slt|j|dd|jjD}tt|jjD],djkr0fdd}t|jdq0t}t|jjD]\}j dkrpt j t j jsqptj jjdkrqptj jjdjt jkrqpjdtjd\}}tjd\}}d } t|t|kr\t||D]F\} } | | kr2d } | j| jkrDd } t | dkrd } q| rxj t jjjjkrxqpj} tjd } | |krqptfd d jDdkrqptjd } | }t!|jd }t"||}t||dk}|sqpj t#kr|krt#j }|j$Zjd}jdd}|j%d|ft&|j'}|j%dt jjj(j||fi}W5QRX|)nt*j }|dkrqp|_ tjd }| +||+| |+|t,-g|D]8jdfdd j.D}|D] }g}|j}fdd}t/t0||j|_t/t0||j'|_'tjd\}}t|jd\}}dd|D} dd|D}!t| dkr2t|!dkr2| |!kr2tjd\}"}dd|"D}#t|#dkst1| \}$|#\}%|!\}&|&+|%|%+|&q2q qp|D]}'|j2|'qL|3|S)a Given an fx.GraphModule, modifies it to perform "reinplacing", mutating the nodes of the graph. We look for out-of-place op call sites like `b = a.add(...)`, and convert them to be inplace (`b = a.add_(...)`), as long as the input to the current operator ("a") isn't re-used anywhere later in the graph. This pass currently expects to operate on a **functional, ATen** graph. This can be obtained by running `make_fx(functionalize(f))`. Sample inputs are needed to determine aliasing relationships of the inputs. In general, we can't reinplace node `b = a.add(...)` if "a" aliases any of the inputs to the program. Given a node "b = foo(a, args...) the algorithm for re-inplacing is as follows: (1) Perform some initial checks on the metadata of "a" and "args..." that can disqualify them from being reinplaced. (1a) Check that the self argument we're attempting to reinplace has acceptable dtype/size metadata to reinplace with. For example, if we have: a = torch.ones(1) b = torch.ones(10) out = torch.add(a, b) We can't turn that into a.add_(b) Because that would require resizing "a". Similarly, we can't convert torch.ge(a, b) into a.ge_(b), because that would require changing a's dtype (from e.g. float32 to bool). Note that in this specific example, we could technically do better.. If we see the pattern: a_1 = a.ge(b) a_2 = aten._to_copy(a_1, a.dtype) Then we this should be valid to completely re-inplace (this is exactly what functionalization will emit when it sees a.ge_(b)). This optimization is only really important for user programs that directly use inplace comparison ops though. We also cannot re-inplace on tensors that have overlapping memory, e.g. torch.ones(1).expand(4, 4).add_(1) (1b) Check if "a" is an alias of any of the program inputs. If it is, skip and move to the next node. Inplace'ing an op that would cause it to mutate a program is not sound, because that would be a side effect visible to the user. NOTE: there's a future optimization that we should make: if "a" is a (alias of a) program input, but later in the program there is a node that looks like "a.copy_(...)", Then re-inplacing is ok to do - we are temporarily re-using a's buffer, which will later be overwritten by the copy_() call. This will be an important optimization to have for programs that mutate their inputs. It currently isn't implemented though. (1c) Check if "a" and "args..." alias For example, re-inplacing to create code like the below isn't guaranteed to be sound: aten.mul_(a, a) (2) Check that "a" and all of its outstanding aliases are not used anywhere later in the graph. If this is the case, then it's safe to re-inplace to "b = foo_(a)". There are a few caveats to this, explained in more detail below: (a) If "a" is used later as an argument to a view op, that is okay. It's only a problem if "a" (or that view) is later passed into a normal operator, or if it is returned as the program output. (b) If "a" is a repeat argument in `foo()`, then don't reinplace. Most ATen kernels don't make any guarantees that this is sound, e.g. if you do aten.mul_(a, a). So we'll just ban re-inplacing in this case. It's only a problem if "a" (or that view) is later passed (c) If "a" is used as an input into a view "inverse" / "scatter" operator, it is potentially fine to re-inplace (and remove that scatter operator from the graph). See below for a more detailed example. NOTE: there is an optimization in this step that is crucial to fully recovering performance from functionalization. Given this program: def f(x): a = torch.ops.aten.add(x, x) b = torch.ops.aten.diagonal(a) torch.ops.aten.fill_(b, 0) return d Functionalization will emit the following: def f(x): a = torch.ops.aten.add(x, x) b = torch.ops.aten.diagonal(a, 0, 1) b_updated = torch.ops.aten.fill(b, 0) a_updated = torch.ops.aten.diagonal_scatter(a, b_updated, 0, 1) return a_updated Ordinarily, we would not be able to reinplace the fill, because "b" aliases with "a" which is used by the diagonal_scatter call. "re-inplacing" is on the hook for figuring out that it is ok to completely, the expensive diagonal_scatter call, if we re-inplace the add(). So, for every `alias in alias_set(a)`, instead of checking that "alias" is not used anywhere later in the graph, we check that EITHER: (a) alias is not used anywhere later in the graph OR: (b) alias is used exactly once later on in the graph, in the following op: out = foo_scatter(alias, x, args...) where the following must hold: (i) "foo_scatter" is the "inverse" operator for foo. This only applies to "foo" ops that are view operators, which view into a subset of the original tensor's memory. In practice, there are ~4 operators where this applies: diagonal -> diagonal_scatter slice -> slice_scatter select -> select_scatter as_strided -> as_strided_scatter (ii) "args..." are the same between the foo() and foo_scatter() calls. (3) Perform the actual re-inplacing on foo! (3b) is the common case, but special care is needed for {view}_scatter (3a) (3a) {view}_scatter ops. Consider this program: a = torch.zeros(2, 2) b = torch.ones(2) a[0] = b Post functionalization, that will look like: a = torch.zeros(2) b = torch.ones(1) a_updated = torch.select_scatter(a, b, 0, 0) In this case though, there is no "functional" op to re-inplace! Instead, we'd like to directly remove toe select_scatter call. We already know from (3) that this is valid, because "a" has no later usages in the graph. We perform the re-inplacing on the {view}_scatter op like so Before: a_updated = torch.select_scatter(a, b, args...) After: a_slice = a.select(a, args...) a_slice.copy_(b) (3b) Otherwise, replace the functional op with its inplace variant. Before: b = foo(a, args...) After: a.foo_(args...) (4) Finally, after converting either: Before: b = foo(a) After: foo_(a) or Before: b = {slice}_scatter(a, mutated_slice, args...) After: slice = {slice}(a, args...) slice.copy_(mutated_slice) We now need to find all later nodes that use "b" as an argument and update them to take in "a" instead. Note that for the majority of inplace ops, this isn't actually necessary (because most inplace ops return "self" as their output). This isn't generally true for all mutable ops though, which is why we need to actually replace all of the arguments. We also need to update our metadata of Dict[StorageWeakRef, Set[Node]], That maps a given tensor storage to the set of all nodes that take in that storage as an input. Specifically, re-inplacing `b = foo(a)` causes "a" and "b"'s sets to get fused together. (5) Any "view_inverse/scatter" nodes that were identified as "it's ok to ignore them" during step (3) get manually deleted from the graph. Their outputs are no longer used, so technically standard DCE would be able to do this, but we can no longer run FX's DCE pass now that we have mutable ops in the graph. cSs(h|] }|jdkrt|jdqS) placeholderr+)r9r r2r?)rFr*rrr s   zreinplace..r+cs$t|tr t|dSrO)rrr r?r_rn)rfstorage_to_nodesrr _add_to_maps zreinplace.._add_to_mapr-rrFTcsg|]}|kr|qSrrrFr`)self_argrrrJszreinplace..r,rNcs$g|]}|jdjdkr|qS)r,rm)rFrfr)rrrJdscs|kr S|SrOr)rG)newoldrr replace_argiszreinplace..replace_argcSs"h|]}t|trt|qSrrrr r?r|rrrryvs  cSs"h|]}t|trt|qSrrr|rrrryzs  cSs"h|]}t|trt|qSrrr|rrrrys  )4r(rLgraphZnodesr rcr2r enumerater9rr4rrrrrr rPZ TensorTyper3rrUZnumelZdtypeZ_debug_has_internal_overlapr5r6resizer8rRr r?rgrwrsZinserting_beforeZ create_nodetuplertr7r_r\update itertoolschainrdrrr:Z erase_nodeZ recompile)(ZgmZ sample_argsZinput_storagesr{Z&all_later_view_inverse_nodes_to_deleteidxZself_flattenedrNZnode_flattenedZself_has_wrong_metadataZ self_metaZ node_metaZ self_arg_nameZself_arg_storagerirhZlater_view_inverse_node_usagesZ can_reinplaceZview_opZmutated_slice_nodeZremaining_slice_argsZ slice_nodeZ copy_noderYZcurr_node_storageZnodes_to_updateZnode_to_updatenew_argsr3rZold_flattened_resZnode_flattened_resZold_res_storageZnode_res_storageZnew_flattened_resZnew_res_storageZold_refZnew_refZnode_refZ to_deleter)rfr~r*rr}rzrrsJ                &)1rZtorch.fxrZtorch.fx._compatibilityrZtorch._subclasses.fake_tensorrrZtorch.utils._pytreerrrZ torch.multiprocessing.reductionsr r<enumr rtypingr r collectionsr __all__rr$r'ZfxZ Interpreterr(rVr\r5r6Zdiagonal_scatterr8ZdiagonalZselect_scatterselectintZ slice_scattersliceZTensorZas_strided_scatterZ as_stridedrsrgrwrrrrrs>     @#     ,