U ,-e @s2ddlmZmZmZmZmZmZmZmZm Z ddl m Z ddl Z ddl Z ddlZddlmZddlmZmZmZddlmZmZmZmZmZddlmZmZdd lmZd d l m!Z!m"Z"m#Z#m$Z$m%Z%dd l&m'Z'm(Z(m)Z)m*Z*dd l+m,Z,m-Z-ddl.m/Z/d dl0m1Z1m2Z2d dl3m4Z4m5Z5ddl6m7Z7d dlm8Z8m9Z9m:Z:m;Z;mZ>m?Z?m@Z@ddlAmBZBmCZCddlDmEZEddlFmGZGd dlHmIZImJZJd dlKmKZKd dlLmMZMddlNZNddddgZOe jPe jQjRjSjTe jUe jQjRjVjTiZWe jXjYeeeeZe jXjYfeeZeeZe[ffeeZefddddZ\e jXjYeeeeZe jXjYfeeZeeZe[ffeeZefddd d!Z]eedd"d#d$Z^e jXjYe_d%d&d'Z`eeeZefe_d(d)d*Zaee,dd+d,d-Zbeeedd.d/d0ZceeeZeeZe[ffeeZefeeZeZfd1d2d3Zdeedd"d4d5ZeeeeZe jXjYfee jXjYd6d7d8ZfeeeZe jXjYfe jgje_ee,dd9d:dZhdFeeeZe jXjYfeeZeeZefe,e_e_dd<d=dZieeedd>d?d@ZjeeeeZe jXjYfee eeeffeeddAdBdZkdGee_eeIeeZefdfe_e_eeeeZefdfee,eeZefdfe_edD dEdZldS)H) AnyDictListOptionalSetTupleUnionTypeCallable) QuantTypeN) GraphModule)GraphNodeArgument)"activation_is_statically_quantizedweight_is_quantizedget_qparam_dict _parent_nameget_swapped_custom_module_class) QConfigAnyqconfig_equals)QConfigMapping)_generate_node_name_to_qconfig)_compare_prepare_convert_qconfig_mappings_update_qconfig_for_fusion&_is_qconfig_supported_by_dtype_configs_update_qconfig_for_qat)-get_root_module_to_quantized_reference_moduleget_pattern_to_dtype_configsget_fused_module_classesget_qat_module_classes) BackendConfigget_native_backend_config)_is_activation_post_process)_is_observed_module_is_observed_standalone_module)update_obs_for_equalizationconvert_eq_obs)type_before_parametrizations) _get_module_is_custom_module_lstm_is_custom_module_mhaassert_and_get_unique_deviceget_custom_module_class_keyscreate_getattr_from_valuecollect_producer_nodes graph_module_from_producer_nodesnode_arg_is_weight)is_per_channelto_underlying_dtype)_remove_qconfig) DeQuantStub)ConvertCustomConfigPrepareCustomConfig)lower_to_fbgemm)quantized_decomposed_libconvertconvert_custom_moduleconvert_standalone_moduleconvert_weighted_module)modelgraphnodemodulesnode_name_to_scopenode_name_to_qconfigreturnc& s\|dk s tt|jtstt||\}}||j}tfddt|jt|j D} | slt |s| | | |jd| |W5QRXdS|j} d} t|dr|j} | tjtjtjfkr\| s\d} d} |\}}t|jr@t|j}tjjjj} tjjjj}|j}|j }t!| }||||||d}nLtjjj"j} tjjj#j}t$|}t|}|j}|j }t!| }|||||d }| ||jd}|g}|%D]P\}}|d krt|t$tfst&||||||}|'|n |'|q|(| | t)|i}|g|d d}|*|t)|i}| || |W5QRXn| rDd} tjjj"j+} t!| }|tj,tj-fkstd |j}|j }t.|d tj/}t.|dt0tj1j2}||||d}t3|} | ||jd}|jdg}!|%D]\}}"|!'|"q|(d| t)|!i}#|(dt4j5|#dfi}$|(dt4j5|#d fi}%|d}|d}|d} |$|%||| d }W5QRX| ||jd}|g}|%D].\}}|d kr|}|'|n |'|q|(| | t)|i}|g|d d}tjjj#j+}|*|t)|i}| || |W5QRXn| tj6krXt7ddS)av Replace activation_post_process module call node with quantize and dequantize node working with decomposed Tensor Before: ... -> observer_0(x) -> ... After: ... -> torch.ops.quantized_decomposed.quantize_per_tensor(x, ...) -> torch.ops.quantized_decomposed.dequantize_per_tensor() -> ... or quantize_per_channel and dequantize_per_channel Nc3s|]}t|VqdSN_has_none_qconfig.0nrEa/var/www/html/Darija-Ai-Train/env/lib/python3.8/site-packages/torch/ao/quantization/fx/convert.py rszM_replace_observer_with_quantize_dequantize_node_decomposed..rF is_dynamic call_function)_scale_ _zero_point__axis_ _quant_min_ _quant_max__dtype_)rSrTrVrWrXrSrTrzVonly uint8 and int8 are supported in reference flow for dynamic quantization right nowqschemeeps)rVrWZ_eps_rXrVrWrXz,decomposed to float16 op not implemented yet)8AssertionError isinstancetargetstr_get_module_path_and_prefixalllistargsuserskeys_is_conversion_supportedinserting_beforereplace_all_uses_with erase_nodedtypehasattrrQtorchquint8qint8qint32calculate_qparamsr4rZintch_axisopsquantized_decomposedquantize_per_channeldefaultZdequantize_per_channel quant_min quant_maxr5quantize_per_tensorZdequantize_per_tensorfloatitemsr0append create_nodetuplerRtensorZuint8Zint8getattrper_tensor_affineZfinfoZfloat32r[_QSCHEME_TO_CHOOSE_QPARAMS_OPoperatorgetitemfloat16NotImplementedError)&r@rArBrCrDrE module_pathprefixactivation_post_processskip_replacementrjrQ node_type quantize_opscale zero_pointrrZ dequantize_oprwrxZdtype_qparams input_nodequantize_op_inputskey value_or_node qparam_nodequantized_nodeZ dq_inputsdequantized_noderZr[Zchoose_qparams_opZchoose_qparams_op_inputsvalueZchoose_qparams_nodeZ scale_nodeZzero_point_noderNrMrO:_replace_observer_with_quantize_dequantize_node_decomposedZs                              rc s&|dk s tt|jtstt||\}}||j}tfddt|jt|j D} | slt |s| | | |jd| |W5QRXdS|j} d} t|dr|j} | tjtjtjfkr| sd} d} |\}}t|jrt|j}|||| d}tj} n"t|}t|}||| d }tj} | ||jd}|g}|D]@\}}|d krt||||||}||n ||q\| | | t!|i}|j"d |fd }| || |W5QRXn>| rd} tj#} tj$j%j&d k}| |d}| |l|jd}|g}|D]\}}||q0| | | t!|i}|j"d |fd }| || |W5QRXn| tj'kr"d} d} d| i}| |l|jd}|g}|D]\}}||q| | | t!|i}|j"d |fd }| || |W5QRXdS)z Replace activation_post_process module call node with quantize and dequantize node Before: ... -> observer_0(x) -> ... After: ... -> torch.quantize_per_tensor(x, ...) -> x.dequantize() -> ... Nc3s|]}t|VqdSrGrHrJrMrNrOrPOszB_replace_observer_with_quantize_dequantize_node..rFrQrR)rSrTrUrX)rSrTrXrY dequantize)rc)Zfbgemmx86)rXZ_reduce_range_ call_methodtorX)(r\r]r^r_r`rarbrcrdrerfrgrhrirjrkrQrlrmrnrorpr4rZrqrrrurzryr{r0r|r}r~rZquantize_per_tensor_dynamicbackendsZ quantizedZenginer)r@rArBrCrDrErrrrrjrQrrrrrrrrrrrrrrZ reduce_rangerrNrMrO/_replace_observer_with_quantize_dequantize_node:s                         r)rBrArFcCsD|jd}t|ts"td|||||t||dS)Nrz@Expecting the for call custom module node to be a Node, but got )rcr]rr\rhri_insert_dequantize_node)rBrAZcall_custom_module_noderNrNrO6_replace_observer_or_dequant_stub_with_dequantize_nodes    r)rrFcCsB|j}d}t|dr|j}|tjtjtjfkr4| p@|p@|tjkS)NFrQ)rjrkrQrlrmrnror)rrjrQrNrNrOrfs rf)rBrErFcCs"t|to |j|ko ||jdkS)z] Check if a node has a qconfig of None, i.e. user requested to not quantize the node N)r]rname)rBrErNrNrOrIsrI)observedbackend_configrFcCsZ|jjD]L}|jdkrq|jD]4}|rt||rt|}|dkrBqt||}|qqdS)z Extract the subgraph that produces the weight for dynamic quant or weight only quant node and run the subgraph to observe the weight. Note that the observers of dynamic quant or weight only quant ops are run during the convert step. rRN)rAnodesoprcr3r1r2)rrrBZnode_argZweight_observer_nodesZweight_observer_modulerNrNrO_run_weight_observerss   r)argrBrArFcCst|tr6|jdkr6|jdkr6|jd}|||n`t|ttfr\|D]}t|||qHn:t|t r| D]}t|||qnnt dt |dS)z If the arg is a dequantize Node, or a list/tuple/dict of dequantize Node, we'll recursively remove the dequantize Node rrrz6Unsupported node type in recursive remove dequantize: N)r]rrr^rcreplace_input_withrbr~"_maybe_recursive_remove_dequantizedictvalueswarningswarntype)rrBrA quantize_nodeZ arg_elementrNrNrOrs    r)obs_noderDrErFc Cs|jd}t|ts"td||j|kr:||jdknd}|rt|j}|rX|dnd}d}|D]&}|jdkrd|jt j j j krd|}qqd|r|}d} n|}d} |r|j|kr||j\} } nd} | | fS)aj Given and observer node, get the `Scope` or the fully qualified name for the submodule containing the observed node, also return a prefix of "_input" when the observed node is an input of a F.linear op, and not the output of another quantized op. TODO: this logic is hacky, we should think about how to remove it or make it more general rz.Expecting observed node to be a Node, but got NrR_input) rcr]rr\rrbrdrr^rlnnZ functionalZlinear) rrDrE observed_nodeZis_input_observer_onlyrdZfirst_linear_use_or_first_useZ linear_noderLrr_rNrNrOr`s2   r`c CsL||8|d|f}t|jD]}||k r$|||q$W5QRXdS)z3 Inserts dequantize node for `node` in `graph` rN)Zinserting_afterrrrdr)rBrAZdequantize_nodeZ user_noderNrNrOr-s  r)rBrCrFcCs<|jD],}|jdkr |t|j}t|r |Sq dS)z\ If the node is observed, return the observer instance. Otherwise, return None. call_moduleN)rdrerr_r^r%)rBrCZmaybe_obs_node maybe_obsrNrNrO_maybe_get_observer_for_node8s   r)rBrCr@ is_referencerrFcCs|rtjjjj}n tjjjj}|t|j}|jdj }t |j }t t |D]X} | |krN|| } | jdkrN| jdkrN| j d} || | t | jdkrN|j| qN|jdj} t | dkr| ddkstdt||j|||d} t|j\}}t|||| | |t|j<dS)a Converts a observed standalone module to a quantized standalone module by calling the fx convert api, currently using the same `is_reference` flag as parent, but we may changing this behavior in the future (e.g. separating quantization and lowering for standalone module as well) Args: - node: The call_module node of the observed standalone module - modules: named_module of original model - model: original model - is_reference: a flag from parent provided by user to decide if we want to produce a reference model or a fbgemm/qnnpack model - backend_config: backend configuration of the target backend of quantization _observed_graph_module_attrsrrrzCurrently only quantized)rN)rlaoZ quantizationZ quantize_fxZconvert_to_reference_fxZ convert_fxr_r^metaZ&standalone_module_input_quantized_idxsrbrcrangelenrrrdrAriZ'standalone_module_output_quantized_idxsr\rrsetattr)rBrCr@rrZ convert_fnZobserved_standalone_moduleZsm_input_quantized_idxsrcidxrrZsm_output_quantized_idxsZquantized_standalone_module parent_namerrNrNrOr>Gs>      F)rBrCobserved_node_namesrEr is_decomposedrrFc Cs||t|j}|j}d} t|} t|| rV|j} |}t|j\} } t|| | ||j |k} |dksvt ||sv| szdSt |}| t |g}t||sdSt|}|sdSd}|}t|tjjjjr|}|d}d|i}t|tjjr4|}|}||j||jt|}t|}|||dnt|tjjtjjfr|jD]R}t||rR|drRt ||}|} | j!tj"kr| |t| ||<qRn^| dk}|r|} t#|}|r| $|| }|r|r|s| |j|t| t%|}| t&|d}|dk s:t'dt&||(||}|dk rZ||d<nt|j\} } t|| | |dS)a Convert a weighted module to reference quantized module in the model If the QConfig of a QAT module is not set, the module will still be converted to a float module. Args: - node: The call_module node of the observed standalone module - modules: named_module of original model - observed_node_names: names for the set of observed fx node, we can skip this conversion if the node is not observed Nrr) weight_ih weight_hhweightz3No reference quantized module class configured for ))r_r^qconfigr"r]Zweight_fake_quantZto_floatrrrrIr getrrrrlrrZ intrinsicZ _FusedModuleZ RNNCellBaserrrrupdateZLSTMZGRUZ_flat_weights_namesrk startswithrrjrnr.rrr*r\ from_float) rBrCrrErrrZoriginal_modulerZweight_post_processqat_module_classesrrZ is_observedZpattern_to_dtype_configsZ dtype_configsZis_weight_quantizedZ fused_moduleZ float_moduleZ wq_or_wq_dictZweight_post_process_ihZweight_post_process_hhZweight_qparams_ihZweight_qparams_hhZwnrZis_ptqZdeviceis_qat)root_module_to_quantized_reference_moduleZref_qmodule_clsZ ref_qmodulerNrNrOr?s             )rB prev_noderArFcCsZt|tstd||jdkrV|jdkrV|||jdt|jdkrV| |dS)z Given a custom module `node`, if the previous node is a dequantize, reroute the custom as follows: Before: quantize - dequantize - custom_module After: quantize - custom_module \ - dequantize zDExpecting the argument for custom module node to be a Node, but got rrrN) r]rr\rr^rrcrrdri)rBrrArNrNrO,_remove_previous_dequantize_in_custom_module s r)rBrArCcustom_module_class_mapping(statically_quantized_custom_module_nodesrFcCs|t|j}t||}|j}t|r||t||rt|jdkrlt |jdt rlt|jddkspt |j\}\} } t |t st t | t st t | t st t |||t || |t || |nt||rNt|jdkst |j\} } } t | t st t | t st t | t s(t t || |t || |t || |nD|jd}t |t sht t |||t||}|dk st ||_t|||}||}t|j\}}t||||dS)a Converts an observed custom module to a quantized custom module based on `custom_module_class_mapping` For static quantization, we'll also remove the previous `dequantize` node and attach the observer node for output to the module, the observer for the node will be converted to a dequantize node instead of quantize-dequantize pairs later in the graph. In the end we would have a quantized custom module that has the same interface as a default quantized module in nn.quantized namespace, i.e. quantized input and quantized output. Args: - node: The call_module node of the observed standalone module - graph: The graph containing the node - modules: named_module of original model - custom_module_class_mapping: mapping from observed custom module class to quantized custom module class, used to swap custom modules - statically_quantized_custom_module_nodes: we'll add the custom module node if we find it is statically quantized, this will be used later when converting observers to quant/dequant node pairs, if the observed node is a statically quantized custom module nodes, we'll convert the observer to a dequantize node, this is to keep the interface the same as the default quantized module. TODO: maybe we want to redesign this part to align with reference model design as well, but there has been some discussions around the interface, so we can do it later. rrrN)r_r^rrraddr,rrcr]r~r\rrr-rrZ from_observedrr)rBrArCrrZobserved_custom_modulerrinputsZhidden0Zhidden1queryrrrrZquantized_custom_module_classZquantized_custom_modulerrrNrNrOr=sV              T) r@rconvert_custom_configis_standalone_module_remove_qconfig_flagqconfig_mappingrrrFc& Cs|dkrt}t|tr,tdt|}t|trRtd|rNt|nd}t|}|dksrt|tsrt t|trtdt |}|dkrt }t |st d|j d}|j} |j} |j} |j} t|jdd} |r|j}t| }|jrt||t||t||t|||j|| }| D]`\}}||ksXt d |d ||dk r6t|||s6t d |d |d ||q6|} t|j}|j}|jdk rt || }t!|| |t"||g}|jj#D]}|j$dkr|%|j&qd}| j'}| j(}t)|}t*|+}t,|}t-|}t.}t/|jj#D]}|j$dkr|} |d7}| |krPt0||jqJ|j$dkr t1|dkrqJ|}!|j2d}"t|"t/t*fr|D]}#t3|"|#|!|jqn>t|"t4tfrd|krt3|"|!|jntdt5|"qJ|j$dkrJt6|| }$|$dk sDt t7|$r|j2d}%|%|krpt8||jn0|rt9||j|| | | nt:||j|| | | nt|$t;rt8||jnt<|$rt=|| |||nxt>|$t.|?|?|kr0t>|$|krt>|$d|krqJt@|| | | |||n t>|$|krJtA||j| ||qJ|jBtC||j}|s|tD|| | }|rtE||F|j Gdd|S)a We will convert an observed model (a module with observer calls) to a reference quantized model, the rule is simple: 1. for each observer module call in the graph, we'll convert it to calls to quantize and dequantize functions based on the observer instance 2. for weighted operations like linear/conv, we need to convert them to reference quantized module, this requires us to know whether the dtype configured for the weight is supported in the backend, this is done in prepare step and the result is stored in observed_node_names, we can decide whether we need to swap the module based on this set Args: * `is_standalone_module`: when this flag is True, it means we are quantizing a submodule that is not inlined in parent module, and will be quantized separately as one unit. * `is_decomposed`: a boolean flag to indicate whether we want to use the quantize operator for decomposed quantized tensor (torch.ops.quantized_decomposed.quantize_per_tensor) or default/standalone quantized tensor (torch.quantize_per_tensor) Returns: a quantized standalone module, whether input/output is quantized is specified by prepare_custom_config, with input_quantized_idxs, output_quantized_idxs, please see docs for :func:`~torch.ao.quantization.prepare_fx` for details NzPassing a convert_custom_config_dict to convert is deprecated and will not be supported in a future version. Please pass in a ConvertCustomConfig instead.zPassing a QConfig dictionary to convert is deprecated and will not be supported in a future version. Please pass in a QConfigMapping instead.zPassing a backend_config_dict to prepare is deprecated and will not be supported in a future version. Please pass in a BackendConfig instead.z-incoming model must be produced by prepare_fxrF)Zremove_duplicatez Expected key z in convert node_name_to_qconfigz Expected k zD to have the same value in prepare and convert QConfigMappings, but z was updated to placeholderrroutputz1Unsupported node type for output_quantized_idxs: r)Hr8r]rrr from_dictrcopydeepcopyr\r#r$r&rrDprepare_custom_configrrErZ named_modulesrrrrrrrAr{rr/Zobserved_to_quantized_mappingZ!equalization_node_name_to_qconfigr(r)rrrr|rZinput_quantized_indexesZoutput_quantized_indexesrr~rer"r!setrbrrrcrrrr+r%rrrr7r'r>r*unionr?r=Zeliminate_dead_coder r:r6Zdelete_all_unused_submodulespop)&r@rrrrrrrZobserved_graph_module_attrsrDrrrErCZprepare_qconfig_mappingZ modules_copyZconvert_node_name_to_qconfigkvZcustom_module_classesrZweight_eq_obs_dictZ graph_inputsrBZplaceholder_node_seen_cntZinput_quantized_idxsZoutput_quantized_idxsrZroot_module_classesrZfused_module_classesrZcur_placeholder_node_idxZ return_noderrmodrrNrNrOr<ps<#                                    )FF)FNFTNNF)mtypingrrrrrrrr r Z torch.ao.quantization.quant_typer rlrrZtorch.fxr Ztorch.fx.graphr rrutilsrrrrrrrrrrZqconfig_mapping_utilsrrrrrZ*torch.ao.quantization.backend_config.utilsrr r!r"Z$torch.ao.quantization.backend_configr#r$Ztorch.ao.quantization.observerr%Z graph_moduler&r'Z _equalizer(r)Ztorch.nn.utils.parametrizer*r+r,r-r.r/r0r1r2r3Ztorch.ao.quantization.utilsr4r5Ztorch.ao.quantization.quantizer6Ztorch.ao.quantization.stubsr7Z custom_configr8r9r:Z _decomposedr;r__all__rrsrtZchoose_qparamsrZper_tensor_symmetricZchoose_qparams_symmetricrrModuler_rrrrboolrfrIrrr`rrZfxr>r?rr=r<rNrNrNrOs,     ,         b    1   D   T