U dR@s,ddlmZmZddlZGdddeZGdddeZGdddeZGd d d eZ Gd d d eZ Gd ddeZ GdddeZ GdddeZ GdddeZGdddeZGdddeZGdddeZGdddeZGdddeZGdd d eZGd!d"d"eZGd#d$d$eZdS)%)coreutilsNc@seZdZdZdZdS)RegularizationByZafter_optimizerZon_lossN)__name__ __module__ __qualname__ZAFTER_OPTIMIZERZON_LOSSrr=/tmp/pip-unpacked-wheel-ua33x9lu/caffe2/python/regularizer.pyr src@sBeZdZddZdddZdddZdd Zd d Zdd dZdS) RegularizercCs d|_dS)Ng& .>)kEpsilon)selfrrr __init__szRegularizer.__init__NcCsvt|tjsttt}||ks>td|j ||d|}t ||sbtd|j |t ||||||S)Nz>Regularizer of type {} is called with invalid by={}, not in {}Z_run_z5Regularizer of type {} does not implement function {}) isinstancerZ BlobReferenceAssertionErrorrZEnumClassKeyValsrvaluesformat __class__hasattrgetattr)r netparam_init_netparamgradZbyZby_enumZrun_funcrrr __call__s( zRegularizer.__call__cCsdSNrr rrrrrrr _run_on_loss'szRegularizer._run_on_losscCsdSrrrrrr _run_after_optimizer*sz Regularizer._run_after_optimizercCsL|||g|dg}||g|dg}|j|g|dgdd}|S)N param_mul param_reducedgrouped_feature_weight_vec?exponent)MulNextScopedBlobReduceFrontSumPow)r rrrrr rrr _feature_grouping-s  zRegularizer._feature_groupingFc Cst|dk r|s|r||jn|}|dk r8|s.|r8||jn|}t|tjrV||j|jgn|g} |j| |g||ddS)N)minmax)r rr GradientSliceindicesrZ EnsureClipped) r rrrr)r* open_range left_open right_openZ input_blobsrrr _ensure_clipped=s*    zRegularizer._ensure_clipped)NN)N)NNNFFF) rrrr rrrr(r0rrrr r s  r cs&eZdZfddZdddZZS)L1Normcs(tt||dkstd||_dSNr6factor ahead of regularization should be 0 or positive)superr1r r reg_lambdar r5rrr r [szL1Norm.__init__NcCs<||d}|j|g|gdd|j|g|g|jd|S)NZ_l1_regularizationpZscaler%LpNormScaler5r rrrr output_blobrrr raszL1Norm._run_on_loss)Nrrrr r __classcell__rrr7r r1Zs r1cs(eZdZdfdd ZdddZZS) r=r!cs>tt||dkstd|dks.td||_||_dS)a  reg_lambda: parameter to scale regularization by p_value: determines what type of Lp norm to calculate. If p > 0, we will calculate Lp norm with the formula: pow( sum_i { pow(theda_i, p) } , 1/p) r7factor ahead of regularization should be greater than 0z'p_value factor should be greater than 0N)r4r=r rp_valuer5)r r5rDr7rr r hs zLpNorm.__init__Nc Cs||d}|||}|j|g|dg|jd}||g|dg}|j|g|dgd|jd} |j| g|g|jd|S)N_dense_feature_regularization lp_vec_raisedr" lp_vec_summedZlp_vecr8r;)r%r(r'rDr&r>r5) r rrrrr@r rFrGZlp_normrrr rws$    zLpNorm._run_on_loss)r!)NrArrr7r r=gsr=cs(eZdZdfdd Zd ddZZS) L0ApproxNorm{Gz?rcsXtt||dkstd|dks.td|dks>td||_||_t||_dS)a9 reg_lambda: parameter to scale regularization by alpha: hyper parameter to tune that is only used in the calculation of approximate L0 norm budget: desired number of features. If the number of features is greater than the budget amount, then the least important features will be penalized. If there are fewer features than the desired budget, no penalization will be applied. Optional parameter, if 0, then no budget is used rrCz4alpha factor must be a positive value greater than 0z0budget factor must be greater than or equal to 0N)r4rHr rr5alphafloatbudget)r r5rJrLr7rr r s zL0ApproxNorm.__init__NcCs||d}|||}||g|dg}|j|g|dg|jd}||g|dg} |j| g|dgd|jd} |jr|jgd dg|jd } | | | g|d g} | | g|d g} |j| g|g|j dn|j| g|g|j d|S) NrEl0_absl0_min)r* l0_summedl0_normr8r;rL)shapevalueZ l0_budgetrelu_l0_sub_budget) r%r(AbsCliprJr&r>rL ConstantFillSubZRelur5)r rrrrr@r rMrNrOrPZ budget_blobZ l0_sub_budgetrSrrr rs.    zL0ApproxNorm._run_on_loss)rIr)NrArrr7r rHsrHcs*eZdZdZfddZdddZZS) L1NormTrimmedzV The Trimmed Lasso: Sparsity and Robustness. https://arxiv.org/abs/1708.04527 csPtt||dkstdt|ts0td|dks@td||_||_dS)Nrr3z6k should be an interger as expected #. after selectionr8zk should be larger than 1)r4rXr rrintr5k)r r5rZr7rr r s zL1NormTrimmed.__init__Nc Cs||d}||g|dg}|j|g|dgdd}|j|g|d|d|dg|jd \}} } |j|g|d gdd} ||| g|g|j|g|g|jd |S) NZ_l1_trimmed_regularizationabssum_absFZaveragetopkidflat_idrZtopk_sumr;)r%rT SumElementsTopKrZrWr>r5) r rrrrr@r[r\r^_rbrrr rs2zL1NormTrimmed._run_on_loss)Nrrr__doc__r rrBrrr7r rXs rXcs&eZdZfddZdddZZS)L2Normcs(tt||dkstd||_dSr2)r4rhr rr5r6r7rr r szL2Norm.__init__NcCs<||d}|j|g|gdd|j|g|g|jd|S)NZ_l2_regularizationr9r;r<r?rrr rszL2Norm._run_on_loss)NrArrr7r rhs rhcs&eZdZfddZdddZZS) ElasticNetcstt|||_||_dSr)r4rjr l1l2)r rkrlr7rr r szElasticNet.__init__NcCs||d}||d}||d}|j|g|gdd|j|g|gdd|j|g|g|jd|j|g|g|jd|||g|g|S)NZ_elastic_net_regularization_l2_blob_l1_blobrir9r8r;)r%r=r>rlrkAdd)r rrrrr@l2_blobl1_blobrrr rszElasticNet._run_on_loss)NrArrr7r rjs rjcs&eZdZfddZdddZZS)ElasticNetL1NormTrimmedcs$tt|||_||_||_dSr)r4rrr rkrlrZ)r rkrlrZr7rr r sz ElasticNetL1NormTrimmed.__init__Nc Cs||d}||d}|j|g|gdd|j|g|g|jd||d}||g|dg}|j|g|dgd d } |j|g|d |d |d g|jd\} } } |j| g|dgd d } || | g|g|j|g|g|j d| ||g|g|S)NZ&_elastic_net_l1_trimmed_regularizationrmrir9r;rnr[r\Fr]r^r_r`rarb) r%r=r>rlrTrcrdrZrWrkro) r rrrrr@rprqr[r\r^rerbrrr r s2z$ElasticNetL1NormTrimmed._run_on_loss)NrArrr7r rrs rrcs&eZdZdfdd ZddZZS)MaxNorm?Ncstt|||_||_dSr)r4rsr normdtype)r rurvr7rr r szMaxNorm.__init__cCsv|jdkstdt|tjrj|jrL|jdkrL|j||jg|gd|jdqr|j||jg|gd|jdnt ddS)Nrnorm should be bigger than 0.Zfp16TZ use_max_normruz-MaxNorm is not supported for dense parameters) rurrrr+rvZFloat16SparseNormalizer,SparseNormalizeNotImplementedErrorrrrr r!s  zMaxNorm._run_after_optimizer)rtNrrrr rrBrrr7r rssrscs&eZdZdfdd ZddZZS) ConstantNormrtcstt|||_dSr)r4r|r ru)r rur7rr r 7szConstantNorm.__init__cCsH|jdkstdt|tjr<|j||jg|gd|jdntddS)NrrwFrxz2ConstantNorm is not supported for dense parameters)rurrrr+ryr,rzrrrr r;s z!ConstantNorm._run_after_optimizer)rtr{rrr7r r|6sr|cs$eZdZfddZddZZS) SparseLpNormcs>tt||dkstd|dks.td||_||_dS)N)rt@zBSparse Lp regularization only implemented for p = 1.0 and p = 2.0.rz8factor ahead of regularization should be greater than 0.)r4r}r rr:r5)r r:r5r7rr r Ks zSparseLpNorm.__init__cCs8t|tjr,|j||jg|g|j|jdntddS)Nr:r5z2SparseLpNorm is not supported for dense parameters)rrr+ZSparseLpRegularizerr,r:r5rzrrrr rRs z!SparseLpNorm._run_after_optimizerr{rrr7r r}Js r}cseZdZfddZZS) SparseL1Normcstt|jd|ddS)Nrtr)r4rr r6r7rr r _szSparseL1Norm.__init__rrrr rBrrr7r r^srcseZdZfddZZS) SparseL2Normcstt|jd|ddS)Nr~r)r4rr r6r7rr r dszSparseL2Norm.__init__rrrr7r rcsrcs4eZdZdZd fdd Zd ddZdd ZZS) LogBarrierzr Wright, S., & Nocedal, J. (1999). Numerical optimization. Springer Science, 35(67-68), 7. Chapter 19 invNcs>tt||dkstd||_||_|p6ddd|_dS)z discount is a positive weight that is decreasing, and here it is implemented similar to the learning rate. It is specified by a learning rate policy and corresponding options rr3rt)gammapowerN)r4rr rr5discount_policydiscount_options)r r5rrr7rr r ns zLogBarrier.__init__c Cst||}||d}|j|g|gf|j |jd|j||d}|j|g|g|jd||d}| |g|g||d} | |g| g||d} |j | |g| gdd | S) NZ_log_barrier_discount)Zbase_lrpolicyZ_non_neg)r)_logZ_log_sumZ _log_barrierr8 broadcast) rZBuildUniqueMutexIterr%Z LearningRater5rrrUr Logrcr$) r rrrr iterationZdiscountZ param_non_negZ param_logZ param_log_sumr@rrr rzs(  zLogBarrier._run_on_losscCs|j|||ddddS)NrT)r)r-)r0rrrr rszLogBarrier._run_after_optimizer)rN)N)rrrrgr rrrBrrr7r rhs rcs*eZdZdZdfdd ZddZZS) BoundedGradientProjectionzr Wright, S., & Nocedal, J. (1999). Numerical optimization. Springer Science, 35(67-68), 7. Chapter 16 NFcstt||dk rt|nd}|dk r2t|nd}|dk rFt|n|j}|dksdtdj|d|dks|dks||r~|nd||r|ndkstdj|||rdnd|rdnd |d ||_||_||_||_ ||_ dS) Nrz2Bounded Gradient Projection with invalid eps={eps})epsgzLBounded Gradient Projection with invalid {lp}ub={ub}, lb={lb}{rp}, eps={eps}([)])lbublprpr) r4rr rKr rrr.r/rr)r rrr.r/epsilonr7rr r s8  z"BoundedGradientProjection.__init__c Cs$|j||||j|j|j|jddS)N)r)r*r.r/)r0rrr.r/rrrr rsz.BoundedGradientProjection._run_after_optimizer)NNFFN)rrrrgr rrBrrr7r rs!rcs,eZdZdZdfdd Zd ddZZS) GroupL1Norma Scardapane, Simone, et al. "Group sparse regularization for deep neural networks." Neurocomputing 241 (2017): 81-89. This regularizer computes l1 norm of a weight matrix based on groups. There are essentially three stages in the computation: 1. Compute the l2 norm on all the members of each group 2. Scale each l2 norm by the size of each group 3. Compute the l1 norm of the scaled l2 norms rcsFtt||dkstdt|ts0td||_||_||_dS)a Args: reg_lambda: The weight of the regularization term. groups: A list of integers describing the size of each group. The length of the list is the number of groups. Optional Args: stabilizing_val: The computation of GroupL1Norm involves the Sqrt operator. When values are small, its gradient can be numerically unstable and causing gradient explosion. Adding this term to stabilize gradient calculation. Recommended value of this term is 1e-8, but it depends on the specific scenarios. If the implementation of the gradient operator of Sqrt has taken into stability into consideration, this term won't be necessary. rz-regularization weight should be 0 or positivezgroups needs to be a listN) r4rr rrlistr5groupsstabilizing_val)r r5rrr7rr r szGroupL1Norm.__init__Nc Cs||}|j|dgdd}|||jgdt|jg|jdg}|jrl|j||jgd|jdg|gdd| |}| ||j gt|jgt |j|jdgdg} |j| dgdd } | S) a Args: param: The input blob to regularize. It should be a weight matrix blob with shape (output_dim, input_dim). input_dim should be equal to the sum of self.groups. Returns: group_l1_norm: The output blob after applying regularization. These are the steps of computation: 1. square all elements 2. sum by row 3. lengthssum by group 4. square_root all elements 5. normalize each group based on group size 6. compute l1 norm of each group 7. scale the result with the regularization lambda r)ZaxesZkeepdimsr8)rQr)rRrZnormalized_l2_norm_scaledZ group_l1_nromr9)ZSqrZ ReduceSumZ LengthsSumZGivenTensorIntFilllenrrrorVZSqrtr$ZGivenTensorFillnpsqrtr5r=) r rrrrZsquaredZ reduced_sumZ lengths_sumrZ l2_scaledZ group_l1_normrrr rs>      zGroupL1Norm._run_on_loss)r)Nrfrrr7r rs r)Z caffe2.pythonrrZnumpyrobjectrr r1r=rHrXrhrjrrrsr|r}rrrrrrrrr s$L ,7.3