C++博客-杰-随笔分类-Optimizationhttp://www.cppblog.com/guijie/category/20090.html杰哥好,哈哈!zh-cnTue, 09 Apr 2019 13:20:33 GMTTue, 09 Apr 2019 13:20:33 GMT60How to solve AX + XB = C for X using matlab?http://www.cppblog.com/guijie/archive/2015/07/06/211161.html杰哥杰哥Mon, 06 Jul 2015 07:28:00 GMThttp://www.cppblog.com/guijie/archive/2015/07/06/211161.htmlhttp://www.cppblog.com/guijie/comments/211161.htmlhttp://www.cppblog.com/guijie/archive/2015/07/06/211161.html#Feedback0http://www.cppblog.com/guijie/comments/commentRss/211161.htmlhttp://www.cppblog.com/guijie/services/trackbacks/211161.htmlX = sylvester(A,B,C)
http://cn.mathworks.com/help/matlab/ref/sylvester.html

杰哥 2015-07-06 15:28 发表评论
]]>Alternating optimizationhttp://www.cppblog.com/guijie/archive/2015/05/24/210729.html杰哥杰哥Sun, 24 May 2015 04:58:00 GMThttp://www.cppblog.com/guijie/archive/2015/05/24/210729.htmlhttp://www.cppblog.com/guijie/comments/210729.htmlhttp://www.cppblog.com/guijie/archive/2015/05/24/210729.html#Feedback0http://www.cppblog.com/guijie/comments/commentRss/210729.htmlhttp://www.cppblog.com/guijie/services/trackbacks/210729.html      我个人理解，这几个概念都是等价的。

‘alternating optimization’ or ‘alternative optimization’?

Sue (UTS) comment: ‘Alternating’ means you use this optimization with another optimization, one after the other. ‘Alternative’ means you use this optimization instead of any other.

我的GSM-PAF最后用的‘alternating optimization’



杰哥 2015-05-24 12:58 发表评论
]]>完全掌握 最大似然估计http://www.cppblog.com/guijie/archive/2013/12/05/204609.html杰哥杰哥Thu, 05 Dec 2013 11:21:00 GMThttp://www.cppblog.com/guijie/archive/2013/12/05/204609.htmlhttp://www.cppblog.com/guijie/comments/204609.htmlhttp://www.cppblog.com/guijie/archive/2013/12/05/204609.html#Feedback0http://www.cppblog.com/guijie/comments/commentRss/204609.htmlhttp://www.cppblog.com/guijie/services/trackbacks/204609.html
这是属于概率论与数理统计中参数估计的内容，见教材第七章P168；模式识别笔记的Section 3.11.1(Section 3.11到Section 3.11.1的内容应该记住)
总结：最大似然函数估计法，首先是假设所得的样本服从某一分布，目标是估计出这个分布中的参数，方法是得到这一组样本的概率最大时就对应了该模型的参数值，写出似然函数，再求对数（得到对数似然），再求对数似然函数的平均（对数平均似然），再对其求导，得出参数值。目前我理解的需要求对数的原因是，通常概率是小数，连乘之后会非常小，对计算机而言，容易造成浮点数下溢，所以用了取对数。
Zhengxia也提到过似然(likelihood)就是概率，观测到的概率。
https://en.wikipedia.org/wiki/Likelihood_function


杰哥 2013-12-05 19:21 发表评论
]]>How to use matlab solve optimization quadratic?http://www.cppblog.com/guijie/archive/2012/11/21/195475.html杰哥杰哥Wed, 21 Nov 2012 10:31:00 GMThttp://www.cppblog.com/guijie/archive/2012/11/21/195475.htmlhttp://www.cppblog.com/guijie/comments/195475.htmlhttp://www.cppblog.com/guijie/archive/2012/11/21/195475.html#Feedback0http://www.cppblog.com/guijie/comments/commentRss/195475.htmlhttp://www.cppblog.com/guijie/services/trackbacks/195475.html

杰哥 2012-11-21 18:31 发表评论
]]>Taylor series in several variableshttp://www.cppblog.com/guijie/archive/2012/10/31/194113.html杰哥杰哥Wed, 31 Oct 2012 02:48:00 GMThttp://www.cppblog.com/guijie/archive/2012/10/31/194113.htmlhttp://www.cppblog.com/guijie/comments/194113.htmlhttp://www.cppblog.com/guijie/archive/2012/10/31/194113.html#Feedback0http://www.cppblog.com/guijie/comments/commentRss/194113.htmlhttp://www.cppblog.com/guijie/services/trackbacks/194113.htmlhttp://en.wikipedia.org/wiki/Taylor_series

Taylor series in several variables

The Taylor series may also be generalized to functions of more than one variable with

T(x_1,\dots,x_d) = \sum_{n_1=0}^\infty \sum_{n_2=0}^\infty \cdots \sum_{n_d = 0}^\infty \frac{(x_1-a_1)^{n_1}\cdots (x_d-a_d)^{n_d}}{n_1!\cdots n_d!}\,\left(\frac{\partial^{n_1 + \cdots + n_d}f}{\partial x_1^{n_1}\cdots \partial x_d^{n_d}}\right)(a_1,\dots,a_d).\!

For example, for a function that depends on two variables, x and y, the Taylor series to second order about the point (a, b) is:

\begin{align} f(x,y) & \approx f(a,b) +(x-a)\, f_x(a,b) +(y-b)\, f_y(a,b) \\ & {}\quad + \frac{1}{2!}\left[ (x-a)^2\,f_{xx}(a,b) + 2(x-a)(y-b)\,f_{xy}(a,b) +(y-b)^2\, f_{yy}(a,b) \right], \end{align}

where the subscripts denote the respective partial derivatives.

A second-order Taylor series expansion of a scalar-valued function of more than one variable can be written compactly as

T(\mathbf{x}) = f(\mathbf{a}) + \mathrm{D} f(\mathbf{a})^T (\mathbf{x} - \mathbf{a}) + \frac{1}{2!} (\mathbf{x} - \mathbf{a})^T \,\{\mathrm{D}^2 f(\mathbf{a})\}\,(\mathbf{x} - \mathbf{a}) + \cdots\! \,,

where D f(\mathbf{a})\! is the gradient of \,f evaluated at \mathbf{x} = \mathbf{a} and D^2 f(\mathbf{a})\! is the Hessian matrix. Applying the multi-index notation the Taylor series for several variables becomes

T(\mathbf{x}) = \sum_{|\alpha| \ge 0}^{}\frac{(\mathbf{x}-\mathbf{a})^{\alpha}}{\alpha !}\,({\mathrm{\partial}^{\alpha}}\,f)(\mathbf{a})\,,

which is to be understood as a still more abbreviated multi-index version of the first equation of this paragraph, again in full analogy to the single variable case.

[edit]Example

Second-order Taylor series approximation (in gray) of a function f(x,y) = e^x\log{(1+y)}around origin.

Compute a second-order Taylor series expansion around point (a,b) = (0,0) of a function

f(x,y)=e^x\log(1+y).\,

Firstly, we compute all partial derivatives we need

f_x(a,b)=e^x\log(1+y)\bigg|_{(x,y)=(0,0)}=0\,,
f_y(a,b)=\frac{e^x}{1+y}\bigg|_{(x,y)=(0,0)}=1\,,
f_{xx}(a,b)=e^x\log(1+y)\bigg|_{(x,y)=(0,0)}=0\,,
f_{yy}(a,b)=-\frac{e^x}{(1+y)^2}\bigg|_{(x,y)=(0,0)}=-1\,,
f_{xy}(a,b)=f_{yx}(a,b)=\frac{e^x}{1+y}\bigg|_{(x,y)=(0,0)}=1.

The Taylor series is

\begin{align} T(x,y) = f(a,b) & +(x-a)\, f_x(a,b) +(y-b)\, f_y(a,b) \\ &+\frac{1}{2!}\left[ (x-a)^2\,f_{xx}(a,b) + 2(x-a)(y-b)\,f_{xy}(a,b) +(y-b)^2\, f_{yy}(a,b) \right]+ \cdots\,,\end{align}

which in this case becomes

\begin{align}T(x,y) &= 0 + 0(x-0) + 1(y-0) + \frac{1}{2}\Big[ 0(x-0)^2 + 2(x-0)(y-0) + (-1)(y-0)^2 \Big] + \cdots \\ &= y + xy - \frac{y^2}{2} + \cdots. \end{align}

Since log(1 + y) is analytic in |y| < 1, we have

e^x\log(1+y)= y + xy - \frac{y^2}{2} + \cdots

for |y| < 1.



杰哥 2012-10-31 10:48 发表评论
]]>Jensen's inequalityhttp://www.cppblog.com/guijie/archive/2012/10/30/194080.html杰哥杰哥Tue, 30 Oct 2012 04:04:00 GMThttp://www.cppblog.com/guijie/archive/2012/10/30/194080.htmlhttp://www.cppblog.com/guijie/comments/194080.htmlhttp://www.cppblog.com/guijie/archive/2012/10/30/194080.html#Feedback0http://www.cppblog.com/guijie/comments/commentRss/194080.htmlhttp://www.cppblog.com/guijie/services/trackbacks/194080.htmlhttp://en.wikipedia.org/wiki/Jensen's_inequality

If λ1 and λ2 are two arbitrary nonnegative real numbers such that λ1 + λ2 = 1 then convexity of \scriptstyle\varphi implies

\varphi(\lambda_1 x_1+\lambda_2 x_2)\leq \lambda_1\,\varphi(x_1)+\lambda_2\,\varphi(x_2)\text{ for any }x_1,\,x_2.  [这就是凸函数的定义]

This can be easily generalized: if λ1, λ2, ..., λn are nonnegative real numbers such that λ1 + ... + λn = 1, then

\varphi(\lambda_1 x_1+\lambda_2 x_2+\cdots+\lambda_n x_n)\leq \lambda_1\,\varphi(x_1)+\lambda_2\,\varphi(x_2)+\cdots+\lambda_n\,\varphi(x_n),

例如-log(x)是凸函数


杰哥 2012-10-30 12:04 发表评论
]]>Gradient Descent(梯度下降法)(两例对应两牛文均用该法求解目标函数)http://www.cppblog.com/guijie/archive/2012/10/19/193522.html杰哥杰哥Fri, 19 Oct 2012 05:33:00 GMThttp://www.cppblog.com/guijie/archive/2012/10/19/193522.htmlhttp://www.cppblog.com/guijie/comments/193522.htmlhttp://www.cppblog.com/guijie/archive/2012/10/19/193522.html#Feedback0http://www.cppblog.com/guijie/comments/commentRss/193522.htmlhttp://www.cppblog.com/guijie/services/trackbacks/193522.htmlhttp://en.wikipedia.org/wiki/Gradient_descent 
http://zh.wikipedia.org/wiki/%E6%9C%80%E9%80%9F%E4%B8%8B%E9%99%8D%E6%B3%95
 Gradient descent is based on the observation that if the multivariable function F(\mathbf{x}) is defined and differentiable in a neighborhood of a point \mathbf{a}, then F(\mathbf{x}) decreases fastest if one goes from \mathbf{a} in the direction of the negative gradient of F at \mathbf{a}, -\nabla F(\mathbf{a}) 
为啥步长要变化？Tianyi的解释很好：如果步长过大，可能使得函数值上升，故要减小步长 (下面这个图片是在纸上画好，然后scan的)。
Andrew NG的coursera课程Machine learning的II. Linear Regression with One Variable的Gradient descent Intuition中的解释很好，比如在下图在右侧的点，则梯度是正数， -\nabla F(\mathbf{a})是负数，即使当前的a减小
例1：Toward the Optimization of Normalized Graph Laplacian(TNN 2011)的Fig. 1. Normalized graph Laplacian learning algorithm是很好的梯度下降法的例子.只要看Fig1，其他不必看。Fig1陶Shuning老师课件 非线性优化第六页第四个ppt，对应教材P124，关键直线搜索策略，应用 非线性优化第四页第四个ppt，步长加倍或减倍。只要目标减少就到下一个搜索点，并且步长加倍；否则停留在原点，将步长减倍。
例2： Distance Metric Learning for Large Margin Nearest Neighbor Classification(JLMR),目标函数就是公式14，是矩阵M的二次型，展开后就会发现，关于M是线性的，故是凸的。对M求导的结果，附录公式18和19之间的公式中没有M

我自己额外的思考：如果是凸函数，对自变量求偏导为0，然后将自变量求出来不就行了嘛，为啥还要梯度下降？上述例二是不行的，因为对M求导后与M无关了。和tianyi讨论，正因为求导为0 没有解析解采用梯度下降，有解析解就结束了
http://blog.csdn.net/yudingjun0611/article/details/8147046

1. 梯度下降法

梯度下降法的原理可以参考：斯坦福机器学习第一讲。

我实验所用的数据是100个二维点。

如果梯度下降算法不能正常运行，考虑使用更小的步长(也就是学习率)，这里需要注意两点：

1）对于足够小的,  能保证在每一步都减小；
2）但是如果太小，梯度下降算法收敛的会很慢；

总结：
1）如果太小，就会收敛很慢；
2）如果太大，就不能保证每一次迭代都减小，也就不能保证收敛；
如何选择-经验的方法：
..., 0.001, 0.003, 0.01, 0.03, 0.1, 0.3, 1...
约3倍于前一个数。

matlab源码：

[cpp] view plaincopy
  1. function [theta0,theta1]=Gradient_descent(X,Y);  
  2. theta0=0;  
  3. theta1=0;  
  4. t0=0;  
  5. t1=0;  
  6. while(1)  
  7.     for i=1:1:100 %100个点  
  8.         t0=t0+(theta0+theta1*X(i,1)-Y(i,1))*1;  
  9.         t1=t1+(theta0+theta1*X(i,1)-Y(i,1))*X(i,1);  
  10.     end  
  11.     old_theta0=theta0;  
  12.     old_theta1=theta1;  
  13.     theta0=theta0-0.000001*t0 %0.000001表示学习率  
  14.     theta1=theta1-0.000001*t1  
  15.     t0=0;  
  16.     t1=0;  
  17.     if(sqrt((old_theta0-theta0)^2+(old_theta1-theta1)^2)<0.000001) % 这里是判断收敛的条件，当然可以有其他方法来做  
  18.         break;  
  19.     end  
  20. end  


2. 随机梯度下降法

随机梯度下降法适用于样本点数量非常庞大的情况，算法使得总体向着梯度下降快的方向下降。

matlab源码：

[cpp] view plaincopy
  1. function [theta0,theta1]=Gradient_descent_rand(X,Y);  
  2. theta0=0;  
  3. theta1=0;  
  4. t0=theta0;  
  5. t1=theta1;  
  6. for i=1:1:100  
  7.     t0=theta0-0.01*(theta0+theta1*X(i,1)-Y(i,1))*1  
  8.     t1=theta1-0.01*(theta0+theta1*X(i,1)-Y(i,1))*X(i,1)  
  9.     theta0=t0  
  10.     theta1=t1  
  11. end  



杰哥 2012-10-19 13:33 发表评论
]]>[zz]Newton Raphson算法http://www.cppblog.com/guijie/archive/2012/10/16/193347.html杰哥杰哥Mon, 15 Oct 2012 23:21:00 GMThttp://www.cppblog.com/guijie/archive/2012/10/16/193347.htmlhttp://www.cppblog.com/guijie/comments/193347.htmlhttp://www.cppblog.com/guijie/archive/2012/10/16/193347.html#Feedback0http://www.cppblog.com/guijie/comments/commentRss/193347.htmlhttp://www.cppblog.com/guijie/services/trackbacks/193347.htmlhttp://blog.csdn.net/flyingworm_eley/article/details/6517853 

Newton-Raphson算法在统计中广泛应用于求解MLE的参数估计。

对应的单变量如下图：

 

多元函数算法：

 

 

 

Example：（implemented in R）

#定义函数f(x)

f=function(x){
    1/x+1/(1-x)
}

#定义f_d1为一阶导函数

f_d1=function(x){
    -1/x^2+1/(x-1)^2
}

#定义f_d2为二阶导函数

f_d2=function(x){
    2/x^3-2/(x-1)^3
}

 

#NR算法　
NR=function(time,init){
    X=NULL
    D1=NULL   #储存Xi一阶导函数值
D2=NULL   #储存Xi二阶导函数值
    count=0

    X[1]=init
    l=seq(0.02,0.98,0.0002)
    plot(l,f(l),pch='.')
    points(X[1],f(X[1]),pch=2,col=1)

 

    for (i in 2:time){
        D1[i-1]=f_d1(X[i-1])
        D2[i-1]=f_d2(X[i-1])
        X[i]=X[i-1]-1/(D2[i-1])*(D1[i-1])   #NR算法迭代式
        if (abs(D1[i-1])<0.05)break 
        points(X[i],f(X[i]),pch=2,col=i)
        count=count+1
    }
    return(list(x=X,Deriviative_1=D,deriviative2=D2,count))
}


o=NR(30,0.9)

结果如下图：图中不同颜色的三角形表示i次迭代产生的估计值Xi

 

 

o=NR(30,0.9)

 

#另取函数f(x)

f=function(x){
    return(exp(3.5*cos(x))+4*sin(x))
}

 

f_d1=function(x){
    return(-3.5*exp(3.5*cos(x))*sin(x)+4*cos(x))
}

 

f_d2=function(x){
    return(-4*sin(x)+3.5^2*exp(3.5*cos(x))*(sin(x))^2-3.5*exp(3.5*cos(x))*cos(x))
}

 

得到结果如下：

Reference from:

Kevin Quinn

Assistant Professor

Univ Washington



杰哥 2012-10-16 07:21 发表评论
]]>