A classification of critical points with the Hessian matrix

수학 모음 (Maths collection) 전체보기

For a function which depends on two variables $(x,y)$, $$ \textrm f = \textrm f(x,y) $$ we find the critical points by considering 2-dimensional gradient and second-order derivatives. (In 1D, the critical points are usually called the stationary points.)

 

(i) Critical points: $$ \begin{align} \nabla \textrm f = \left( \frac{ \partial \textrm f }{ \partial x }, \frac{ \partial \textrm f }{ \partial y } \right) = 0 \end{align} $$

 

(ii) A classification: We consider a 2-dimensional Taylor expansion $$ \begin{align} \textrm f( \textbf x + \Delta \textbf x) &= \textrm f( \textbf x ) + \left( \frac{ \partial \textrm f }{ \partial x } \Delta x + \frac{ \partial \textrm f }{ \partial y } \Delta y \right) + \frac12 \left[ \frac{ \partial^2 \textrm f }{ \partial x^2 } (\Delta x)^2 + \frac{ \partial^2 \textrm f }{ \partial x \partial y } \Delta x \Delta y + \frac{ \partial \textrm f }{ \partial y } (\Delta y)^2 \right] + \cdots \\ &= \textrm f( \textbf x ) + \underbrace{ \begin{pmatrix} \frac{ \partial \textrm f }{ \partial x } \\ \frac{ \partial \textrm f }{ \partial y } \end{pmatrix} }_{ \nabla \textrm f } \cdot \underbrace{ \begin{pmatrix} \Delta x \\ \Delta y \end{pmatrix} }_{ \Delta \textbf x } + \frac12 \underbrace{ \begin{pmatrix} \Delta x & \Delta y \end{pmatrix} }_{ \Delta \textbf x^\intercal } \underbrace{ \begin{pmatrix} \frac{ \partial^2 \textrm f }{ \partial x^2 } & \frac{ \partial^2 \textrm f }{ \partial x \partial y } \\ \frac{ \partial^2 \textrm f }{ \partial y \partial x } & \frac{ \partial^2 \textrm f }{ \partial y^2 } \end{pmatrix} }_{ H } \underbrace{ \begin{pmatrix} \Delta x \\ \Delta y \end{pmatrix} }_{ \Delta \textbf x } \\ &= \textrm f( \textbf x ) + \nabla \textrm f \cdot \Delta \textbf x + \frac12 \Delta \textbf x^\intercal H \Delta \textbf x + \cdots \end{align}$$ where the Hessian matrix $H$ is defined by $$ \begin{align} H = \begin{pmatrix} \frac{ \partial^2 \textrm f }{ \partial x^2 } & \frac{ \partial^2 \textrm f }{ \partial x \partial y } \\ \frac{ \partial^2 \textrm f }{ \partial y \partial x } & \frac{ \partial^2 \textrm f }{ \partial y^2 } \end{pmatrix} \end{align} $$

 

Aside. A multi-dimensional Taylor expansion reads $$ \begin{align} \textrm f( \textbf x + \Delta \textbf x) &= \textrm f( \textbf x ) + \sum_{i=1}^n \frac{ \partial \textrm f }{ \partial x_i } \Delta x_i + \sum_{i,j} \frac{ \partial^2 \textrm f }{ \partial x_i \partial x_j } \Delta x_i \Delta x_j + \sum_{i,j,k} \frac{ \partial^3 \textrm f }{ \partial x_i \partial x_j \partial x_k } \Delta x_i \Delta x_j \Delta x_k + \cdots \end{align} $$

 

(iii) Diagonalisation: As we diagonalise the Hessian matrix using the eigenvalue equation, $$ \begin{align} && H \begin{pmatrix} \textbf e_1 & \textbf e_2 \end{pmatrix} = \underbrace{ \begin{pmatrix} \textbf e_1 & \textbf e_2 \end{pmatrix} }_{ P^{-1} } \begin{pmatrix} \lambda_1 & 0 \\ 0 & \lambda_2 \end{pmatrix} \\ \\ &\Rightarrow& H = \begin{pmatrix} \textbf e_1 & \textbf e_2 \end{pmatrix} \begin{pmatrix} \lambda_1 & 0 \\ 0 & \lambda_2 \end{pmatrix} \begin{pmatrix} \textbf e_1 & \textbf e_2 \end{pmatrix}^{-1} = P^{-1} \Lambda P \end{align} $$ Since $H$ is real symmetric, the eigenvectors form an orthonormal basis and $P$ is thus an orthogonal matrix, i.e. $$ P^\intercal P = \mathbb I \qquad \Leftrightarrow \qquad P^{-1} = P^\intercal $$ which gives $$ H = P^\intercal \Lambda P $$

 

The Taylor expansion may be re-written as $$ \begin{align} \textrm f( \textbf x + \Delta \textbf x) &= \textrm f( \textbf x ) + \Delta \textbf x \cdot \nabla \textrm f + \frac12 ( P \Delta \textbf x)^\intercal \Lambda ( P \Delta \textbf x ) + \cdots \end{align} $$ For critical points $\textbf x_0$, where $\nabla \textrm f = 0$, this gives $$ \begin{align} \textrm f( \textbf x_0 + \Delta \textbf x) &= \textrm f( \textbf x_0 ) + \frac12 ( P \Delta \textbf x)^\intercal \Lambda ( P \Delta \textbf x ) + \cdots \\ &= \textrm f( \textbf x_0 ) + \frac12 ( P \Delta \textbf x)^\intercal \begin{pmatrix} \lambda_1 & 0 \\ 0 & \lambda_2 \end{pmatrix} ( P \Delta \textbf x ) + \cdots \end{align} $$ Thus, the eigenvalues of $H$ tells about the nature of the critical point(s), i.e.

• If $ \lambda_1 > 0 $ and $ \lambda_2 > 0 $, $ \textrm f( \textbf x_0 + \Delta \textbf x) > \textrm f ( \textbf x_0 ) $ for all $ \Delta \textbf x $, hence the critical point is a local minimum.
• If $ \lambda_1 < 0 $ and $ \lambda_2 < 0 $, $ \textrm f( \textbf x_0 + \Delta \textbf x) < \textrm f ( \textbf x_0 ) $ for all $ \Delta \textbf x $, hence the critical point is a local maximum.
• If $ \lambda_1 \lambda_2 < 0 $, i.e. the two eigenvalues take the opposite signs, $ \textrm f( \textbf x_0 + \Delta \textbf x) > \textrm f ( \textbf x_0 ) $ in one direction while $ \textrm f( \textbf x_0 + \Delta \textbf x) < \textrm f ( \textbf x_0 ) $ in its orthogonal direction, henc the critical point is a saddle point.
• If $ \lambda_1 \lambda_2 = 0 $, i.e. at least one of them is zero, then the critical point is degenerate as the surface is flat in the relevant direction - the direction of the eigenvector whose eigenvalue is zero.

 

(iv) Eigenvalues: The eigenvalues of the Hessian matrix are given by $$ \begin{align} && \begin{vmatrix} \textrm f_{xx} - \lambda & \textrm f_{xy} \\ \textrm f_{xy} & \textrm f_{yy} - \lambda \end{vmatrix} = 0 \\ \\ & \Rightarrow & ( \textrm f_{xx} - \lambda )( \textrm f_{yy} - \lambda ) - \textrm f_{xy}^2 = 0 \\ & \Rightarrow & \lambda^2 - ( \textrm f_{xx} + \textrm f_{yy} ) \lambda + \left( \textrm f_{xx} \textrm f_{yy} - \textrm f_{xy}^2 \right) = 0 \\ & \Rightarrow & \lambda^2 - ( \textrm{tr} \, H ) \lambda + \det H = 0 \\ \\ & \Rightarrow & \lambda = \frac{ \textrm{tr} \, H \pm \sqrt{ ( \textrm{tr} \, H )^2 - 4 \det H } }{ 2 } \end{align} $$ which gives $$ \begin{align} & \Rightarrow & \lambda &= \frac{ ( \textrm f_{xx} + \textrm f_{yy} ) \pm \sqrt{ ( \textrm f_{xx} + \textrm f_{yy} )^2 - 4 \left( \textrm f_{xx} \textrm f_{yy} - \textrm f_{xy}^2 \right) } }{ 2 } \\ &&&= \frac{ ( \textrm f_{xx} + \textrm f_{yy} ) \pm \sqrt{ \left( \textrm f_{xx}^2 + 2 \textrm f_{xx} \textrm f_{yy} + \textrm f_{yy}^2 \right) - 4 \left( \textrm f_{xx} \textrm f_{yy} - \textrm f_{xy}^2 \right) } }{ 2 } \\ &&&= \frac{ ( \textrm f_{xx} + \textrm f_{yy} ) \pm \sqrt{ \left( \textrm f_{xx}^2 - 2 \textrm f_{xx} \textrm f_{yy} + \textrm f_{yy}^2 \right) + 4 \textrm f_{xy}^2 } }{ 2 } \\ &&&= \frac{ ( \textrm f_{xx} + \textrm f_{yy} ) \pm \sqrt{ ( \textrm f_{xx} - \textrm f_{yy} )^2 + 4 \textrm f_{xy}^2 } }{ 2 } \end{align} $$ So we see that the discriminant is always non-negative, i.e. $$ \Delta = ( \textrm{tr} \, H )^2 - 4 \det H = ( \textrm f_{xx} - \textrm f_{yy} )^2 + 4 \textrm f_{xy}^2 \ge 0 $$

 

(1) For $\Delta = 0$, i.e. $$ ( \textrm{tr} \, H )^2 = 4 \det H \qquad \Leftrightarrow \qquad \textrm f_{xx} = \textrm f_{yy} \quad \textrm{and} \quad \textrm f_{xy} = 0 $$ the eigenvalues are double roots and $ \det H > 0$. $$ p = \frac{ \textrm{tr} \, H }{ 2 } = \frac{ \textrm f_{xx} + \textrm f_{yy} }{ 2 } $$

• For $ \textrm{tr} \, H > 0$, it gives a local minimum.
• For $ \textrm{tr} \, H < 0$, it gives a local maximum.
• For $ \textrm{tr} \, H = 0$, we also find $ \det H = 0 $ and the critical point is degenerate.

 

(2) For $\Delta > 0$: then there are two distinct eigenvalues.

• If $\det H = \textrm f_{xx} \textrm f_{yy} - \textrm f_{xy}^2 <0$, the two eigenvalues are of the opposite signs. The critical point is a saddle point.
• If $\det H = \textrm f_{xx} \textrm f_{yy} - \textrm f_{xy}^2 = 0$, one of the eigenvalues is zero. $$ p_1 = 0 \quad \textrm{and} \quad p_2 = \textrm{tr} \, H $$ The critical point is degenerate as the surface is flat in the relevant direction - the direction of the eigenvector whose eigenvalue is zero.

• If $\det H = \textrm f_{xx} \textrm f_{yy} - \textrm f_{xy}^2 >0$,

  (i) if $ \textrm{tr} \, H = \textrm f_{xx} + \textrm f_{yy} > 0 $, the critical point is a local minimum;

  (ii) if $ \textrm{tr} \, H = \textrm f_{xx} + \textrm f_{yy} < 0 $, the critical point is a local maximum.

• If $ \textrm{tr} \, H = \textrm f_{xx} + \textrm f_{yy} = 0 $, then $$ p = \pm \sqrt{ - \det H } $$ and $$ \begin{align} \det H &= \textrm f_{xx} \textrm f_{yy} - \textrm f_{xy}^2 \\ &= - \textrm f_{xx}^2 - \textrm f_{xy}^2 \le 0 \end{align} $$

  (i) if $ \det H = 0 $, i.e. $ \textrm f_{xx} = \textrm f_{yy} = \textrm f_{xy} = 0 $, the two eigenvalues are both zero so the critical point is degenerate;

  (ii) if $ \det H < 0 $, the two eigenvalues take the opposite signs so the critical point is a saddle point.

The results can be summarised: $$ \begin{align} \begin{array}{|c|c|c|c|} \hline & \det H = \textrm f_{xx} \textrm f_{yy} - \textrm f_{xy}^2 > 0 & \det H = \textrm f_{xx} \textrm f_{yy} - \textrm f_{xy}^2 = 0 & \det H = \textrm f_{xx} \textrm f_{yy} - \textrm f_{xy}^2 < 0 \\\hline \textrm{tr} \, H = \textrm f_{xx} + \textrm f_{yy} > 0 & \textrm{A lcoal minimum} & \textrm{Degenerate} & \textrm{A saddle point} \\\hline \textrm{tr} \, H = \textrm f_{xx} + \textrm f_{yy} < 0 & \textrm{A lcoal maximum} & \textrm{Degenerate} & \textrm{A saddle point} \\\hline \textrm{tr} \, H = \textrm f_{xx} + \textrm f_{yy} = 0 & \textrm{Not possible} & \textrm{Degenerate} & \textrm{A saddle point} \\\hline \end{array} \end{align} $$

 

	$ \det H = \textrm f_{xx} \textrm f_{yy} - \textrm f_{xy}^2 > 0 $	$ \det H = \textrm f_{xx} \textrm f_{yy} - \textrm f_{xy}^2 = 0 $	$ \det H = \textrm f_{xx} \textrm f_{yy} - \textrm f_{xy}^2 < 0$
$ \textrm{tr} \, H = \textrm f_{xx} + \textrm f_{yy} > 0$	A local minimum	Degenerate	A saddle point
$ \textrm{tr} \, H = \textrm f_{xx} + \textrm f_{yy} < 0$	A local maximum	Degenerate	A saddle piont
$ \textrm{tr} \, H = \textrm f_{xx} + \textrm f_{yy} = 0 $	Not possible	Degenerate	A saddle point