深度学习中的梯度计算（矩阵向量求导）

0. 标量、向量、矩阵互相求导的形状

PS：默认使用列向量和分子布局（分子不变，分母转置）。

1. 标量对向量求导 $frac{partial y}{partialtextbf x}$　

　　$textbf xleft ( n,1 right )= begin{bmatrix}
x_{1}\
x_{2}\
vdots\
x_{n}
end{bmatrix}$ ，y为标量，

　　$frac{partial y}{partialtextbf x}left ( 1,n right )= begin{bmatrix}
frac{partial y}{partial x_{1}} & frac{partial y}{partial x_{2}} & cdots & frac{partial y}{partial x_{n}}
end{bmatrix}$

　　PS：标量对列向量求导，变为行向量，标量对向量每一元素求导。

2. 向量对标量求导 $frac{partialtextbf y}{partial x}$

　　x为标量，$textbf yleft ( m,1 right )= begin{bmatrix}
y_{1}\
y_{2}\
vdots\
y_{m}
end{bmatrix}$ ，

　　$frac{partialtextbf y}{partial x}left ( m,1 right )= begin{bmatrix}
frac{partial y_{1}}{partial x}\
frac{partial y_{2}}{partial x}\
vdots\
frac{partial y_{m}}{partial x}
end{bmatrix}$

　　PS：向量对标量求导，形状不变，向量每一元素对标量求导。

3. 向量对向量求导 $frac{partialtextbf y}{partialtextbf x}$

　　$textbf xleft ( n,1 right )= begin{bmatrix}
x_{1}\
x_{2}\
vdots\
x_{n}
end{bmatrix}$，$textbf yleft ( m,1 right )= begin{bmatrix}
y_{1}\
y_{2}\
vdots\
y_{m}
end{bmatrix}$，

　　$frac{partialtextbf y}{partialtextbf x}left ( m,n right )=
begin{bmatrix}
frac{partial y_{1}}{partialtextbf x}\
frac{partial y_{2}}{partialtextbf x}\
vdots\
frac{partial y_{m}}{partialtextbf x}
end{bmatrix} =
begin{bmatrix}
frac{partial y_{1}}{partial x_{1}} & frac{partial y_{1}}{partial x_{2}} & cdots & frac{partial y_{1}}{partial x_{n}}\
frac{partial y_{2}}{partial x_{1}} & frac{partial y_{2}}{partial x_{2}} & cdots & frac{partial y_{2}}{partial x_{n}}\
vdots & vdots & ddots & vdots\
frac{partial y_{m}}{partial x_{1}} & frac{partial y_{m}}{partial x_{2}} & cdots & frac{partial y_{m}}{partial x_{n}}
end{bmatrix}$

　　PS：向量对向量求导，形状为矩阵，可以理解为一列标量分别对向量求导。

4. 标量对矩阵求导 $frac{partial y}{partialtextbf X}$

　　$textbf Xleft ( n,k right )=
begin{bmatrix}
x_{11} & x_{12} & cdots & x_{1k}\
x_{21} & x_{22} & cdots & x_{2k}\
vdots & vdots & ddots & vdots\
x_{n1} & x_{n2} & cdots & x_{nk}
end{bmatrix}$，y为标量，

　　$frac{partial y}{partialtextbf X}left ( k,n right )=
begin{bmatrix}
frac{partial y}{partialtextbf x_{:,1}} & frac{partial y}{partialtextbf x_{:,2}} & cdots & frac{partial y}{partialtextbf x_{:,k}}
end{bmatrix}=
begin{bmatrix}
frac{partial y}{partial x_{11}} & frac{partial y}{partial x_{21}} & cdots & frac{partial y}{partial x_{n1}}\
frac{partial y}{partial x_{12}} & frac{partial y}{partial x_{22}} & cdots & frac{partial y}{partial x_{n2}}\
vdots & vdots & ddots & vdots\
frac{partial y}{partial x_{1k}} & frac{partial y}{partial x_{2k}} & cdots & frac{partial y}{partial x_{nk}}
end{bmatrix}$

　　PS：标量对矩阵求导，形状为转置的矩阵，可以理解为标量分别对k个列向量求导。

5. 矩阵对标量求导 $frac{partialtextbf Y}{partial x}$

　　x为标量，$textbf Yleft ( m,l right )=
begin{bmatrix}
y_{11} & y_{12} & cdots & y_{1l}\
y_{21} & y_{22} & cdots & y_{2l}\
vdots & vdots & ddots & vdots\
y_{m1} & y_{m2} & cdots & y_{ml}
end{bmatrix}$，

　　$frac{partialtextbf Y}{partial x}left ( m,l right )=
begin{bmatrix}
frac{partialtextbf y_{:,1}}{partial x} & frac{partialtextbf y_{:,2}}{partial x} & cdots & frac{partialtextbf y_{:,l}}{partial x}
end{bmatrix}=
begin{bmatrix}
frac{partial y_{11}}{partial x} & frac{partial y_{12}}{partial x} & cdots & frac{partial y_{1l}}{partial x}\
frac{partial y_{21}}{partial x} & frac{partial y_{22}}{partial x} & cdots & frac{partial y_{2l}}{partial x}\
vdots & vdots & ddots & vdots\
frac{partial y_{m1}}{partial x} & frac{partial y_{m2}}{partial x} & cdots & frac{partial y_{ml}}{partial x}
end{bmatrix}$

　　PS：矩阵对标量求导，形状不变，可以理解为l个列向量分别对标量求导。

6. 向量对矩阵求导 $frac{partialtextbf y}{partialtextbf X}$

　　$textbf Xleft ( n,k right )=
begin{bmatrix}
x_{11} & x_{12} & cdots & x_{1k}\
x_{21} & x_{22} & cdots & x_{2k}\
vdots & vdots & ddots & vdots\
x_{n1} & x_{n2} & cdots & x_{nk}
end{bmatrix}$，$textbf yleft ( m,1 right )= begin{bmatrix}
y_{1}\
y_{2}\
vdots\
y_{m}
end{bmatrix}$，

　　$frac{partialtextbf y}{partialtextbf X}left ( m,k,n right )=
begin{bmatrix}frac{partial y_{1}}{partialtextbf X}end{bmatrix},begin{bmatrix}frac{partial y_{2}}{partialtextbf X}end{bmatrix},cdots,begin{bmatrix}frac{partial y_{m}}{partialtextbf X}end{bmatrix}=begin{bmatrix}
frac{partial y_{1}}{partial x_{11}} & frac{partial y_{1}}{partial x_{21}} & cdots & frac{partial y_{1}}{partial x_{n1}}\
frac{partial y_{1}}{partial x_{12}} & frac{partial y_{1}}{partial x_{22}} & cdots & frac{partial y_{1}}{partial x_{n2}}\
vdots & vdots & ddots & vdots\
frac{partial y_{1}}{partial x_{1k}} & frac{partial y_{1}}{partial x_{2k}} & cdots & frac{partial y_{1}}{partial x_{nk}}
end{bmatrix},begin{bmatrix}
frac{partial y_{2}}{partial x_{11}} & frac{partial y_{2}}{partial x_{21}} & cdots & frac{partial y_{2}}{partial x_{n1}}\
frac{partial y_{2}}{partial x_{12}} & frac{partial y_{2}}{partial x_{22}} & cdots & frac{partial y_{2}}{partial x_{n2}}\
vdots & vdots & ddots & vdots\
frac{partial y_{2}}{partial x_{1k}} & frac{partial y_{2}}{partial x_{2k}} & cdots & frac{partial y_{2}}{partial x_{nk}}
end{bmatrix},cdots,begin{bmatrix}
frac{partial y_{m}}{partial x_{11}} & frac{partial y_{m}}{partial x_{21}} & cdots & frac{partial y_{m}}{partial x_{n1}}\
frac{partial y_{m}}{partial x_{12}} & frac{partial y_{m}}{partial x_{22}} & cdots & frac{partial y_{m}}{partial x_{n2}}\
vdots & vdots & ddots & vdots\
frac{partial y_{m}}{partial x_{1k}} & frac{partial y_{m}}{partial x_{2k}} & cdots & frac{partial y_{m}}{partial x_{nk}}
end{bmatrix}$

　　PS：向量对矩阵求导，形状为3维数组，可以理解为y的每个元素（标量）分别对矩阵求导，结果为m个k*n矩阵的组合。

7. 矩阵对向量求导 $frac{partialtextbf Y}{partialtextbf x}$

　　$textbf xleft ( n,1 right )=begin{bmatrix}
x_{1}\
x_{2}\
vdots\
x_{n}
end{bmatrix}$，$textbf Yleft ( m,l right )=
begin{bmatrix}
y_{11} & y_{12} & cdots & y_{1l}\
y_{21} & y_{22} & cdots & y_{2l}\
vdots & vdots & ddots & vdots\
y_{m1} & y_{m2} & cdots & y_{ml}
end{bmatrix}$，

　　$frac{partialtextbf Y}{partialtextbf x}left ( m,l,n right )=begin{bmatrix}
frac{partial y_{11}}{partialtextbf x} & frac{partial y_{12}}{partialtextbf x} & cdots & frac{partial y_{1l}}{partialtextbf x}\
frac{partial y_{21}}{partialtextbf x} & frac{partial y_{22}}{partialtextbf x} & cdots & frac{partial y_{2l}}{partialtextbf x}\
vdots & vdots & ddots & vdots\
frac{partial y_{m1}}{partialtextbf x} & frac{partial y_{m2}}{partialtextbf x} & cdots & frac{partial y_{ml}}{partialtextbf x}
end{bmatrix}$

　　PS：矩阵对向量求导，形状为3维的数组，没搞懂，搞懂了再来写哈哈。

8. 矩阵对矩阵求导 $frac{partialtextbf Y}{partialtextbf X}$

　　$textbf Xleft ( n,k right )=
begin{bmatrix}
x_{11} & x_{12} & cdots & x_{1k}\
x_{21} & x_{22} & cdots & x_{2k}\
vdots & vdots & ddots & vdots\
x_{n1} & x_{n2} & cdots & x_{nk}
end{bmatrix}$，$textbf Yleft ( m,l right )=
begin{bmatrix}
y_{11} & y_{12} & cdots & y_{1l}\
y_{21} & y_{22} & cdots & y_{2l}\
vdots & vdots & ddots & vdots\
y_{m1} & y_{m2} & cdots & y_{ml}
end{bmatrix}$，

　　PS：搞懂再来写。

• 还没有人评论，欢迎说说您的想法！

相关课程

基于 OpenVINO™ 的 AI 视觉应用基础课

56016 0元 限免

英特尔® OpenVINO™工具套件初级课程

293013 0元 限免

英特尔 OpenCV 初级认证课程

7696 0元 限免