让 GPT 给我做了个学习计划, 猜猜我能坚持几天.
预备知识 线性代数 向量, 矩阵, 张量
dim=k 表示沿第 k 个维度做运算, 并通常把这个维度压缩掉.
点积: 两个向量相乘得到标量, 也就是 Hadamard 积后对结果求和
Hadamard 积: 两个同形状张量对应元素分别相乘, 结果仍是同形状张量
矩阵向量积, 矩阵乘法
x = torch.tensor([1.0 , 2.0 , 3.0 ]) y = torch.tensor([4.0 , 5.0 , 6.0 ]) dot_result = torch.dot(x, y) dot_result = x @ y hadamard_result = x * y hadamard_result = torch.mul(x, y) A = torch.tensor([ [1.0 , 2.0 , 3.0 ], [4.0 , 5.0 , 6.0 ] ]) result = A @ x
降维求和, 非降维求和
范数 一个标量, 用于衡量向量/矩阵的大小, 满足非负, 齐次, 三角不等式.
最常见的一类是 L p L_p L p
( ∑ i = 1 n ∣ x i ∣ p ) 1 / p \left(
\sum_{i=1}^{n}|x_i|^p
\right)^{1/p}
( i = 1 ∑ n ∣ x i ∣ p ) 1/ p
最重要的是 L 1 L_1 L 1 L 2 L_2 L 2
也有平方 L 2 L_2 L 2 L 2 L_2 L 2
矩阵也可以定义范数, 常见如 Frobenius 范数, 即把矩阵所有元素拉平成一个长向量, 然后计算 L 2 L_2 L 2
微积分 多年以后, 面对 HAL9000 Skynet, 甜力怕会想起他学习微积分的那个遥远的下午.
梯度 ∇ x f ( x ) = [ ∂ f ( x ) ∂ x 1 , ∂ f ( x ) ∂ x 2 , … , ∂ f ( x ) ∂ x n ] T \nabla_{\mathbf{x}} f(\mathbf{x}) = \left[ \frac{\partial f(\mathbf{x})}{\partial x_1}, \frac{\partial f(\mathbf{x})}{\partial x_2}, \ldots, \frac{\partial f(\mathbf{x})}{\partial x_n} \right]^{\mathrm{T}}
∇ x f ( x ) = [ ∂ x 1 ∂ f ( x ) , ∂ x 2 ∂ f ( x ) , … , ∂ x n ∂ f ( x ) ] T
其中 ∇ x f ( x ) \nabla_{\mathbf{x}} f (\mathbf{x}) ∇ x f ( x ) ∇ f ( x ) \nabla f (\mathbf{x}) ∇ f ( x )
假设 x \mathbf{x} x n n n
对于所有 A ∈ R m × n \mathbf{A} \in \mathbb{R}^{m \times n} A ∈ R m × n ∇ x A x = A ⊤ \nabla_{\mathbf{x}} \mathbf{A} \mathbf{x} = \mathbf{A}^\top ∇ x Ax = A ⊤
对于所有 A ∈ R n × m \mathbf{A} \in \mathbb{R}^{n \times m} A ∈ R n × m ∇ x x ⊤ A = A \nabla_{\mathbf{x}} \mathbf{x}^\top \mathbf{A} = \mathbf{A} ∇ x x ⊤ A = A
对于所有 A ∈ R n × n \mathbf{A} \in \mathbb{R}^{n \times n} A ∈ R n × n ∇ x x ⊤ A x = ( A + A ⊤ ) x \nabla_{\mathbf{x}} \mathbf{x}^\top \mathbf{A} \mathbf{x} = (\mathbf{A} + \mathbf{A}^\top)\mathbf{x} ∇ x x ⊤ Ax = ( A + A ⊤ ) x
∇ x ∥ x ∥ 2 = ∇ x x ⊤ x = 2 x \nabla_{\mathbf{x}} \|\mathbf{x}\|^2 = \nabla_{\mathbf{x}} \mathbf{x}^\top \mathbf{x} = 2\mathbf{x} ∇ x ∥ x ∥ 2 = ∇ x x ⊤ x = 2 x
同样, 对于任何矩阵 X \mathbf{X} X ∇ X ∥ X ∥ F 2 = 2 X \nabla_{\mathbf{X}} \|\mathbf{X}\|_F^2 = 2\mathbf{X} ∇ X ∥ X ∥ F 2 = 2 X
向量对向量求导, 得到雅可比矩阵.
J F ( x ) = [ ∂ y 1 ∂ x 1 ∂ y 1 ∂ x 2 ⋯ ∂ y 1 ∂ x n ∂ y 2 ∂ x 1 ∂ y 2 ∂ x 2 ⋯ ∂ y 2 ∂ x n ⋮ ⋮ ⋱ ⋮ ∂ y m ∂ x 1 ∂ y m ∂ x 2 ⋯ ∂ y m ∂ x n ] J_{\mathbf{F}}(\mathbf{x}) = \begin{bmatrix}
\frac{\partial y_1}{\partial x_1} & \frac{\partial y_1}{\partial x_2} & \cdots & \frac{\partial y_1}{\partial x_n} \\
\frac{\partial y_2}{\partial x_1} & \frac{\partial y_2}{\partial x_2} & \cdots & \frac{\partial y_2}{\partial x_n} \\
\vdots & \vdots & \ddots & \vdots \\
\frac{\partial y_m}{\partial x_1} & \frac{\partial y_m}{\partial x_2} & \cdots & \frac{\partial y_m}{\partial x_n}
\end{bmatrix}
J F ( x ) = ∂ x 1 ∂ y 1 ∂ x 1 ∂ y 2 ⋮ ∂ x 1 ∂ y m ∂ x 2 ∂ y 1 ∂ x 2 ∂ y 2 ⋮ ∂ x 2 ∂ y m ⋯ ⋯ ⋱ ⋯ ∂ x n ∂ y 1 ∂ x n ∂ y 2 ⋮ ∂ x n ∂ y m
概率 概率.
自动微分 反向传播
特征
正向累积
反向累积
传播方向
输入 (\to) 输出
输出 (\to) 输入
传播内容
输入扰动
输出敏感度
一次得到
雅可比的一列, 或 (Jv)
雅可比的一行, 或 (J^\top v)
适合
输入少, 输出多
输入多, 输出少
是否需要保存中间值
通常较少
通常需要保存前向中间结果
神经网络训练
很少作为主要方式
广泛使用
线性神经网络 线性回归 损失函数:
L ( w , b ) = 1 n ∑ i = 1 n l ( i ) ( w , b ) = 1 n ∑ i = 1 n 1 2 ( w ⊤ x ( i ) + b − y ( i ) ) 2 L(\mathbf{w}, b) = \frac{1}{n} \sum_{i=1}^{n} l^{(i)}(\mathbf{w}, b) = \frac{1}{n} \sum_{i=1}^{n} \frac{1}{2} \left( \mathbf{w}^\top \mathbf{x}^{(i)} + b - y^{(i)} \right)^2
L ( w , b ) = n 1 i = 1 ∑ n l ( i ) ( w , b ) = n 1 i = 1 ∑ n 2 1 ( w ⊤ x ( i ) + b − y ( i ) ) 2
其中 1 2 \frac{1}{2} 2 1
生成数据集:
def synthetic_data (w, b, num_examples ): """生成y=Xw+b+噪声""" X = torch.normal(0 , 1 , (num_examples, len (w))) y = torch.matmul(X, w) + b y += torch.normal(0 , 0.01 , y.shape) return X, y.reshape((-1 , 1 )) true_w = torch.tensor([2 , -3.4 ]) true_b = 4.2 features, labels = synthetic_data(true_w, true_b, 1000 )
读取数据集:
def data_iter (batch_size, features, labels ): num_examples = len (features) indices = list (range (num_examples)) random.shuffle(indices) for i in range (0 , num_examples, batch_size): batch_indices = torch.tensor( indices[i: min (i + batch_size, num_examples)]) yield features[batch_indices], labels[batch_indices]
初始化参数:
def data_iter (batch_size, features, labels ): num_examples = len (features) indices = list (range (num_examples)) random.shuffle(indices) for i in range (0 , num_examples, batch_size): batch_indices = torch.tensor( indices[i: min (i + batch_size, num_examples)]) yield features[batch_indices], labels[batch_indices]
定义模型, 损失函数, 优化算法:
def linreg (X, w, b ): """线性回归模型""" return torch.matmul(X, w) + b def linreg (X, w, b ): """线性回归模型""" return torch.matmul(X, w) + b def sgd (params, lr, batch_size ): """小批量随机梯度下降""" with torch.no_grad(): for param in params: param -= lr * param.grad / batch_size param.grad.zero_()
训练:
lr = 0.03 num_epochs = 3 net = linreg loss = squared_loss for epoch in range (num_epochs): for X, y in data_iter(batch_size, features, labels): l = loss(net(X, w, b), y) l.sum ().backward() sgd([w, b], lr, batch_size) with torch.no_grad(): train_l = loss(net(features, w, b), labels) print (f'epoch {epoch + 1 } , loss {float (train_l.mean()):f} ' ) w = net[0 ].weight.data print ('w的估计误差: ' , true_w - w.reshape(true_w.shape))b = net[0 ].bias.data print ('b的估计误差: ' , true_b - b)
利用深度学习框架简单实现:
import numpy as npimport torchfrom torch.utils import datafrom d2l import torch as d2ltrue_w = torch.tensor([2 , -3.4 ]) true_b = 4.2 features, labels = d2l.synthetic_data(true_w, true_b, 1000 ) def load_array (data_arrays, batch_size, is_train=True ): """构造一个PyTorch数据迭代器""" dataset = data.TensorDataset(*data_arrays) return data.DataLoader(dataset, batch_size, shuffle=is_train) batch_size = 10 data_iter = load_array((features, labels), batch_size) from torch import nnnet = nn.Sequential(nn.Linear(2 , 1 )) net[0 ].weight.data.normal_(0 , 0.01 ) net[0 ].bias.data.fill_(0 ) loss = nn.MSELoss() trainer = torch.optim.SGD(net.parameters(), lr=0.03 ) num_epochs = 3 for epoch in range (num_epochs): for X, y in data_iter: l = loss(net(X) ,y) trainer.zero_grad() l.backward() trainer.step() l = loss(net(features), labels) print (f'epoch {epoch + 1 } , loss {l:f} ' ) w = net[0 ].weight.data print ('w的估计误差: ' , true_w - w.reshape(true_w.shape))b = net[0 ].bias.data print ('b的估计误差: ' , true_b - b)
softmax 回归 回归可以用于预测多少的问题, 而我们也对分类问题感兴趣.
one-hot encoding
softmax 函数能够将未规范化的预测变换为非负数并且总和为 1, 同时让模型保持可导的性质.
y ^ j = exp ( o j ) ∑ k exp ( o k ) \hat{y}_j = \frac{\exp(o_j)}{\sum_{k} \exp(o_k)}
y ^ j = ∑ k exp ( o k ) exp ( o j )
softmax 函数的导数就是 softmax 模型分配的概率与实际发生的情况之差:
∂ o j l ( y , y ^ ) = exp ( o j ) ∑ k = 1 q exp ( o k ) − y j = softmax ( o ) j − y j \partial_{o_j} l(\mathbf{y}, \hat{\mathbf{y}}) = \frac{\exp(o_j)}{\sum_{k=1}^{q} \exp(o_k)} - y_j = \operatorname{softmax}(\mathbf{o})_j - y_j
∂ o j l ( y , y ^ ) = ∑ k = 1 q exp ( o k ) exp ( o j ) − y j = softmax ( o ) j − y j
交叉熵损失 衡量模型预测出来的概率分布, 与真实标签所代表的概率分布相差多大.
L = − ∑ i = 1 C y i log ( y ^ i ) L = -\sum_{i=1}^{C} y_i \log(\hat{y}_i)
L = − i = 1 ∑ C y i log ( y ^ i )
最小化交叉熵即最大化训练标签的似然.
交叉熵可以拆解为真实分布熵和 KL 散度之和:
H ( p , q ) = H ( p ) + D K L ( p ∥ q ) H(p, q) = H(p) + D_{KL}(p \parallel q)
H ( p , q ) = H ( p ) + D K L ( p ∥ q )
softmax 回归的实现 样本是 28*28 的图像, 这里展平每个图像, 将其看作长度为 784 的向量.
在 softmax 回归中, 我们的输出与类别一样多, 因此权重构成一个 784*10 的矩阵, 偏置构成一个 1*10 的行向量.
num_inputs = 784 num_outputs = 10 W = torch.normal(0 , 0.01 , size=(num_inputs, num_outputs), requires_grad=True ) b = torch.zeros(num_outputs, requires_grad=True )
实现 softmax 操作:
def softmax (X ): X_exp = torch.exp(X) partition = X_exp.sum (1 , keepdim=True ) return X_exp / partition X = torch.normal(0 , 1 , (2 , 5 )) X_prob = softmax(X) X_prob, X_prob.sum (1 )
定义模型, 损失函数:
def net (X ): return softmax(torch.matmul(X.reshape((-1 , W.shape[0 ])), W) + b) def cross_entropy (y_hat, y ): return - torch.log(y_hat[range (len (y_hat)), y])
计算预测正确的数量:
def accuracy (y_hat, y ): """计算预测正确的数量""" if len (y_hat.shape) > 1 and y_hat.shape[1 ] > 1 : y_hat = y_hat.argmax(axis=1 ) cmp = y_hat.type (y.dtype) == y return float (cmp.type (y.dtype).sum ())
评估任意模型 net 的精度:
def evaluate_accuracy (net, data_iter ): """计算在指定数据集上模型的精度""" if isinstance (net, torch.nn.Module): net.eval () metric = Accumulator(2 ) with torch.no_grad(): for X, y in data_iter: metric.add(accuracy(net(X), y), y.numel()) return metric[0 ] / metric[1 ]
定义 Accumulator 用于对多个变量进行累加:
class Accumulator : """在n个变量上累加""" def __init__ (self, n ): self .data = [0.0 ] * n def add (self, *args ): self .data = [a + float (b) for a, b in zip (self .data, args)] def reset (self ): self .data = [0.0 ] * len (self .data) def __getitem__ (self, idx ): return self .data[idx]
训练:
def train_epoch_ch3 (net, train_iter, loss, updater ): """训练模型一个迭代周期(定义见第3章)""" if isinstance (net, torch.nn.Module): net.train() metric = Accumulator(3 ) for X, y in train_iter: y_hat = net(X) l = loss(y_hat, y) if isinstance (updater, torch.optim.Optimizer): updater.zero_grad() l.mean().backward() updater.step() else : l.sum ().backward() updater(X.shape[0 ]) metric.add(float (l.sum ()), accuracy(y_hat, y), y.numel()) return metric[0 ] / metric[2 ], metric[1 ] / metric[2 ]
绘制数据类 Animator 用于简化代码:
class Animator : """在动画中绘制数据""" def __init__ (self, xlabel=None , ylabel=None , legend=None , xlim=None , ylim=None , xscale='linear' , yscale='linear' , fmts=('-' , 'm--' , 'g-.' , 'r:' 1 , ncols=1 , figsize=(3.5 , 2.5 ): if legend is None : legend = [] d2l.use_svg_display() self .fig, self .axes = d2l.plt.subplots(nrows, ncols, figsize=figsize) if nrows * ncols == 1 : self .axes = [self .axes, ] self .config_axes = lambda : d2l.set_axes( self .axes[0 ], xlabel, ylabel, xlim, ylim, xscale, yscale, legend) self .X, self .Y, self .fmts = None , None , fmts def add (self, x, y ): if not hasattr (y, "__len__" ): y = [y] n = len (y) if not hasattr (x, "__len__" ): x = [x] * n if not self .X: self .X = [[] for _ in range (n)] if not self .Y: self .Y = [[] for _ in range (n)] for i, (a, b) in enumerate (zip (x, y)): if a is not None and b is not None : self .X[i].append(a) self .Y[i].append(b) self .axes[0 ].cla() for x, y, fmt in zip (self .X, self .Y, self .fmts): self .axes[0 ].plot(x, y, fmt) self .config_axes() display.display(self .fig) display.clear_output(wait=True )
训练函数:
def train_ch3 (net, train_iter, test_iter, loss, num_epochs, updater ): """训练模型(定义见第3章)""" animator = Animator(xlabel='epoch' , xlim=[1 , num_epochs], ylim=[0.3 , 0.9 ], legend=['train loss' , 'train acc' , 'test acc' ]) for epoch in range (num_epochs): train_metrics = train_epoch_ch3(net, train_iter, loss, updater) test_acc = evaluate_accuracy(net, test_iter) animator.add(epoch + 1 , train_metrics + (test_acc,)) train_loss, train_acc = train_metrics assert train_loss < 0.5 , train_loss assert train_acc <= 1 and train_acc > 0.7 , train_acc assert test_acc <= 1 and test_acc > 0.7 , test_acc lr = 0.1 def updater (batch_size ): return d2l.sgd([W, b], lr, batch_size)
预测:
def predict_ch3 (net, test_iter, n=6 ): """预测标签(定义见第3章)""" for X, y in test_iter: break trues = d2l.get_fashion_mnist_labels(y) preds = d2l.get_fashion_mnist_labels(net(X).argmax(axis=1 )) titles = [true +'\n' + pred for true, pred in zip (trues, preds)] d2l.show_images( X[0 :n].reshape((n, 28 , 28 )), 1 , n, titles=titles[0 :n]) predict_ch3(net, test_iter)
softmax 的简洁实现 softmax 回归的输出层是一个全连接层, 因此, 为了实现我们的模型, 我们只需在Sequential中添加一个带有 10 个输出的全连接层.
import torchfrom torch import nnfrom d2l import torch as d2lbatch_size = 256 train_iter, test_iter = d2l.load_data_fashion_mnist(batch_size) net = nn.Sequential(nn.Flatten(), nn.Linear(784 , 10 )) def init_weights (m ): if type (m) == nn.Linear: nn.init.normal_(m.weight, std=0.01 ) net.apply(init_weights);
从计算角度来看, 指数可能会造成数值稳定性问题. 当模型输出非常大时, 其指数可能会大于数据类型允许的最大数字, 导致溢出.
一个解决方案是, 在继续 softmax 计算之前, 先从每个 o k o_k o k max ( o k ) \max (o_k) max ( o k )
尽管我们要计算指数函数, 但我们最终在计算交叉熵损失时会取它们的对数. 通过将 softmax 和交叉熵结合在一起, 可以避免反向传播过程中可能会困扰我们的数值稳定性问题. 如下面的等式所示, 我们避免计算 exp ( o j − max ( o k ) ) \exp (o_j - \max (o_k)) exp ( o j − max ( o k )) o j − max ( o k ) o_j - \max (o_k) o j − max ( o k ) log ( exp ( ⋅ ) ) \log (\exp (\cdot)) log ( exp ( ⋅ ))
log ( y ^ j ) = log ( exp ( o j − max ( o k ) ) ∑ k exp ( o k − max ( o k ) ) ) = log ( exp ( o j − max ( o k ) ) ) − log ( ∑ k exp ( o k − max ( o k ) ) ) = o j − max ( o k ) − log ( ∑ k exp ( o k − max ( o k ) ) ) . \begin{aligned}
\log(\hat{y}_j) &= \log \left( \frac{\exp(o_j - \max(o_k))}{\sum_k \exp(o_k - \max(o_k))} \right) \\
&= \log(\exp(o_j - \max(o_k))) - \log \left( \sum_k \exp(o_k - \max(o_k)) \right) \\
&= o_j - \max(o_k) - \log \left( \sum_k \exp(o_k - \max(o_k)) \right).
\end{aligned}
log ( y ^ j ) = log ( ∑ k exp ( o k − max ( o k )) exp ( o j − max ( o k )) ) = log ( exp ( o j − max ( o k ))) − log ( k ∑ exp ( o k − max ( o k )) ) = o j − max ( o k ) − log ( k ∑ exp ( o k − max ( o k )) ) .
当然我们的深度学习框架早就帮我们写好了.
loss = nn.CrossEntropyLoss(reduction='none' )
优化算法:
trainer = torch.optim.SGD(net.parameters(), lr=0.1 )
训练:
num_epochs = 10 d2l.train_ch3(net, train_iter, test_iter, loss, num_epochs, trainer)