Deep Learning Systems: Algorithms and Implementation

2022-10-02

736- 4m- -

Default-dlsystem

课程官网 dlsyscourse.org Youtube Videos

2. Softmax Regression

lecture02: softmax_regression youtube linke

Three ingredients of a machine learning algorithm:

The hypothesis class: describes how we map inputs to outputs.
The loss function: specifies how well a given hypothesis.
An optimization method: minimize the sum of losses over the training set.

Softmax / cross-entropy loss

Use softmax function normalize every dim of output, get the probability of every classes.

$$ z_i = p(label = i) = \frac{exp(h_i(x))}{\sum_{j=1}^{k}exp(h_j(x))} \Longleftrightarrow z \equiv normalize(exp(h(x))) $$ Define a loss to be the negative log probability of the true class, This is call **softmax or cross-entropy loss**.

$$ \ell_{ce}(h(x), y) = -log\ p(label = y) = -h_y(x) + log\sum_{j=1}^{k}exp(h_j(x)) $$ use gradient descent minmizing the average loss on the training set.

$$ \begin{aligned} f(\theta)&=\mathop{minimize}_{\theta} \frac{1}{m}\sum_{i=1}^{m}\ell(h_{\theta}(x^{(i)}, y^{(i)}) \\ \theta :&= \theta - \alpha\nabla_{\theta}f(\theta) \\ batch\ version: \theta :&= \theta - \frac{\alpha}{B}\sum_{i=1}^{B}\nabla_{\theta}\ell(h_{\theta}(x^{(i)}, y^{(i)}) \\ \end{aligned} $$

The gradient of the softmax objective.

$$ \begin{aligned} \frac{\partial \ell_{ce}(h,y)}{\partial h_i} &= \frac{\partial}{\partial h_i}\left({-h_y + log \sum_{j=1}^k exp\ h_j}\right)\\ &=-1\{i=y\} + \frac{exp\ h_i}{\sum_{j=1}^k\ exp\ h_j}\\ \nabla_h\ell_{ce}(h,y)&=z-e_y,\ z = normalize(exp(h)) \end{aligned} $$

How to compute the gradient $\nabla_{\theta}\ell_{ce}(\theta^{T}x, y)$?

Approach #1 (right way): Use matrix differential calculus, Jacobians, Kronecker products, and vectorization

Approach #2 (hacky quick way)：Pretend everything is a scalar, use the typical chain rule, and then rearrange/transpose matrices/vectors to make the sizes work (and check your answer numerically).

$$ \begin{aligned} \frac{\partial}{\partial\theta}\ell_{ce}(\theta^{T}x, y) &= \frac{\partial \ell_{ce}(\theta^Tx, y)}{\partial \theta^{T}x}\frac{\partial\theta^{T}x}{\partial\theta}\\ &=(z-e_y) (x) \end{aligned} $$