Skip to article frontmatterSkip to article content

Neural nets

Dense layer

Consider a dense affine layer with 10 inputs and 20 outputs. How many parameters does it have (assume there is a “bias” term)?

Solutions

20×10=20020\times 10=200 weight parameters and 20 bias parameters so 220.

Convolutional layer

Consider a 2d convolutional layer with filterwidth 2 and filterheight 6, 10 input channels and 20 output channels. We are using the layer for images with width 50 and height 50. How many parameters does the layer have?

Solutions

Image size doesn’t matter one whit.

There are 20 filters, each with 2×6×10=1202\times 6 \times 10=120 parameters. So 2400 filter parameters. Then there is a bias term for each output channel. So 2420 in total.

Composing layers

Consider a network architecture formed by composing three simple families of functions. The final network will involve an affine layer

(f(x;β))k=βk0+∑jβkjxj,(f(x;\beta))_k = \beta_{k0}+ \sum_j \beta_{kj} x_j,
(1)

another affine layer

(g(z;γ))k=γk0+∑jγkjzj,(g(z;\gamma))_k = \gamma_{k0}+ \sum_j \gamma_{kj} z_j,
(2)

and a nonlinearity (ReLU(x))k=max⁡(xk,0)(\mathrm{ReLU}(x))_k = \max(x_k,0). Which would be a more typical way to compose these layers into a final neural network architecture?

  1. h(x;β,γ)=ReLU(f(g(x;γ);β))h(x;\beta,\gamma) = \mathrm{ReLU}(f(g(x;\gamma);\beta))

  2. h(x;β,γ)=f(ReLU(g(x;γ));β)h(x;\beta,\gamma) = f(\mathrm{ReLU}(g(x;\gamma));\beta)

  3. h(x;β,γ)=f(g(ReLU(x);γ);β)h(x;\beta,\gamma) = f(g(\mathrm{ReLU}(x);\gamma);\beta)

Solutions

Middle answer is most typical.

Convolution by hand

Compute the convolution of the input X=(1,1,2,3,4,5,6,6)X=(1,1,2,3,4,5,6,6) with the filter (1,−2,1)(1,-2,1).

Solutions

  • First window is (1,1,2)(1,1,2), so we’re looking at 1−2(1)+2=11-2(1)+2=1.

  • Next window is (1,2,3)(1,2,3), so we’re looking at 1−2(2)+3=1−4+3=01-2(2)+3=1-4+3=0.

  • Next window is (2,3,4)(2,3,4), so we’re looking at 2−2(3)+4=2−6+4=02-2(3)+4=2-6+4=0. Ah there’s a pattern here.

  • Next window is (3,4,5)(3,4,5), so we’re looking at 3−2(4)+5=3−8+5=03-2(4)+5=3-8+5=0. Yeah these will all be zero.

  • ... (5,6,6)(5,6,6) will be different. We’ll get 5−2(6)+6=5−12+6=−7+6=−15 -2(6)+6 = 5-12+6=-7+6=-1.

Final answer is (1,0,0,0,0,−1)(1,0,0,0,0,-1).

Note there is some ambiguity about what we meant by convolution here (because sometimes people think about flipping the filter before applying it at each window). Here we avoided that ambiguity by making the filter symmetric.

Function families

Consider the following three functions.

  • g1(x)=x2−cos⁡xg_1(x) = x^2 - \cos x

  • g2(x)=(x+3)2g_2(x) = (x+3)^2

  • g3(x)=(x−3)2+20cos⁡xg_3(x) = (x-3)^2 + 20 \cos x

Devise a two-parameter family of functions that could include all three of those functions. Your answer should define a layer f(x;β1,β2)f(x;\beta_1,\beta_2) that takes a scalar input xx and two parameters β1,β2\beta_1,\beta_2 and returns a number. The layer should be designed so that, for each function gig_i listed above, we can write gi(x)=f(x;β1,β2)g_i(x)=f(x;\beta_1,\beta_2) by choosing β1,β2\beta_1,\beta_2 appropriately.

Solutions

f(x;β1,β2)=(x−β1)2+β2cos⁡xf(x;\beta_1,\beta_2) = (x-\beta_1)^2 + \beta_2 \cos x would suffice.

More fun with function families

Consider the family of linear functions with positive slope: F={g:g(x)=β0+β1x}β1>0F=\{g: g(x)=\beta_0+ \beta_1x\}_{\beta_1> 0}. Devise a one-parameter architecture that includes all these functions, and does not include any functions not in FF. That is, write a layer f(x;β)f(x;\beta) that takes a scalar input and parameter β∈R2\beta \in \mathbb{R}^2 and returns a number. The layer should be designed so that for any g∈Fg \in F it is possible to find β so that f(x;β)=g(x)f(x;\beta)=g(x).

Solutions

f(x;β)=β0+exp⁡(β1)xf(x;\beta) = \beta_0 + \exp(\beta_1) x will suffice.

Gradient descent by hand

Consider the objective L(β)=(β−5)2L(\beta) = (\beta-5)^2. Run one iteration of gradient descent with β initialized at 6. Use a learning rate of 3. What is the loss at initialization? What is the new value of β you get? What is the loss with the new value of β? What can we say about the choice of learning rate in this case?

Solutions

At initialization loss is (6−5)2=1(6-5)^2=1

L′(β)=2(β−5)L'(\beta) = 2(\beta-5), so the gradient at initialization is L′(6)=2(6−5)=2L'(6)=2(6-5)=2. We then take a descent step, β←β−ϵL′(β)\beta \gets \beta - \epsilon L'(\beta), which in this case comes to β←6−3×2=0\beta \gets 6 - 3\times 2 = 0.

Now loss is (0−5)2=25(0-5)^2=25. Not good! Things got worse! Learning rate was way too big.

Improvements to vanilla gradient descent

Consider the loss L(β1,β2)=cos⁡(β1−10)eβ12+500cos⁡(β2−3)2eβ22L(\beta_1,\beta_2) = \cos(\beta_1-10)e^{\beta_1^2} + 500 \cos(\beta_2-3)^2 e^{\beta_2^2}. This loss can be difficult to minimize using gradient descent. Of the following, which would be the most typical method we might employ to help gradient descent find a local optima more quickly?

  1. Minibatch stochastic gradient descent

  2. Gradient descent with momentum

  3. Using GPUs (allowing us to quickly compute a large number of iterations of SGD with very small step-sizes)

  4. Ridge regularization on β

Solutions

Second option. Momentum can help with the unequal magnitude of curvature (function is much curvier in β2\beta_2 parameter).

Minibatch stochastic gradient descent makes no sense because we can very rapidly compute the gradient of LL.

GPUs won’t help here because computing the gradient at each step is pretty fast. GPUs would help if we had to compute many different quantities all at the same time, but they won’t help us compute a problem like this where the result at each stage depends on the result from the previous stage.

Ridge regularization could conceivably help, maybe? But I can’t actually think how right now. Would not be typical. Ridge regularization is usually employed in the context of improving the final estimate, not to make gradient descent converge faster.

The chain rule

All autodiff methods rely on the chain rule to help them figure out how to take derivatives of complicated functions. If h(x)=f(g(x))h(x)=f(g(x)), the chain rule states that...

  1. h′(x)=f′(g(x))g′(x)h'(x)=f'(g(x))g'(x)

  2. h′(x)=f(g′(x))g(x)h'(x)=f(g'(x))g(x)

  3. h′(x)=f′(g(x))/g′(x)h'(x)=f'(g(x))/g'(x)

  4. h′(x)=f(g′(x))/g(x)h'(x)=f(g'(x))/g(x)

Solutions

First answer is correct.

Featurization, standardization, and the magic of splines
Reinforcement learning
Featurization, standardization, and the magic of splines
Misc