1.1 Background

Artificial neurons date back to the 1940s and neural networks to the 1980s. These are not new techniques, but we surprisingly still have a lot to learn about how they work and how to best create them. Research into neural networks accelerated in the late 2000s as we found ourselves with more data and more compute power.

Neural networks (NN) are a type of machine learning (ML) technique. ML falls under the artificial intelligence (AI) umbrella. AI is a broad area, and it doesn’t hurt to think of it as including any system that appears to do something useful or complex.

You’ll find that techniques often start out as AI, but then we remove that label after we start to better understand them. ML is just one type of AI, and it comprises all techniques that automatically learn from data. NNs learn from data, and specifically they do so using very little input from the designer.

All of this is a bit vague, so let’s discuss some specific applications. Maybe we want a NN to:

• Tell us if an Amazon review is positive or negative based on text alone.
• Tell us if an image contains a cat or a dog.
• Translate an English sentence to German.
• Tell us where in an image we can find a boat.
• Automatically generate a caption for an image.
• Direct a robot around a building.
• Play a board game or a video game.
• Tell us about the orientation of a persons limb’s for a virtual reality game.
• Prevent an autonomous car from driving off the road.
• Group together all users of a social network that are likely to listen to the same music.
• Create a new piece of art.
• Predict the sale price of a house.
• Predict the future sale price of an investment.
• Suggest products to purchase or movies to watch.
• Diagnose an injury from an X-ray CT scan.
• Automatically summarize a news article.
• Label a news article as fake or real.

This is just a small subset of what we could do. Nearly all applications have the same basic flow:

The core of the NN is the ability to take and input, perform some mathematical computations, and then produce the output. The “learning” part includes comparing the output to a known-to-be-correct output (aka the “label” or “target”) and then using this comparison to iteratively improve the NN.

This setup, where we know the correct output, is known as “supervised learning.” Later parts of this guide will touch on “unsupervised learning” and “reinforcement learning,” but it is safe to say that most ML applications are in the area of supervised learning.

1.2 Additional Material

• A Whirlwind Tour of Python (free Book)
• The Matrix Calculus You Need For Deep Learning (web-page)
• Introduction — Dive into Deep Learning (free Book)

2.1 Key Topics

This is not going to be an exhaustive discussion on ethics in AI and NNs. Instead, I’ll point you to the resources in the Additional Material section. The topics below are taken from Ethics — fastbook.

Topics to consider:

1. Recourse and accountability: who is responsible (and liable) for the developed system? The user, developer, manager, owner, company, other?
2. Feedback loops: does the system control creation of the next round of input data (such as a video recommendation system)?
3. Bias: all systems have bias; what bias is in your system? Is the source of bias historical, from measurement, from aggregation, from the representation, other?
4. Disinformation: can your system be used for nefarious goals?

2.2 Strategies

Here are some questions you can ask to prevent running into trouble:

• General Questions:

  • Should we even be doing this?
  • What might be the accuracy of a simple non-ML alternative?
  • What processes will we use to handle appeals/mistakes?
  • How diverse is our team?

• Data Questions:

  • Is our data valid for its intended use?
  • What bias could be in our data? (All data contains bias.)
  • How could we minimize bias in our data and model?
  • How should we “audit” our code and data?

• Impact Questions:

  • Do we expect different errors rates for different sub-groups in the data?
  • What are likely misinterpretations of the results and what can be done to prevent those misinterpretations?
  • How might we impinge individuals’ privacy and/or anonymity?

When should you ask these questions? The Markkula Center for Applied Ethics recommends scheduling regular meeting in which you perform ethical risk sweeping. See their Ethical Toolkit for more information.

2.3 Additional Material

• Practical Data Ethics
• Fair ML Book
• Machine Ethics Podcast
• Codes of Ethics from the ACM, IEEE, and the Data Science Association

3.1 Loading MNIST Using PyTorch

We’ve discuss notation and general concepts, but how would we write this out in code? Here is an example how how to load the MNIST dataset using PyTorch.

   1 #!/usr/bin/env python
   2 
   3 from torch.utils.data import DataLoader
   4 from torchvision.datasets import MNIST
   5 from torchvision.transforms import Compose, Normalize, ToTensor
   6 
   7 # Location in which to store downloaded data
   8 data_dir = "../Data"
   9 
  10 # I used torch.std_mean to find the values given to Normalize
  11 # We will discuss normalization in section 4
  12 mnist_xforms = Compose([ToTensor(), Normalize((0.1307,), (0.3081,))])
  13 
  14 # Load data files (training and validation partitions)
  15 train_data = MNIST(root=data_dir, train=True, download=True, transform=mnist_xforms)
  16 valid_data = MNIST(root=data_dir, train=False, download=True, transform=mnist_xforms)
  17 
  18 # Data loaders provide an easy interface for interactive with data
  19 train_loader = DataLoader(train_data, batch_size=len(train_data))
  20 valid_loader = DataLoader(valid_data, batch_size=len(valid_data))
  21 
  22 # This odd bit of code forces the train loader to give us all inputs and targets
  23 X_train, y_train = next(iter(train_loader))
  24 X_valid, y_valid = next(iter(valid_loader))
  25 
  26 # Let's start by simply printing out some basic information
  27 print("Training input shape    :", X_train.shape)
  28 print("Training target shape   :", y_train.shape)
  29 print("Validation input shape  :", X_valid.shape)
  30 print("Validation target shape :", y_valid.shape)

Training input shape    : torch.Size([60000, 1, 28, 28])
Training target shape   : torch.Size([60000])
Validation input shape  : torch.Size([10000, 1, 28, 28])
Validation target shape : torch.Size([10000])

3.2 Similarity Digit Classifier

Before we get into training NNs, we will start with a non-ML classifier. This will provide a nice comparison, and show that ML must be learning something beyond simple comparisons.

Let’s try to solve the following problem:

For reference, here is what the “average” looks like for each of the ten digits.

Before we show a solution, however, we should take a guess at how well a random guesser might perform.

And now some code for finding the most similar digit.

   1 #!/usr/bin/env python
   2 
   3 from math import inf
   4 from matplotlib import pyplot as plt
   5 
   6 import torch
   7 from torch.utils.data import DataLoader
   8 from torchvision.datasets import MNIST
   9 from torchvision.transforms import Compose, Normalize, ToTensor
  10 
  11 # Location in which to store downloaded data
  12 data_dir = "../Data"
  13 
  14 # I used torch.std_mean to find the values given to Normalize
  15 # We will discuss normalization in section 4
  16 mnist_xforms = Compose([ToTensor(), Normalize((0.1307,), (0.3081,))])
  17 
  18 # Load data files (training and validation partitions)
  19 train_data = MNIST(root=data_dir, train=True, download=True, transform=mnist_xforms)
  20 valid_data = MNIST(root=data_dir, train=False, download=True, transform=mnist_xforms)
  21 
  22 # Data loaders provide an easy interface for interactive with data
  23 train_loader = DataLoader(train_data, batch_size=len(train_data))
  24 valid_loader = DataLoader(valid_data, batch_size=len(valid_data))
  25 
  26 # This odd bit of code forces the train loader to give us all inputs and targets
  27 X_train, y_train = next(iter(train_loader))
  28 X_valid, y_valid = next(iter(valid_loader))
  29 
  30 # Let's get the average for each digit based on all training examples
  31 digit_averages = {}
  32 for digit in range(10):
  33     digit_averages[digit] = X_train[y_train == digit].mean(dim=0).squeeze()
  34 
  35 
  36 # Next up we need to compare "unknown" images to our average images
  37 def get_most_similar(image: torch.Tensor, averages: dict):
  38     """Compare the image to each of the averaged images.
  39 
  40     Args:
  41         image (torch.Tensor): an image represented as a tensor
  42         averages (dict): a dictionary of averaged images
  43 
  44     Returns:
  45         the most similar label
  46     """
  47     closest_label = None
  48     closest_distance = inf
  49     for label in averages:
  50         distance = (image - averages[label]).abs().mean()
  51         if distance < closest_distance:
  52             closest_label = label
  53             closest_distance = distance
  54     return closest_label
  55 
  56 
  57 # Now we can get the most similar label for each validation image
  58 num_correct = 0
  59 for image, label in zip(X_valid, y_valid):
  60     num_correct += label == get_most_similar(image, digit_averages)
  61 
  62 print(f"Percent guessed correctly: {num_correct/len(X_valid)*100:.2f}%")

4.1 Notation and Diagram

Here is a diagram representing a single neuron (as we’ll see later, some neural networks are just many of these neurons interconnected):

The diagram represents the following equations (note that I removed the parenthesis superscript from the diagram to make it a bit easier to read):

\[\begin{align} z\i &= \sum_{k=1}^{n_x} x_k\i w_k + b\\ a\i &= g(z\i) \end{align}\]

For these two equations:

• \(x_k\i\) are the input features for the \(i^{th}\) example (e.g., \(k=76\) and \(i=7436\) would denote pixel 76 of 784 for image 7436 of 60000)
• \(w_k\) (weights) and \(b\) (bias) are the learned parameters
• \(z\i\) is a weighted sum of the input features plus the additional bias term
• \(a\i\) is the output of a non-linear activation function \(g(\mathord{\cdot})\) applied to \(z\i\)
• \(\yhat\i\) (pronounced “y hat”) is the label we often give to the output (\(a\i = \yhat\i\))

For this model, we want to find parameters \(w_k\) and \(b\) such that the neuron outputs \(\yhat\i \approx y\i\) for any input. Before we discuss optimization we should take a moment to code up this single neuron model.

(Below is a more common representation of a neuron model. The image above separates the linear and activation components into distinct nodes, but it is more common to show them together as below.)

4.2 Neuron with Python Standard Libraries

This code does not include any “learning” (i.e., optimization), but it is worth showing just how simple it is to write a single neuron from scratch. Most of the code below is necessary only to create some faked input data.

   1 #!/usr/bin/env python
   2 
   3 from math import exp
   4 from random import gauss
   5 
   6 
   7 def sigmoid(z: float) -> float:
   8     """The sigmoid/logistic activation function."""
   9     return 1 / (1 + exp(-z))
  10 
  11 
  12 # The number of examples in our dataset
  13 N = 100
  14 
  15 # Randomly generate some input data
  16 nx = 4
  17 x1 = [gauss(0, 1) for _ in range(N)]
  18 x2 = [gauss(0, 1) for _ in range(N)]
  19 x3 = [gauss(0, 1) for _ in range(N)]
  20 x4 = [gauss(0, 1) for _ in range(N)]
  21 
  22 # Generate random neuron parameters
  23 w1 = gauss(0, 1)
  24 w2 = gauss(0, 1)
  25 w3 = gauss(0, 1)
  26 w4 = gauss(0, 1)
  27 b = 0
  28 
  29 # Compute neuron output for each of the N examples
  30 for x1i, x2i, x3i, x4i in zip(x1, x2, x3, x4):
  31     zi = w1 * x1i + w2 * x2i + w3 * x3i + w4 * x4i + b
  32     ai = sigmoid(zi)

In this code listing I use the sigmoid activation function (when not using a specific activation function we use \(g(\mathord{\cdot})\) in most equations). This function is plotted below.

Some nice properties of this function include:

• An output range of [0, 1] (all inputs are “squashed” into this range).
• An easy to compute derivative.
• Easy to interpret and understand.
• Well-known.

We often use sigmoid activation functions for binary classification (i.e., models trained to predict whether an input belongs to one of two classes). If the output is \(≤0.5\) we say the neuron predicts class \(A\) otherwise class \(B\).

4.3 The Dot-Product

We compute \(z\i\) using a summation, but we can express this same bit of math using the dot-product from linear algebra.

\[ z\i = \sum_{k=1}^{n_x} x_k\i w_k + b = \vx^{(i)T} \vw + b \]

The \(\vx^{(i)T} \vw\) part of the equation computes the dot-product between \(\vx^{(i)T}\) and \(\vw\). We need to transpose \(\vx\i\) to make the dimensions work (i.e., we need to multiply a row vector by a column vector).

This not only turns out to be easier to write/type, but it is more efficiently computed by a neural network library. The code listing below uses PyTorch to compute \(z\i\) (zi). Libraries like PyTorch and Tensorflow make use of both vectorized CPU instructions and graphics cards (GPUs) to quickly compute the output of matrix multiplications.

 1  │ 1  │#!/usr/bin/env python
 2  │ 2  │
 3  │    │from math import exp
 4  │    │from random import gauss
 5  │    │
 6  │    │
 7  │    │def sigmoid(z: float) -> float:
 8  │    │    """The sigmoid/logistic activation function."""
 9  │    │    return 1 / (1 + exp(-z))
    │ 3  │import torch
 10 │ 4  │
 11 │ 5  │
 12 │ 6  │# The number of examples in our dataset

 14 │ 8  │
 15 │ 9  │# Randomly generate some input data
 16 │ 10 │nx = 4
 17 │    │x1 = [gauss(0, 1) for _ in range(N)]
 18 │    │x2 = [gauss(0, 1) for _ in range(N)]
 19 │    │x3 = [gauss(0, 1) for _ in range(N)]
 20 │    │x4 = [gauss(0, 1) for _ in range(N)]
    │ 11 │X = torch.randn(N, nx)
 21 │ 12 │
 22 │ 13 │# Generate random neuron parameters
 23 │    │w1 = gauss(0, 1)
 24 │    │w2 = gauss(0, 1)
 25 │    │w3 = gauss(0, 1)
 26 │    │w4 = gauss(0, 1)
    │ 14 │w = torch.randn(nx)
 27 │ 15 │b = 0
 28 │ 16 │
 29 │ 17 │# Compute neuron output for each of the N examples
 30 │    │for x1i, x2i, x3i, x4i in zip(x1, x2, x3, x4):
 31 │    │    zi = w1 * x1i + w2 * x2i + w3 * x3i + w4 * x4i + b
 32 │    │    ai = sigmoid(zi)
    │ 18 │for xi in X:
    │ 19 │    zi = xi @ w + b
    │ 20 │    ai = torch.sigmoid(zi)

The code snippet above shows a diff between the previous code snippet and an updated one using the dot product. You will see many diffs throughout this document. The key points are that: (1) red indicates text or entire lines that have been removed and (2) green indicates updated or newly added lines.

We do not need to transpose xi in code because when we iteration through X we get row vectors. As it happens, we can improve efficiency even further.

4.4 Vectorizing Inputs

In addition to using a dot-product in place of a summation, we can use a matrix multiplication in place of looping over all examples in the dataset. In the two equations below we perform a matrix multiplication that computes the output of the network for all examples at once. A neural network library can turn this into highly efficient CPU or GPU operations.

\[\begin{align} \vz &= X \vw + \mathbf{1} b \\ \va &= g(\vz) \end{align}\]

 15 │ 15 │b = 0
 16 │ 16 │
 17 │ 17 │# Compute neuron output for each of the N examples
 18 │    │for xi in X:
 19 │    │    zi = xi @ w + b
 20 │    │    ai = torch.sigmoid(zi)
    │ 18 │z = X @ w + b
    │ 19 │yhat = torch.sigmoid(z)

In the code snippet above, a matrix multiplication is indicated in PyTorch using the @ symbol (a * is used for element-wise multiplications). A key to understanding matrix math is to examine the shapes of all matrices involved. Above, \(X\) has a shape of \((N \times n_x)\), \(\vw\) has a shape of \((n_x \times 1)\), and \(b\) is a scalar multiplied by an appropriately-shaped matrix of all ones (so that we can add \(b\) to each element of the \(X\vw\) result). Inner dimensions (the last dimension of the left matrix and the first dimension of the right matrix) must be the same for any valid matrix multiplication.

In the code snippet, the scalar \(b\) is added element-wise to every element in the final matrix due to broadcasting (this is a common library feature, not necessarily standard linear algebra).

So far, we have random parameters and we ignore the output. But what if we want to train the neuron so that the output mimics a real function or process? The next subsection tackles this very problem.

4.5 Optimization with Batch Gradient Descent

We must find values for parameters \(\vw\) and \(b\) to make \(\yhat\i \approx y\i\). As you might expect from the title of this subsection, we are going to use gradient descent to optimize the parameters. This means that we are going to need an objective function (something to minimize) and to compute some derivatives.

But what is an appropriate objective function (I’ll refer to this as the loss function going forward)? How about the mean-difference?

\[ℒ(\vyhat, \vy) = \sum_{i=1}^N (\yhat\i - y\i) \quad \color{red}{\text{Don't use this loss function.}}\]

A quick “fix” for the above loss function is to change it into the mean-absolute-error (MAE):

\[\begin{align} ℒ(\vyhat, \vy) &= \sum_{i=1}^N |\yhat\i - y\i|\\ &= \mae \end{align}\]

The second line shows a vectorized version using the L1-norm, which is the sum of the absolute values of the given vector. MAE is a good choice if your dataset includes outliers. MAE is also simple to interpret: it is the average deviation between your models guess and the correct answer.

A common choice for a loss function when training a regression model is Half mean-square-error (Half-MSE):

\[\begin{align} ℒ(\vyhat, \vy) &= \frac{1}{2N} \sum_{i=1}^N (\yhat\i - y\i)^2\\ &= \vhmse \end{align}\]

We are again using the L1-norm, but this time the vector we are norming is the element-wise squared values of the difference between the vectors \(\vyhat\) and \(\vy\). Interpreting Half-MSE is a bit harder than MAE—you should multiply the result by two and then take the square-root.

The standard choice when performing classification with a neuron is binary cross-entropy (BCE):

\[\begin{align} ℒ(\vyhat, \vy) &= \vbce\\ &= -\text{mean}_0\left(\vy \cdot \log{\vyhat} + (1 - \vy) \cdot \log{(1 - \vyhat)}\right) \end{align}\]

In the vectorized version of BCE, I’ve used the non-standard \(\text{mean}_0\) notation to indicate that we’re taking the average across the rows, dimension zero. This is closer to the code that you’ll actually write. Very rarely will you want to put a summation loop in your code.

Let’s move forward using binary cross-entropy loss and the sigmoid activation function.

We can only reduce loss by adjusting parameters. It doesn’t make sense, for example, to minimize loss by changing the input values \(X\) or the output targets \(Y\). Take a look at the following fictitious loss landscape.

The diagram above shows a curve for loss as a function of a single parameter, \(w_k\). For this figure, we’ll momentarily ignore that we might have dozens (or thousands or millions) of parameters. We want to find a new value for \(w_k\) such that loss is reduced. You might wonder why I said “loss is reduced” instead of “loss is minimized.” You might be familiar with techniques for finding an exact answer using an analytical (aka closed-form) solution.

For complex models, such as a neural network, analytical solutions are sometimes too slow or complicated to compute. Instead, we use an iterative (aka numerical) solution. You can think of numerical solutions as finding a good enough approximate solution as opposed to the exact correct solution. Surprisingly, the numerical solution is often more general than the exact solution—we’ll discuss this in later sections.

To determine how we should adjust parameters, we start the same way as finding the exact location and take the partial derivative of loss with respect to each parameter. Taking the single neuron, binary cross-entropy loss, and the sigmoid activation function the chain rule in matrix form is as follows.

\[\begin{align} \frac{∂ ℒ}{∂ \vw} &= \frac{∂ ℒ}{∂ \vyhat} \frac{∂ \vyhat}{∂ \vz} \frac{∂ \vz}{∂ \vw} \\ &= \frac{1}{N} X^T (\vyhat - \vy)\\[20pt] \frac{∂ ℒ}{∂ b} &= \frac{∂ ℒ}{∂ \vyhat} \frac{∂ \vyhat}{∂ \vz} \frac{∂ \vz}{∂ b} \\ &= \frac{1}{N} \sum_{i=1}^N (\yhat\i - y\i) \end{align}\]

Don’t worry too much about the derivations or notation at this stage. We’ll go into more details when we follow the same process for a full neural network in the next section.

With the two update equations shown in the previous answer we have everything we need to train our neuron model. Looking at these two equations you might wonder about the purpose of \(η\) (i.e., the “learning rate”). This factor enables us to tune how fast or slow we learn. If \(η\) is set too high we might not be able to learn, and it it is set too low we might learn prohibitively slowly. We will go into more details on optimization in sec. 7.

4.6 Neuron Batch Gradient Descent

Here is a complete example in which we train a neuron to classify images as either being of the digit 1 or the digit 7. Data processing details are hidden in the get_binary_mnist_one_batch function, but you can find that code in the repository for this guide.

   1 #!/usr/bin/env python
   2 
   3 from utilities import get_binary_mnist_one_batch, format_duration_with_prefix
   4 from timeit import default_timer as timer
   5 import torch
   6 
   7 
   8 def compute_accuracy(yhat, y):
   9     valid_N = y.shape[0]
  10     return 1 - (torch.round(yhat) - y).abs().sum() / valid_N
  11 
  12 
  13 # Get training and validation data loaders for classes A and B
  14 data_dir = "../../Data"
  15 classA, classB = 1, 7
  16 flatten = True
  17 train_X, train_y, valid_X, valid_y = get_binary_mnist_one_batch(
  18     data_dir, classA, classB, flatten
  19 )
  20 
  21 # Neuron parameters
  22 nx = 28 * 28
  23 w = torch.randn(nx) * 0.01
  24 b = torch.zeros(1)
  25 
  26 # Batch gradient descent hyper-parameters
  27 num_epochs = 4
  28 learning_rate = 0.01
  29 
  30 # Compute initial accuracy (should be around 50%)
  31 valid_yhat = torch.sigmoid(valid_X @ w + b)
  32 valid_accuracy = compute_accuracy(valid_yhat, valid_y)
  33 print(f"Accuracy before training: {valid_accuracy:.2f}")
  34 
  35 # Learn values for w and b that minimize loss
  36 for epoch in range(num_epochs):
  37 
  38     start = timer()
  39 
  40     # Make predictions given current paramters and then compute loss
  41     yhat = torch.sigmoid(train_X @ w + b)
  42     losses = -(train_y * torch.log(yhat) + (1 - train_y) * torch.log(1 - yhat))
  43 
  44     # Compute derivatives for w and b (dz is common to both derivatives)
  45     dz = yhat - train_y
  46     dw = (1 / train_y.shape[0]) * (dz @ train_X)
  47     db = dz.mean()
  48 
  49     # Update parameters
  50     w -= learning_rate * dw
  51     b -= learning_rate * db
  52 
  53     # Report on progress
  54     valid_yhat = torch.sigmoid(valid_X @ w + b)
  55     valid_accuracy = compute_accuracy(valid_yhat, valid_y)
  56 
  57     info = f"{epoch+1:>2}/{num_epochs}"
  58     info += f", Loss={losses.mean():0.1f}"
  59     info += f", Accuracy={valid_accuracy:.2f}"
  60     info += f", Time={format_duration_with_prefix(timer()-start)}"
  61     print(info)

Accuracy before training: 0.54
 1/4, Loss=0.7, Accuracy=0.97, Time=5.5 ms
 2/4, Loss=0.5, Accuracy=0.96, Time=4.8 ms
 3/4, Loss=0.4, Accuracy=0.96, Time=4.6 ms
 4/4, Loss=0.3, Accuracy=0.96, Time=4.4 ms

5.1 Vectorized Equations For a Neural Network

Let’s start with showing the notation for parameters from any layer \(l = 1, 2, ..., L\) where \(L\) is the number of layers in the network.

\[\begin{align} W^{[l]} &= \begin{bmatrix} w_{1,1}^{[l]} & w_{1,2}^{[l]} & \cdots & w_{1,n_{l-1}}^{[l]} \\ w_{2,1}^{[l]} & w_{2,2}^{[l]} & \cdots & w_{2,n_{l-1}}^{[l]} \\ \vdots & \vdots & \ddots & \vdots \\ w_{n_l,1}^{[l]} & w_{n_l,2}^{[l]} & \cdots & w_{n_l,n_{l-1}}^{[l]} \end{bmatrix} \\ \vb^{[l]} &= \begin{bmatrix} b_{1}^{[l]} \\ b_{2}^{[l]} \\ \vdots \\ b_{n_l}^{[l]} \\ \end{bmatrix} \end{align}\]

Compare these equations to the diagram above. Notice how the top neuron in layer 1 would have its associated parameters in the first row of \(W^{[1]}\) and the first value in \(\vb^{[1]}\).

Next we have the vectorized linear and activation equations for each neuron in a layer (these are for all training examples):

\[\begin{align} Z^{[l]} &= A^{[l-1]} W^{[l]T} + \mathbf{1} \vb^{[l]T}\\ A^{[l]} &= g^{[l]}(Z^{[l]}) \end{align}\]

import torch
N, nl = 10, 4
b = torch.randn(nl, 1)
ONE = torch.ones(N, 1)
print(ONE @ b.T)

5.2 Backpropagation

Just like for the single neuron, we want to find values for \(W^{[l]}\) and \(\vb^{[l]}\) (for \(l = 1, 2, ..., L\)) such that \(A^{[L]} \approx Y\) (\(A^{[L]}\) is another name for \(\hat Y\)). Instead of looking at a more general case, let’s work through gradient descent for the two-layer network above where we

• have three inputs (\(n_x=n_0=3\)),
• have two neurons in layer 1 (\(n_1=2\)),
• have three neurons in layer 2 (\(n_y=n_2=3\)),
• are using sigmoid activations for all neurons, and
• are using the binary-cross-entropy (BCE) loss function.

You can imagine that we are performing multi-label classification. For this network, we need to compute these partial derivatives:

\[ \frac{∂ℒ}{∂ W^{[1]}}^①, \frac{∂ℒ}{∂ \vb^{[1]}}^②, \frac{∂ℒ}{∂ W^{[2]}}^③, \frac{∂ℒ}{∂ \vb^{[2]}}^④ \]

We are going to start at layer 2 and work backward through the network to layer 1. As we compute these derivatives answer for yourself “why do we work backward through the network?”

This process of computing derivatives backward through the network is why this process if referred to as backpropagation–we’ll compute values and propagate them backward to earlier layers in the network. This is easier to see when viewing the compute graph. A compute graph depicts the flow of activations (during the forward pass) and gradients (during the backward pass) through the network.

Notice how the input flows forward from top-to-bottom in the compute graph, but gradients flow backward (from bottom-to-top). This image corresponds to the network above if you rotate it 90 degrees anti-clockwise (mostly just so we I had space for the image on this page).

5.3 Neuron Batch Gradient Descent

Let’s put this together into an example similar to that shown in the single neuron.

   1 #!/usr/bin/env python
   2 
   3 """
   4 TODO(AJC): 
   5 - Change X,y
   6 """
   7 
   8 from timeit import default_timer as timer
   9 
  10 import torch
  11 
  12 from utilities import format_duration_with_prefix, get_binary_mnist_one_batch
  13 
  14 
  15 def compute_accuracy(prediction, target):
  16     valid_N = target.shape[0]
  17     return 1 - (torch.round(prediction) - target).abs().sum() / valid_N
  18 
  19 
  20 # Get training and validation data loaders for classes A and B
  21 data_dir = "../../Data"
  22 classA, classB = 1, 7
  23 flatten = True
  24 train_X, train_y, valid_X, valid_y = get_binary_mnist_one_batch(
  25     data_dir, classA, classB, flatten
  26 )
  27 
  28 # Neural network layer sizes for MNIST
  29 n0 = 28 * 28
  30 n1 = 2
  31 n2 = 1
  32 
  33 # Network parameters
  34 W1 = torch.randn(n1, n0)
  35 b1 = torch.randn(n1)
  36 W2 = torch.randn(n2, n1)
  37 b2 = torch.randn(n2)
  38 
  39 
  40 def model(A0):
  41     Z1 = A0 @ W1.T + b1
  42     A1 = torch.sigmoid(Z1)
  43     Z2 = A1 @ W2.T + b2
  44     A2 = torch.sigmoid(Z2)
  45     return Z1, A1, Z2, A2.squeeze()
  46 
  47 
  48 # Batch gradient descent hyper-parameters
  49 num_epochs = 4
  50 learning_rate = 0.01
  51 
  52 # Compute initial accuracy (should be around 50%)
  53 _, _, _, valid_preds = model(valid_X)
  54 valid_accuracy = compute_accuracy(valid_preds, valid_y)
  55 print(f"Accuracy before training: {valid_accuracy:.2f}")
  56 print(valid_preds.shape)
  57 print(valid_y.shape)

5.4 Automatic Differentiation

Let’s agree we should avoid computing those derivatives by hand. The process is time consuming and error prone. Instead let’s rely on a technique known as automatic differentiation, which is built-in to PyTorch and most machine learning frameworks.

An automatic differentiation library:

1. Creates a compute graph from your tensor operations.
2. Performs a topological sort on the compute graph.
3. Compute gradients and back propagates them to all matrices.

Let’s take a look at an example.

   1 #!/usr/bin/env python3
   2 
   3 import torch
   4 
   5 N, n0, n1, n2 = 20, 10, 7, 13
   6 
   7 # Some fake input
   8 A0 = torch.randn(N, n0)
   9 Y = torch.randn(N, n2)
  10 
  11 # Layer 1
  12 W1 = torch.randn(n1, n0, requires_grad=True)
  13 b1 = torch.randn(n1, requires_grad=True)
  14 temp1 = A0 @ W1.T
  15 Z1 = temp1 + b1
  16 A1 = torch.sigmoid(Z1)
  17 
  18 # Layer 2
  19 W2 = torch.randn(n2, n1, requires_grad=True)
  20 b2 = torch.randn(n2, requires_grad=True)
  21 temp2 = A1 @ W2.T
  22 Z2 = temp2 + b2
  23 A2 = torch.sigmoid(Z2)
  24 
  25 # Loss and backward propagation
  26 temp3 = A2 - Y
  27 temp4 = temp3 ** 2
  28 loss = temp4.mean()
  29 loss.backward()

Two key points from the listing above, (1) requires_grad=True tells PyTorch to create the compute graph and compute partial derivatives with respect to the given tensor, and (2) I’ve ensured that each line of code contains a single operation, which makes it easier to match with the diagram below (I’ve provided this one in a bit more detail).

This diagram is (often) constructed dynamically as operations are performed. Edges indicate the flow of gradients in the backward direction. We start at graph source nodes (e.g., the “loss” node) and compute partial derivatives with respect to their inputs until we reach graph sinks (e.g., parameters). This diagram (and the corresponding code) map directly to the hand-computed derivatives from the previous section. Take some time and see if you can see how they map to one another.

If you’d like to see how an automatic differentiation library is coded, please take a look at my simple Match library, which tries to closely mimic the PyTorch interface.

5.5 Why “Deep” Neural Networks?

5.6 The Role of an Activation Function

6.1 Batch Gradient Descent

All examples thus far have used batch gradient descent (BGD). All gradient descent methods are iterative, meaning we continually make small changes to the parameters until we are satisfied or run out of time. BGD looks something like this:

for each epoch
    1. compute gradient with respect to all examples
    2. average gradients across all examples
    3. update parameters using averaged gradients

In all variants of gradient descent, an epoch refers to the process by which we update the parameters with respect to all training examples. In batch gradient descent, we compute all gradients at once and average them across all examples, resulting in the parameters being updated a single time each epoch. This has the advantage of smoothing out the affect of any outliers and leveraging the parallel nature of modern CPUs and GPUs. On the other hand, it can be a waste of resources (mainly time) to only update the parameters once each epoch.

6.2 Stochastic Gradient Descent

In stochastic Gradient Descent (SGD) we update parameter \(N\) times per epoch—once per example. This means that we update parameters more frequently than in BGD.

The stochastic part of SGD refers to a random shuffling of the example each epoch. This tends to reduce loss “cycling” where some sequence of repeated example increases and then decreases loss.

for each epoch
    randomly shuffle all examples
    for each example
        1. compute gradient with respect to single example
        2. update parameters using gradient

Although we update the parameters more frequently, not all updates are good. Outliers will make the model perform worse in the general case. Moreover, SGD does not take advantage of parallel computations.

6.3 Mini-Batch Stochastic Gradient Descent

Mini-Batch SGD provides a middle ground. We chunk the input into some number of batches and take the average gradient over each batch.

for each epoch
    randomly distribute examples into batches
    for each batch
        1. compute gradient with respect to all examples in batch
        2. average gradients across all examples in batch
        3. update parameters using averaged gradients

This enables us to get the best of both worlds:

• less susceptible to outliers and noise,
• a good number of updates per epoch, and
• good utilization of computing resources.

The code-diff below shows how few changes are needed to convert our BGD example into Mini-Batch SGD.

7.1 Momentum

7.2 RMSProp

7.3 Adam

7.4 AMSGrad