Lecture 21: Neural Networks in PyTorch

# Artificial Intelligence

### Neural Networks in PyTorch

---

# Machine Learning in Practice

<blockquote class="twitter-tweet"><p lang="en" dir="ltr">Machine learning pipelines <a href="https://t.co/5FpG3HrdW0">pic.twitter.com/5FpG3HrdW0</a></p>— AI Memes for Artificially Intelligent Teens (@ai_memes) <a href="https://twitter.com/ai_memes/status/1382374419666976771?ref_src=twsrc%5Etfw">April 14, 2021</a></blockquote>

---

# PyTorch

* PyTorch is a python library for Machine Learning

* At every major Machine Learning conference in recent years, the majority of papers used PyTorch

* The other "big one" is Tensorflow, which is a bit older and has therefore more adoption in industry

* PyTorch is easier to get started with, and once you know the core concepts you can easily pick up Tensorflow, too

---

# PyTorch vs. Tensorflow

---

# Book

(<a href="https://pytorch.org/assets/deep-learning/Deep-Learning-with-PyTorch.pdf">Free PDF available</a>)

---

# What is PyTorch

* "Vector" math

* Automatic differentiation

* Gradient descent

* Some useful miscellaneous stuff

---

# Vector and Matrices and Tensors, oh my!

* Vector: An ordered list of numbers

* Matrix: A grid of numbers

* We could store a matrix in a vector, if we remember the dimensions!

* **Tensor**: What if we "need" more dimension?

* For example: We have 10000 images of 28x28 pixels. We store them in a 10000x28x28 tensor

* In memory the pixels will still be stored sequentially, the tensor is really just a different "view" on the data

---

# PyTorch

* (Almost) everything is a `pytorch.Tensor`!

* These tensors really are just views into a sequential array of numbers

* Each tensor can also "remember" where it came from (e.g. if it is the result of an addition)

* This means, you can also view a tensor as a "tree" of computation, with the result at the top, and the inputs as the leaves

* You can also tell your tensors that the operations should be performed on the GPU

---

# PyTorch: Computation Graphs

---

# Tensors: Squeeze, Unsqueeze, View

---

# Tensors in PyTorch

* A Tensor consists of an array of memory and a way to look at that memory

* The **dimensionality** of a tensor defines how many "sizes" it has

* The **shape** of a tensor tells us how many elements exist in each dimension

* A tensor with shape `[24,1]` has a different shape than a tensor using the same data with shape `[12,2]`, but also from a tensor with shape `[24]`, or `[24,1,1]`

---

# Tensors in PyTorch

Let `x` be a tensor with shape `[12,2]`

* `x[0,0]` is the first element

* `x[0]` (or `x[0,:]`) is the first **row** (a tensor of shape `[2]`)

* `x[:,0]` is the first **column** (a tensor of shape `[12]`)

* `x.T` is a tensor with shape `[2,12]` (the transpose)

---

# Squeeze

Squeeze is used to *remove* one/all dimension(s) of size 1:

* If `x.shape` is `[12,2]`, `x.squeeze()` does *nothing*

* If `x.shape` is `[24,1]`, `x.squeeze()` produces a tensor of shape `[24]`

* If `x.shape` is `[24,1,1]`, `x.squeeze()` produces a tensor of shape `[24]`

* If `x.shape` is `[24,1,1]`, `x.squeeze(1)` produces a tensor of shape `[24,1]`

---

# Unsqueeze

Unsqueeze is used to *insert* a dimension of size 1:

* If `x.shape` is `[12,2]`, `x.unsqueeze(0)` produces a tensor of shape `[1,12,2]`

* If `x.shape` is `[12,2]`, `x.unsqueeze(1)` produces a tensor of shape `[12,1,2]`

* If `x.shape` is `[12,2]`, `x.unsqueeze(2)` produces a tensor of shape `[12,2,1]`

---

# View

View is used to convert the shape of a tensor to something "arbitrary" (with the same total number)

* If `x.shape` is `[12,2]`, `x.view(24)` produces a tensor of shape `[24]`

* If `x.shape` is `[24]`, `x.view((24,1))` produces a tensor of shape `[24,1]` (exactly like `x.unsqueeze(1)`)

* If `x.shape` is `[24]`, `x.view((2,3,4))` produces a tensor of shape `[2,3,4]`

* If `x.shape` is `[24,1]`, `x.view(24)` produces a tensor of shape `[24]` (exactly like `x.squeeze(1)`)

* If `x.shape` is `[12,2]`, `x.view((8,3))` produces a tensor of shape `[8,3]`

* If `x.shape` is `[12,2]`, `x.view((8,6))` produces an error

---

# View

**One** dimension passed to `view` can be `-1`. Because `view` knows how many elements there are in total, it will just put "the rest"

* If `x.shape` is `[12,2]`, `x.view(-1)` produces a tensor of shape `[24]`

* If `x.shape` is `[n]`, `x.view((n,-1))` produces a tensor of shape `[n,1]` (exactly like `x.unsqueeze(1)`)

* If `x.shape` is `[24]`, `x.view((2,-1,4))` produces a tensor of shape `[2,3,4]`

* If `x.shape` is `[24,1]`, `x.view(-1)` produces a tensor of shape `[24]` (exactly like `x.squeeze(1)`)

---

# Computation Tree

* You have two tensors `x` and `y`, and you calculate e.g. `z = x.dot(y)`

* `z` will/can remember where it "came from"

* You can then call `z.backward()` to calculate the gradients

* These gradients will be stored in `x.grad` (and `y.grad`)

* (If you want to do this manually, you have to set `x.requires_grad = True`)

---

# Some useful operations

* Dot product: `x.dot(y)`

* Element-wise comparison: `y > 0.5` produces a tensor with True/False for each element in `y`

* Selection: `y[y>0.5]` selects only the elements from `y` which are greater than `0.5`

* More selection: `x[y > 0.5]` to select the x-values that go with those y-values!

* Even more selection: `x[y > 0.5,:]` if `x` is a matrix to select the matching *rows*

---

# More useful operations

* `(y > 0.5).sum()` counts how many y-values are greater than 0.5 (`True` is `1`, `False` is `0`)

* `torch.zeros`, `torch.ones`, `torch.rand` and `torch.randn` to create tensors with a given shape, e.g. `torch.zeros((10,20))`

* You can also make tensors "like" (same shape) another: `torch.ones_like(y)` will make a tensor filled with `1`s with the same shape as `y`

* Concatenate tensors: `torch.cat((torch.randn(4,10), torch.randn(8,10)))` will merge them into a tensor of shape `[12,10]`

---

# Recipe: Draw samples without replacement

* Say you have a tensor `x` with the (oddly specific) shape `[200,784]`

* You want to select 100 random rows from that tensor!

```Python
# Generate random permutation of indices
idx = torch.randperm(x.size(0)) # or just "200"

# Use the first 100 of these indices 
select_idx = idx[:100]

# Select the corresponding rows
selected = x[select_idx, :]
```

---

# Neural Networks in PyTorch

---

# Artificial Neural Networks

$$
\vec{h} = f_1(W_1 \cdot \vec{x})\\\\
\hat{y} = f_2(\vec{w_2} \cdot \vec{h})\\\\
\hat{y} = f_2(\vec{w_2} \cdot f_1(W_1 \cdot \vec{x}))
$$
]

---

# Artificial Neural Networks

---

# Defining a Neural Network

* Our Neural Network consists of Layers

* Each layer consists of a linear part, and an activation function

* The output from one layer is passed as the input to the next layer

* In PyTorch, we can define such a network as a "sequence"

---

# Defining a Neural Network

```Python
model = torch.nn.Sequential(
  # number of inputs: 128*128*3 -> number of outputs: 100 
  torch.nn.Linear(128*128*3, 100),
  
  # Activation function
  torch.nn.Sigmoid(),

# number of inputs: 100 -> number of outputs: 1
  torch.nn.Linear(100,1)  
)

y_hat = model(x)
```

---

# Defining a Neural Network: Alternative

```Python
class TwoLayerNet(torch.nn.Module):
    def __init__(self, D_in, H, D_out):
        super(TwoLayerNet, self).__init__()
        self.linear1 = torch.nn.Linear(D_in, H)
        self.sig1 = torch.nn.Sigmoid()
        
        self.linear2 = torch.nn.Linear(H, D_out)

def forward(self, x):
        h = self.linear1(x)
        h = self.sig1(h)
        return self.linear2(h)

model = TwoLayerNet(128*128*3, 100, 1)
y_hat = model(x)
```

---

# Learning

* Once we have a neural network, we need to train it

* Training works by "showing" the network a training example, and comparing the network's output to the desired output

* Then we change the parameters/weights of the network

* Of course, PyTorch can do all of this (mostly) automatically for us!

---

# Learning: Loss Function

* Last time we talked about the "Mean Squared Error"

* PyTorch provides you with the class `torch.nn.MSELoss`

* You can use it to compute the loss as a 1x1 tensor

* You can then call the method `backward()` on this tensor to calculate the gradient of the loss!

---

# Learning: Optimizers

* An **Optimizer** is an algorithm that changes the network parameters depending on the loss/gradient

* PyTorch provides several optimizers, we will use `Adam` for most tasks

* `Adam` needs you to calculate the error/loss, and call `backward()`

* Then you call the `step()` method

---

# Learning

---

# Who is Adam?

* Adam is not an acronym, but it is derived from "Adaptive moment estimation"

* Remember when we talked about gradient descent, and how we might want to "speed up" our learning sometimes?

* That's the "momentum"

* Adam is generally robust and fast to find a solution

* You still have to set/tweak the initial learning rate

---

# Training a Neural Network

```Python
loss_fn = torch.nn.MSELoss()
# lr = Learning Rate: How fast the parameters change 
# Adam will change the learning rate over time!
optimizer = torch.optim.Adam(model.parameters(), lr=0.1)

for t in range(n):
  y_hat = model(x)
  loss = loss_fn(y_hat, y)
  
  # IMPORTANT: Reset the gradient to zero!
  optimizer.zero_grad()
  
  # Calculate the gradient 
  loss.backward()
  
  # Change the network parameters
  optimizer.step()
```

---

# Classification Using Neural Networks

* So far, we have looked at predicting numbers as outputs

* What if we just want to distinguish classes, like "cat" and "not cat"?

* Idea: Let's set a "threshold". If the output number is higher it means "cat", otherwise it means "not cat"

* Second idea: Let's limit our output to be between 0 and 1 by using a Sigmoid

---

# Training

* Now we train our neural network. We have images of cats and non-cats

* As the desired output we set 1 for all cats, and 0 for all non-cats

* Then we calculate the loss: MSE is not great for such a limited range, so we use something else

* `torch.nn.BCELoss` ("Binary Cross Entropy Loss")

---

# Defining the Classifier

```Python
model = torch.nn.Sequential(
  # number of inputs: 128*128*3 -> number of outputs: 100 
  torch.nn.Linear(128*128*3, 100),
  torch.nn.Sigmoid(),

# number of inputs: 100 -> number of outputs: 1
  torch.nn.Linear(100,1),
  # make sure the output is between 0 and 1
  torch.nn.Sigmoid()
)
```

---

# Code

```Python
# Binary Cross Entropy Loss
loss_fn = torch.nn.BCELoss()

optimizer = torch.optim.Adam(model.parameters(), lr=0.01)

for t in range(n):
  # IMPORTANT: Reset the gradient to zero!
  optimizer.zero_grad()

# Calling "backward()" twice *accumulates* the 
  # gradient of the two classes
  y_pred = model(cats)
  loss = loss_fn(y_pred, torch.ones_like(y_pred))
  loss.backward()
  
  y_pred = model(noncats)
  loss = loss_fn(y_pred, torch.zeros_like(y_pred))
  loss.backward()
  
  optimizer.step()
```

---

# Important Note

* Each `Tensor` has a `shape` attribute, check it if something goes wrong!

* A Neural Network (almost) always needs a **matrix** (2D-tensor) as input

* Say you want to predict the value for a new image, you have to use `img.unsqueeze(0)` first!

* If your error increases instead of decreasing, try a lower learning rate

---

# Odds and Ends

* `torch` comes with a companion module `torchvision`

* `torchvision.datasets` contains several popular data sets

* Of course, for your own applications you will want to load data

* `torch.from_numpy` allows you to convert a `numpy` array to a `Tensor`, using the same memory!

* Also neat: `xc = x.cuda()` **copies** your tensor to GPU memory; you also have to send your model to the GPU!

---

# References

* [Torch Tensor Operations Overview](https://jhui.github.io/2018/02/09/PyTorch-Basic-operations/)
  
  * [DeepLearning With PyTorch](https://pytorch.org/deep-learning-with-pytorch)
  
  * [Adam: A Method for Stochastic Optimization](https://arxiv.org/pdf/1412.6980.pdf)