Lecture 23: Lab 4

# Artificial Intelligence

### Lab 4

---

# Lab 4

* In Lab 4 you will work with Neural Networks in PyTorch

* There are two parts:
   
   - Classification
   
   - Generative Adversarial Networks 
   
You will use the classifier from the first part in the second part!

---

# Lab 4

* You can develop locally using [this python file](/CS4200/assets/lab4.py)

* Install pytorch according to the [instructions](https://pytorch.org/get-started/locally/) (if you are unsure: select Stable, pip, CPU, or CUDA10.2 if you have an NVidia card)

* Alternatively, you can do the lab on Google Colaboratory using [this sheet](https://colab.research.google.com/drive/1XIF19zKKm67h5UKyT_Igr7auHu4YiUYw?usp=sharing)

* If you use Colab, you still have to submit a .py file (File -> Download -> Download .py)!

---

# MNIST

---

# Framework

When you open the .py file (or the colab notebook), you will see that there are several functions

* `show_image` allows you to show or store image files

* `main` will load the data set, do some preprocessing and call your other functions

* `train_classifier` and `classify` are functions you implement in part 1

* `train_discriminator`, `train_generator` and `gan` are functions you implement in part 2

Places you need to edit are marked with "TODO n" where n goes from 1 to 10

---

# Lab 4

* In Lab 4 you will create a GAN to generate new pictures of handwritten digits

* Focus on only one digit (pick one: your birthday, last digit of your BroncoID, your favorite number, ...)

* We need two neural networks: A discriminator and a generator

* We'll start with a simple classifier (TODOs 1-5)

---

# Classifier: TODOs 1+2

* TODO 1: Pick a digit!

* If you run the program without doing TODO 1, you will get an exception

* TODO 2: Pick a number of iterations to train your classifier for

* To start with, pick something very low, like 5, until the rest of your code works

* Then increase the iterations to see if your network actually learns something

---

# Classifier: TODO 3

* TODO 3: implement the `class Discriminator(nn.Module):`

* The image has 28*28 pixels, so your network will have 784 inputs

* The output should be **one** value, between 0 (image is not of the chosen digit) and 1 (it is), use a Sigmoid output layer!

* To start, try one hidden layer with around 100-300 neurons with a LeakyReLU activation function in the hidden layer

---

# Classifier: TODO 4

* TODO 4: Implement `train_classifier`

```
for i in range(n0):
    # 1. reset gradients (from previous iteration) to zero
    
    # 2. Pass training examples through network 
    
    # 3. Calculate the loss
    
    # 4. Calculate the gradient by calling loss.backward()
    
    # 5. Perform an optimizer step
```

---

# Classifier: TODO 5

* TODO 5: Implement `classify`

* Instantiate the network and the optimizer

* Call `train_classifier`

* Evaluate your trained model on the validation set

* Calculate: Accuracy, precision, recall

* Also calculate how often each digit is most often mistaken for your chosen digit

---

# Lab 4

For comparison, reference classifier performance (not optimized!):

* precision: 92%

* recall: 91%

* accuracy: 98%

---

# GAN (TODOs 6-10)

* When you are happy with the classifier performance it is time to create the GAN

* For this, you will create a second network (TODO 7)

* You also need a slightly different function to train your classifier (TODO 8)

* The last two pieces are training the generator (TODO 9), and putting it all together (TODO 10)

---

# GAN: TODO 6

* TODO 6 is where you set the iteration values

* `n` is the number of overall iterations

* In each of these iterations you will train the discriminator and the generator

* `n1` is the number of iterations you train the discriminator for

* `n2` is the number of iterations you train the generator for

* Feel free to rename!

---

# GAN: TODO 7

* TODO 7: Define Generator Network

* It takes 100 inputs and produces 784 outputs (one for each pixel)

* The output values should be between 0 and 1 (sigmoid!)

* You can use the same structure as for the classifier (100-300 hidden units, LeakyReLU) for the hidden layer(s)

---

# GAN: TODO 8

* TODO 8: Implement `train_discriminator`

* This is very similar to `train_classifier` from before

* The main difference is that you will get **two** separate data sets: `x_true` and `x_false` and no (explicit) labels

* Instead, the classifier should predict `1` for each item in `x_true` and 0 for each item in `x_false`

* Use `torch.ones_like` to generate the target vectors

---

# GAN: TODO 9

* TODO 9: Implement `train_generator`

* Your training loop will once again very similar

* Sample inputs randomly from a normal distribution (`torch.randn`) and pass them to the generator network

* For the loss function first pass the generator output through the (trained) discriminator network, then pass the output from that to the `loss_fn`, where we **want our outputs to be 1** (i.e. we want the discriminator to say that the images are real)

* Then do a `backward` call, and optimizer step as normal

* Your generator optimizer will only have the generator parameters as its target and therefore will not change the discriminator!

---

# GAN: TODO 10

* TODO 10: implement `gan`

* This is the overall training loop

* In each iteration, you will first train the discriminator, and then the generator

* Then you generate some new fake images with the generator

* Keep an "episode memory" of fake images, to which you add newly generated fakes in every iteration and discard a few old ones

* **Always** pass a matrix to the networks; if you want to generate a single image, the matrix has one row and 100 columns, e.g. from `torch.randn((1,100))`

---

# Output

* In each iteration, save a few images so you can visually inspect the training process

* You may observe that all generated images are the same (mode collapse)

* Try to add some randomness (Dropout layers in the generator, input- or output noise on the discriminator) if that happens

* It's ok if you can not solve this issue

---

# How to tell if it's working

* Best indicator: Visually!

* You should also print the loss in each iteration of each training loop

* The loss for each network should gradually (**not** rapidly) decrease

* It should not be exactly 0, though (then you are in a degenerate situation)

---

# What to do if it's not working

* Make sure your tensors have the correct shape (matrices!)

* Make sure you use `loss_fn` (which is a `torch.nn.BCELoss()`) and **not** MSELoss!

* Lower the learning rate (divide by 2 or even 10)

* Vary the number of neurons and the activation functions (make sure your outputs are between 0 and 1, though)

---

# GAN: Sample Results

* Your implementation may produce images faster, I deliberately slowed down training here.

* Note: This is one image from each iteration, the others may look exactly the same, or very different

---

# Bonus Points

* Once generating MNIST images works, try other data sets: EMNIST (letters), KMNIST (hiragana), FashionMNIST (clothes)

* These data sets have exactly the same structure as MNIST, so it should just be a matter of replacing which data set is loaded

* Note: Digits are relatively forgiving, for more detail you may need to tweak your networks some more

* You can also try CIFAR, ImageNet or other data sets, but for that you will have to change more of the structure

* Another option for bonus points is doing something against mode collapse, like an entropy/variance based approach!