Lecture 20: Advanced AI

# Creación de Videojuegos

### Advanced AI

---

# Machine Learning

---

# Machine Learning

* The idea behind Machine Learning is to give some data/experiences to an algorithm, and it derives how to solve a problem on other, unseen instances of the same kind of data
  
  * Supervised Learning: The algorithm gets some inputs and outputs, and learns their relationship. 
  
  * Unsupervised Learning: The algorithm gets some inputs, and learns information about them.
  
  * Reinforcement Learning: The algorithm is given a world, some actions, and a reward function, and learns how to obtain rewards in the world. 
  
---

# Optimization

* The basis for many Machine Learning Algorithms is optimization
  
  * Optimization: Given a function `$f$` find the value for `$\vec{x}$` such that `$f(\vec{x})$` is minimal (or maximal)
  
  * For example: In a strategy game there are many units and buildings to build. Which order of combination of these units/buildings gives you the strongest army?
  
  * Often, the function can not be (cheaply) drawn or calculated for every point to find a minimum
  
---

# Optimization: Gradient Descent

* Remember the derivative of a function?
  
  * The derivative gives you the "change" of the function 
  
  * Idea: If we want the minimum, we move in the direction where the function values become less
  
  * Where do we start? Let's just pick something at random 
  
---

# Optimization: Gradient Descent
  
<img src="/CI-2700/assets/img/gradientdescent.png" width="100%"/>

---

# Optimization: Gradient Descent
  
<img src="/CI-2700/assets/img/gradientdescent3D.png" width="80%"/>

---

# Optimization: Gradient Descent

Gradient Descent has many "problems"

* Local minima 
  
  * Overshooting
  
  * Slow movement on plateaus 
  
  * etc.
  
---

# Function approximation

* What if we want to replicate something else?
  
  * For example: The player drives car races, and we want an AI player that has a similar driving style ("Drivatar")
  
  * Let's say our player is a function `$p(\vec{y})$`
  
  * We want a function `$\hat{p}(\vec{y})$` that is "a good approximation" of `$p(\vec{y})$`
  
  * Idea: Let's minimize the difference `$|\hat{p}(\vec{y}) - p(\vec{y})|$`
  
  * But ... we don't actually know `$p(\vec{y})$`?
  
---

# Training examples

* Say we have observed *some* values for `$p(\vec{y})$`
  
  * That means, we have a list  `$p(\vec{y_1}) = z_1, p(\vec{y_2}) = z_2, p(\vec{y_3}) = z_3, \ldots$`
  
  * We "just" want to know values of p between these values, e.g. `$p(\vec{y_?}) = ?$`
  
  * Easiest solution: Linear interpolation! 
  
  * But what if the function is something non-linear?
  
---

# Function approximation

* Imagine we have a quadratic function `$\hat{p}(y) = a y^2 + b$`
  
  * We can choose a and b to change the values of the function
  
  * We can add cubic terms, etc.: `$\hat{p}(y) = \sum_i a_i y^i$`
  
  * Then, we can use gradient descent *on the values for a* to find a version of this function that minimizes the error
  
  * But: Our input is actually a vector, and we might want to combine these values "somehow", too
  
---

# Neural Networks

* Instead of polynomials we can also compose our function of other units 
  
  * For example, somewhere inside our function we have a term like `$ h(\sum_i a_i y_i) $`
  
  * These terms correspond to "neurons", and h is the neuron's activation function 
  
  * If h has a derivative, we can calculate the gradient of the entire network
  
  * And if we can calculate the gradient, we can do gradient descent!
  
---

# Neural Networks

---

# Neural Networks in Games

* Neural Networks can be useful if you want to approximate an unknown function 
  
  * You can also use it to e.g. "blend together" levels (using the existing levels as the target function)
  
  * Training might be computationally expensive, though 
  
  * There's also the issue of control
  
---

# Clustering

* What if you have your players, and you just want to know if there are certain classes of playstyles
  
  * In other words, you want to group players together that play similarly
  
  * This is called *clustering*, because it detects clusters of inputs that are similar 
  
  * One approach: Choose k cluster centers, and assign each element to the closest cluster, then update cluster centers iteratively
  
---

# K-Means

---

# Reinforcement Learning

---

# Reinforcement Learning
 
  * There is a world/task, for example a game 
  
  * The agent performs some actions 
  
  * The observer gives the agent a reward (e.g. score) for the performance
  
  * The agent tries to maximize the reward
  
---

# Q-Learning
 
  * The agent stores the rewards for each action in each state in a table 
  
  * When an agent takes a step, they record the expected reward for the following state

* For example: If an agent tries to jump, they record which reward they expect to get in the future, after jumping

* There is a discount factor to value getting rewards sooner higher  
  
---

---

# Procedural Content Generation

---

# The cost of making games

---

# Procedural Content Generation

* Making games requires a lot of effort 
  
  * Apart from coding, there are also art assets, music, a story, etc. that need to be created 
  
  * Some of these creation processes can be (partially) automated, or assisted: Procedural Content Generation (PCG)
  
  * Two problems
  
     - Generating content is not always easy
     - Generating *good* content is usually even harder
     
---

# What can we generate?

* Levels
  
  * Enemies 
  
  * Quests 
  
  * Dialog
  
  * Narrative 
  
  * Plants 
  
  * 3D objects
  
  * Rules 
  
  * Games 
  
  
---

# Search-based Approaches
  
---

# Search-based approaches

* Remember pathfinding?
  
  * Sometimes we want to "find" a path that doesn't even exist yet 
  
  * For example, to dig a cave/maze 
  
  * Or to connect two cities with roads 
  
  * The cost function for this will involve some gameplay or aesthetic properties
  
---

# Search-based approaches: Road generation

---

# Search-based approaches

* So far, we've used actual geometry to search for paths
  
  * What if every node in our graph is a "level" for a game
  
  * Some levels are incomplete, bad, impossible to finish, etc.
  
  * Say we can connect two levels with their difference (e.g. adding a block in Super Mario)
  
  * If we know what a "good" level looks like, we can search through this graph

---

# Search-based approaches: Story generation

* We can also have the computer generate a story 
  
  * Stories are sequences of actions that reach a conclusion 
  
  * That sounds a lot like planning!
  
  * Idea: Author defines a "world" consisting of actions the actors can take, and a goal for the story
  
---

# Search-based approaches: Story generation

* Let's generate a heist!

* We have a bandit, a bank, a clerk, a safe, etc.
  
  * Goal: The bandit has the safe
  
  * Story: The clerk takes the safe and brings it to the bandit's house
  
  * Is that a good story?
  
---

# Search-based approaches: Story generation

* The problem is that the planner finds an efficient solution
  
  * Stories are not necessarily efficient
  
  * In particular, not every character always works towards the goal of the story 
  
  * Introduce intentions: Actors can only execute actions if they aid their own goal 
  
  * Introduce conflict: Actors can plan to execute actions, but other actors might interfere
  
  * There are many more extension, e.g. to add interactivity
  
---

# Solver-based approaches

* What if we know certain properties a "good" level/quest/art asset has to have?
  
  * If we phrase our level design problem in a logical way these properties become constraints in the formula 
  
  * There are tools that find assignments of logical variables to formulas that take constraints into account 
  
  * Such an assignment is then a level/quest/art asset 
  
---

# Example: CatSAT

* Say we want to generate a quest 
  
  * There are 3 types of quests: Retrieve, deliver, and kill 
  
  * There are 4 types of enemies: Bandits, another kingdom, orcs, and wolves
  
  * There are some constraints on the possible quests:
       - You can't retrieve an item from the wolves
       - No more than 2 types of enemies should be present in any quest 
       - The bandits do not work together with the other kingdom 
       - etc.

---

# Example: CatSAT

```C#
var p = new Problem("quest");
...
var retrieve = (Proposition)"retrieve";
...
var wolves = (Proposition)"wolves";
...
p.Inconsistent(wolves, retrieve);
...
p.AtMost(2, bandits, others, orcs, wolves);
...
p.Solve();

```

---

# Rewriting Systems
 
---

# Grammar-based PCG

Remember grammars?
  
$$
A \rightarrow B A C\\\\
A \rightarrow \varepsilon\\\\
B \rightarrow b\\\\
C \rightarrow c
$$

---

# Tracery

```JS
{
  "origin": "#greeting#, #noun#!",
  
  "greeting": ["Howdy", "Hello", 
               "Greetings", "What's up", 
               "Hey", "Hi"],
  
  "noun": ["world", "solar system", 
           "galaxy", "local cluster", 
           "universe"]
}
```

---

# Cheap Bots, Done Quick

<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">Germans even have a word for worrying about cats:  Katzensorge</p>— GermanWords (@words_german) <a href="https://twitter.com/words_german/status/1069297382268067841?ref_src=twsrc%5Etfw">December 2, 2018</a></blockquote>

---

# L-Systems

* Named after the Hungarian theoretical biologist and botanist Aristid Lindenmayer
  
  * Idea: Instead of words, a grammar generates plant growth
  
  * For example: F draws a line "forward", - turns left, + turns right, the brackets store and retrieve the current drawing position

$$
X \rightarrow F+[[X]-X]-F[-FX]+X\\\\
F \rightarrow FF
$$

---

# L-Systems

---

# Shape Grammars

* We can use a similar idea as L-Systems for 3D geometry 
  
  * In a grammar you start with a sentence and divide it into parts with meaning
  
  * In a Shape Grammar, you start with a block, and divide it into parts with meaning 
  
  * For example: 
     - Start with a cylinder, cut it into n slices radially
     - For each of the slices you cut out a part in the center and decorate the edges 
     - You now have a gothic window 
     
  * Note: If you do this well, you can change n, the cutout shape, and the decoration arbitrarily

---

# Shape Grammars

---

# Midpoint Displacement

* Say we want to generate some (2D) hills 
  
  * We draw a straight line to begin with
  
  * Then we cut the line in half, and add some random noise to the center pointer
  
  * Afterwards, we recursively perform the same operation on the two new line segments
  
---

# Midpoint Displacement

---

# Midpoint Displacement

```JavaScript
var displace = function (a,b, size)
{
    if (a == b || (a+1) == b) return;

var mid = a/2 + b/2;

var x = geometry.vertices[mid].x;
    var y = (geometry.vertices[a].y + geometry.vertices[b].y)/2;
    var z = geometry.vertices[mid].z;

geometry.vertices[mid] = new THREE.Vector3( x, 
                              y + size*(Math.random() - 0.5), 
                              z);

displace(a, mid, size/2);
    displace(mid, b, size/2);

}
```

---

# Midpoint Displacement

---

# Challenges with Procedurally Generated Content

---

# Challenges

* We've seen a few ways to generate content 
  
  * How do we ensure that the content is good?
  
  * "Good" depends on the game, and the content 
  
  * If a random tree doesn't look perfect it's probably less bad than if a generated level is unwinnable
  
  * On the other hand: How many different things can we generate? (Expressive Range)

---

# Overenthusiastic generation

(Keyforge)

---

# References

* [All You Need to Know about Gradient Descent](https://hackernoon.com/gradient-descent-aynk-7cbe95a778da)
  
  * [Drivatars in Forza Motorsport](https://www.youtube.com/watch?v=twI0RSVwnR0)
  
  * [Artificial Intelligence and Games](http://gameaibook.org/) by Georgios N. Yannakakis and Julian Togelius, chapters 2.5-2.7. Free PDF available on the website!
  
  * [Experimental AI in Games Workshop](http://www.exag.org/)
  
---

# References
  
  * [Procedural Generation of Roads](https://onlinelibrary.wiley.com/doi/abs/10.1111/j.1467-8659.2009.01612.x)
  
  * [CatSAT](https://github.com/ianhorswill/CatSAT)
  
  * [Tracery Tutorial](http://air.decontextualize.com/tracery/)
  
  * [Cheap Bots, Done Quick](https://cheapbotsdonequick.com/)
  
  * [German Words Bot Source](https://cheapbotsdonequick.com/source/words_german)
  
  * [Markov Chain Word Generator](http://www.samcodes.co.uk/project/markov-namegen/)
  
  * [Linear levels through n-grams](http://www.kmjn.org/publications/NgramLevels_MT14.pdf)
  
---

# References

* [PCG Book (Free PDF available)](http://pcgbook.com/) by Noor Shaker, Julian Togelius, and Mark J. Nelson
  
  * [Procedural Content Generation Workshop](https://2018.pcgworkshop.com/)
  
  * [PCG Wiki](http://pcg.wikidot.com/)
  
  * [Keyforge: Problematic Deck Names](https://junkee.com/keyforge-computer-generated-names-ban/183365)
  
  * [Games By Angelina: An AI that creates video games](http://www.gamesbyangelina.org/)