Artificial Intelligence

Remember: Everything in AI is either representation or search

• Behavior Trees, State Machines, etc.: Representation

• Planning: Search (Plan-Space planning uses a different representation)

• Belief modeling: Representation improvements

• Monte Carlo Tree Search: Search

The Optimization Problem

Given a function (usually called "fitness function"):

$$f: \mathbb{R}^n \mapsto \mathbb{R}$$

find the $\vec{x} \in \mathbb{R}^n$ for which $f(\vec{x})$ is minimal (or maximal).

The Optimization Problem

Given a function (usually called "fitness function"):

$$f: \mathbb{R}^n \mapsto \mathbb{R}$$

find the $\vec{x} \in \mathbb{R}^n$ for which $f(\vec{x})$ is minimal (or maximal).

First idea:

$$\frac{\text{d}}{\text{d}x}f(\vec{x}) = 0$$

Challenges

• Setting the derivative to zero only finds a minimum/maximum, but not necessarily the global minimum/maximum

• The dimensionality of the vector may be really high, giving us many options to consider

• Who says the derivative can even be calculated?

• We haven't even specified what our vectors represent, and what f looks like

Representation

• We said our function takes vectors as real numbers as input, but what do these vectors represent?

• They can be geometrical, such as the angle and force used to shoot a bird in Angry Birds

• It could be an assignment of values to resources, such as how many of each unit to build in Starcraft

• Another possibility is to view the vector as a sequence of actions

Geometric Interpretation

Differentiability

To be differentiable, a function needs to be continuous:

$$\forall \varepsilon \gt 0\: \exists \delta \gt 0\: \forall \vec{a}:\\ (|\vec{a} - \vec{x}| \lt \delta \rightarrow |f(\vec{a}) - f(\vec{x})| \lt \varepsilon)$$

Or, in words:

If you change the input a little bit, the output also only changes a little bit.

Differentiability?

Is the function that maps from angle and strength to score differentiable?

Gradient Descent

• For now, let's say our function is differentiable

• We know that the derivative is zero at the extrema

• We also know that it defines the slope

• So if we start at any point, we can just go "downhill"

Gradient Descent

• Start with an initial guess $\vec{x}_0$

• Repeat until "convergence":

$$\vec{x}_{i+1} = \vec{x}_i + \alpha\cdot \frac{\text{d}}{\text{d}\vec{x}} f(\vec{x}_i)$$

• $\alpha$ is the learning rate

Gradient Descent

Gradient Descent

Gradient Descent: Some improvements

• Scale the learning rate over time to move quickly at first, and slow down as a solution is found

• Start at multiple different randomly chosen starting locations to avoid being stuck in a local minimum

• Use the learning rate as momentum to get out of small minima

• Add a small random number to the gradient to explore other parts of the function

Genetic Algorithm

• We generate a (random) set of individuals as a population

• Every step we create some new individuals as combinations and/or mutations of existing ones

• Then we keep only the "most fit" individuals, i.e. the ones for which our scoring function returns the lowest values

• Repeat for "a number of steps"

Population

• Our population is a set of vectors

• Each of these vectors is associated with its fitness

• We limit our population to a certain size by only keeping the n "best" individuals

Combination and Mutation

• Two individuals can be combined by selecting elements from each of their two vectors (or averaging, adding, etc.), called "crossover"

• An individual can also be changed by modifying single values in its vector, called "mutation"

• If we view our vector elements as containing a binary representation of numbers, we could also use binary operations such as an xor to combine two values

Why does this work?

• By keeping only the best individuals from one iteration to the next we guarantee that the solution will never get worse over time

• Using combinations of two individuals basically tries to combine the advantages of the two, while removing the drawbacks

• By having a number of different individuals we avoid getting stuck in local minima

Repair

• One problem is that in many domains not every possible vector is also feasible

• For example: If our vector represents actions, jumping while already in the air might not be possible

• Repair is the mechanism by which an infeasible individual is converted into a feasible one

• How this repair works is domain dependent

Swarm-Based Optimization

• Another approach is to keep a set of individuals and "move" them around in space

• The individuals can use information about the other individuals, or the "swarm" as a whole, to determine where to go

• There are several approaches to how this information exchange works

• Typically the individuals are initially spread out over the search space and then follow the path to the best known values

Particle-Swarm Optimization

Ant-Colony Optimization

Interpolation

• Say we know the win rate for having 40 Space Marines and 10 Fire Bats is 80%, and the win rate for having 10 Space Marines and 40 Fire Bats is 20%

• What is the win rate for 25 Space Marines and 25 Fire Bats?

• We can guess 50%, using Linear Interpolation

• Our fitness functions are not usually linear, though

Multi-Objective Optimization

• Sometimes we don't want to just optimize one value, but multiple

• Or the optimization of one value comes at the expense of another

• For example: Building units in a strategy game that do the most damage

• The most expensive units do the most damage

• That does not mean that building the most expensive units is the best strategy. There is a trade-off between cost and army strength

Multi-Objective Optimization: Approaches

• Maybe we can try to optimize the (weighted) sum of both values?

• Or how about a ratio? Let's optimize the army strength per resource spent

• Both of these may run into degenerate solutions. For example, spending 0 resources results in infinite strength per resource

• Better idea: Let the AI agent decide about the trade-off

Pareto-Optimal Solutions

• Say we have two values, x and y, to maximize

• If we have a solution with fitness (x,y), any other solution for which the fitness for x is higher and the fitness for y is higher, is an objectively worse solution. The first solution is said to dominate the other

• However, solutions for which x is higher or y is higher can be considered of the "same" utility

• The result will be a frontier of fitness values

Pareto-Optimal Solutions

Next Week

• Build order optimization in Starcraft