Agents

• An agent is an entity that perceives its environment and acts on it

• Some/Many people frown upon saying that something is "an AI" and prefer the term "agent"

• Agents come in many different forms

  • Robots
  • Software
  • One could view humans or other living entities as agents

Braitenberg Vehicles

• Valentino Braitenberg proposed a thought experiment with simple two-wheel vehicles

• The vehicles had two light sensors, and there was a light in the room

• Each of the two sensors would be connected to one of the wheels

• Depending on how this was done, the vehicle would seek or flee from the light

• The behavior of the agent is fully reactive, with no memory

Braitenberg Vehicles

A first agent: An enemy in an ARPG

• Performance: How much damage it can do to the player?

• Environment: A dungeon in the game

• Actuators: Rotate, move forward, hit

• Sensors: Player position (With that we can compute distance and angle to the player)

Behavior for our ARPG agent

• If the angle to the player is greater than 0, turn left

• (else) If the angle to the player is less than 0, turn right

• (else) If the distance to the player is greater than 0, move forward

• (else) Hit the player

Limitations

• This is, of course, a very simple agent

• Imagine if there were walls

• What if we want the enemy to have different modes of engagement, flee when it is in danger, etc.?

• How did we even come up with these conditions?

• How could we make this a bit friendlier to edit?

Decision Trees

Decision Trees: Limitations

• We haven't actually changed anything from the if statements (other than drawing them)

• Designing a decision tree is still a lot of manual work

• There's also no persistence, the agent will decide a new behavior every time the tree is evaluated

• There is one nice thing: Decision trees can (sometimes) be learned with Machine Learning techniques

States?

• Say we want our enemy to attack more aggressively if they have a lot of health and try to flee when they become wounded

• In other words: The enemy has a state that determines what they do, in addition to their inputs and outputs

• But we'll need new sensors: The enemy needs to know their own health level

• Let's also give them a ranged weapon

Finite State Machines

• States represent what the agent is currently supposed to do

• Each state is associated with actions the agent should perform in that state

• Transitions between the states observe the sensors and change the state when a condition is met

• The agent starts in some designated state, and can only be in one state at a time

Finite State Machines

Hierarchical Finite State Machines

• Finite State Machines define the behavior of the agent

• But we said the nodes are behaviors?!

• We can make each node another sub-machine!

• This leads to some reusability, and eases authoring

Behavior Trees: Common Node types

• Choice/Selector: Execute children in order until one succeeds

• Sequence: Execute children in order until one fails

• Loop: Keep executing child (or children) until one fails

• Random choice: Execute one of the children at random

• etc.

Behavior Trees: How do you make an "if" statement?

• Some actions are just "checks", they return success iff the check passes

• A sequence consisting of a check and another node will only execute the second node if the check passes

• If we put multiple such sequences as children of a choice, the first sequence with a passing condition will be executed

Behavior Trees

Behavior Trees

• Behavior Trees are a very powerful technique and widely used in games

• Halo 2, for example, used them

• Unreal Engine has built-in support for Behavior Trees (there are plugins for Unity)

• The tree structure usually allows for visual editing (which Unreal Engine also has built-in)

Behavior Trees in Unreal Engine

Graphs

• A graph G = (V,E) consists of vertices (nodes) V and edges (connections) $E \subseteq V \times V$

• Graphs can be connected, or have multiple components

• Graphs can be directed (one-way streets) or undirected

• Edges can have weights (costs) associated with them: $w: E \mapsto \mathbb{R}$

• We can represent many things in graphs

The (undirected) Pathfinding problem

Given a graph G = (V,E), with edge weights w, a start node $s \in V$, a destination node $d \in V$, find a sequence of vertices $v_1, v_2, \ldots, v_n$, such that $v_1 = s, v_n = d$ and $\forall i: (v_i, v_{i+1}) \in E$

We call the sequence $v_1, v_2, \ldots, v_n$ a path, and the cost of the path is $\sum_i w((v_i,v_{i+1}))$

The (undirected) Pathfinding problem

Given a graph G = (V,E), with edge weights w, a start node $s \in V$, a destination node $d \in V$, find a sequence of vertices $v_1, v_2, \ldots, v_n$, such that $v_1 = s, v_n = d$ and $\forall i: (v_i, v_{i+1}) \in E$

We call the sequence $v_1, v_2, \ldots, v_n$ a path, and the cost of the path is $\sum_i w((v_i,v_{i+1}))$

This means what you would expect: To find a path from a start node to a destination node means to find vertices to walk through that lead from the start to the destination by being connected with edges. The cost is the sum of the costs of edges that need to be traversed.

Another example: Romania

How could we find a path?

Uninformed Search

Heuristic Search: General algorithm

• We use the same algorithm as above:

  • Keep track of which nodes are candidates for expansion (starting with the start node)
  • Take one of these nodes and expand it
  • If you reach the target, you have found a path

• Instead of using a stack or list, we use a priority queue, where the nodes are ordered according to some value derived from the heuristic

• So how do we determine this value?

Greedy Search

• Let's use our heuristic!

• We order the nodes in the priority queue by heuristic value

• Heuristic: straight line distance to Bucharest

A* Search

• Greedy search sometimes does not give us the optimal result

• It tries to get to the goal as fast as possible, but ignores the cost of actually getting to each node

• Idea: Instead of using the node with the lowest heuristic value, use the node with the lowest sum of heuristic value and cost to get to

• This is called A* search

Dijkstra's algorithm

• You may have heard of Dijkstra's algorithm (and its variants) before

• Dijkstra's algorithm is basically A* without using the heuristic

• In some popular formulations you also let the algorithm compute a path for every possible destination

• This will give you a shortest path tree, which may be useful if you have to repeatedly find a path to different destinations

Applications

• A* is widely applied in games

• Unity's built-in navigation module uses A*

• But how do you apply A* to a 3D world?

• We need a graph!

• Idea: Divide the game world into regions, and assign each region a graph node

Example: World of Warcraft

Example: World of Warcraft

Example: World of Warcraft

Example: World of Warcraft

Example: World of Warcraft

Utility

• Sometimes we can just assign a numerical value ("score") to the observations, and then combine these scores in some way to get a decision

• For example, we can assign a score to the distance from the player, the agent's health, maybe their remaining mana, etc.

• Then, we can calculate a score for a melee attack by weighing the distance as more significant than the health, and mana being irrelevant

• On the other hand, the score for a fireball would be more affected by the remaining mana and less by the distance (up to a threshold, perhaps)

• The agent then simply picks the action with the highest score/utility

Utility: Our ARPG enemy

Three options: melee, fireball or run away

$$u_m = 0.8 \cdot d + 0.2 \cdot h + 0 \cdot m \\ u_f = 0.4 \cdot d + 0.2 \cdot h + 0.4 \cdot m \\ u_r = 0.4 \cdot d + 0.6 \cdot h + 0 \cdot m$$

We are 80 units away from the player, have 90% health and 100% mana. What do we do?

Utility: Our ARPG enemy

Three options: melee, fireball or run away

$$u_m = 0.8 \cdot d + 0.2 \cdot h + 0 \cdot m \\ u_f = 0.4 \cdot d + 0.2 \cdot h + 0.4 \cdot m \\ u_r = 0.4 \cdot d + 0.6 \cdot h + 0 \cdot m$$

We are 80 units away from the player, have 90% health and 100% mana. What do we do?

At which distance would we attack the player?

Utility: Our ARPG enemy

Three options: melee, fireball or run away

$$u_m = 0.8 \cdot d + 0.2 \cdot h + 0 \cdot m \\ u_f = 0.4 \cdot d + 0.2 \cdot h + 0.4 \cdot m \\ u_r = 0.4 \cdot d + 0.6 \cdot h + 0 \cdot m$$

We are 80 units away from the player, have 90% health and 100% mana. What do we do?

At which distance would we attack the player?

We need to define the scores! Let's say

$$d = \frac{80}{\mathit{distance} + 80}, h = \frac{\mathit{health}}{100}, m = \frac{\mathit{mana}}{100}$$

Potential Fields

Potential Fields

Potential Fields

Flow Fields

Utility: Task Assignment

• To assign squads to targets, we calculate the utility of a particular assignment as the sum of all individual utilities

• For example, if squad 1 attacking target 1 has utility 0.4, and squad 2 attacking target 2 has utility 0.1, the total utility is 0.5

• We calculate these utilities for all possible assignments

• Then we pick the assignment with the highest total utility

Another Task Assignment Example

• Instead of squads of units we have students

• Instead of targets to attack we have papers to present

• And y'all sent me the utility values ...

• Papers were assigned to maximize total utility

Paper Task Assignment

u 1 1 =  1.0
u 1 3 =  0.66
u 1 6 =  0.33
u 1 _ = -1.0
utility :: [(Int,Int)] -> Double
utility assignments = sum $ map (uncurry u) assignments
makeAssignment :: [Int] -> [Int] -> [(Int,Int)]
makeAssignment students topics =
        maximumBy (comparing utility) assignments
     where 
        assignments = map (zip students) $ permutations topics

(Source)

See any problem?

Paper Task Assignment

u 1 1 =  1.0
u 1 3 =  0.66
u 1 6 =  0.33
u 1 _ = -1.0
utility :: [(Int,Int)] -> Double
utility assignments = sum $ map (uncurry u) assignments
makeAssignment :: [Int] -> [Int] -> [(Int,Int)]
makeAssignment students topics =
        maximumBy (comparing utility) assignments
     where 
        assignments = map (zip students) $ permutations topics

(Source)

See any problem? Here's a bad word for you: permutations

Adversaries?

• So far we have only looked at making decisions based on our own plans

• What if the other player can make several different decisions in response to our action?

• For example, how can we play chess, accounting for what the opponent will do?

• Adversarial search!

Minimax

Minimax

Minimax

Let's take a game where we "build" a binary number by choosing bits. The number starts with a 1, and each player can choose the next bit in order. The game ends when the number has 6 digits in total (after 5 choices), or if the same bit was chosen twice in a row. If the resulting number is even or prime, we get points equal to the number, otherwise the other player gets that many points. We want to know: What is our best first move assuming the other player plays optimally.

Alpha-beta Pruning

Paper Discussions

• What did we like about the paper?

• Which application can we see for the technique?

• Are there any assumptions the authors made that may need a second look?

• Are there any problems with the experiment?

• What are the limitations of the approach?

• How can the work be expanded upon?

For Next Week

• Read Human Enemy AI in The Last of US

• There is also a GDC talk

• We will discuss this paper next week!