# Example Learning Environments

The Unity ML-Agents toolkit contains an expanding set of example environments
which demonstrate various features of the platform. Environments are located in
`MLAgentsSDK/Assets/ML-Agents/Examples` and summarized below. Additionally, our
[first ML Challenge](https://connect.unity.com/challenges/ml-agents-1) contains
environments created by the community.

This page only overviews the example environments we provide. To learn more on
how to design and build your own environments see our [Making a New Learning
Environment](Learning-Environment-Create-New.md) page.

Note: Environment scenes marked as _optional_ do not have accompanying
pre-trained model files, and are designed to serve as challenges for
researchers.

If you would like to contribute environments, please see our
[contribution guidelines](../CONTRIBUTING.md) page.

## Basic

![Basic](images/basic.png)

* Set-up: A linear movement task where the agent must move left or right to
  rewarding states.
* Goal: Move to the most reward state.
* Agents: The environment contains one agent linked to a single brain.
* Agent Reward Function:
  * +0.1 for arriving at suboptimal state.
  * +1.0 for arriving at optimal state.
* Brains: One brain with the following observation/action space.
  * Vector Observation space: One variable corresponding to current state.
  * Vector Action space: (Discrete) Two possible actions (Move left, move
    right).
  * Visual Observations: None.
* Reset Parameters: None
* Benchmark Mean Reward: 0.94

## [3DBall: 3D Balance Ball](https://youtu.be/dheeCO29-EI)

![3D Balance Ball](images/balance.png)

* Set-up: A balance-ball task, where the agent controls the platform.
* Goal: The agent must balance the platform in order to keep the ball on it for
  as long as possible.
* Agents: The environment contains 12 agents of the same kind, all linked to a
  single brain.
* Agent Reward Function:
  * +0.1 for every step the ball remains on the platform.
  * -1.0 if the ball falls from the platform.
* Brains: One brain with the following observation/action space.
  * Vector Observation space: 8 variables corresponding to rotation of platform,
    and position, rotation, and velocity of ball.
  * Vector Observation space (Hard Version): 5 variables corresponding to
    rotation of platform and position and rotation of ball.
  * Vector Action space: (Continuous) Size of 2, with one value corresponding to
    X-rotation, and the other to Z-rotation.
  * Visual Observations: None.
* Reset Parameters: None
* Benchmark Mean Reward: 100

## [GridWorld](https://youtu.be/gu8HE9WKEVI)

![GridWorld](images/gridworld.png)

* Set-up: A version of the classic grid-world task. Scene contains agent, goal,
  and obstacles.
* Goal: The agent must navigate the grid to the goal while avoiding the
  obstacles.
* Agents: The environment contains one agent linked to a single brain.
* Agent Reward Function:
  * -0.01 for every step.
  * +1.0 if the agent navigates to the goal position of the grid (episode ends).
  * -1.0 if the agent navigates to an obstacle (episode ends).
* Brains: One brain with the following observation/action space.
  * Vector Observation space: None
  * Vector Action space: (Discrete) Size of 4, corresponding to movement in
    cardinal directions.
  * Visual Observations: One corresponding to top-down view of GridWorld.
* Reset Parameters: Three, corresponding to grid size, number of obstacles, and
  number of goals.
* Benchmark Mean Reward: 0.8

## [Tennis](https://youtu.be/RDaIh7JX6RI)

![Tennis](images/tennis.png)

* Set-up: Two-player game where agents control rackets to bounce ball over a
  net.
* Goal: The agents must bounce ball between one another while not dropping or
  sending ball out of bounds.
* Agents: The environment contains two agent linked to a single brain named
  TennisBrain. After training you can attach another brain named MyBrain to one
  of the agent to play against your trained model.
* Agent Reward Function (independent):
  * +0.1 To agent when hitting ball over net.
  * -0.1 To agent who let ball hit their ground, or hit ball out of bounds.
* Brains: One brain with the following observation/action space.
  * Vector Observation space: 8 variables corresponding to position and velocity
    of ball and racket.
  * Vector Action space: (Continuous) Size of 2, corresponding to movement
    toward net or away from net, and jumping.
  * Visual Observations: None.
* Reset Parameters: One, corresponding to size of ball.
* Benchmark Mean Reward: 2.5
* Optional Imitation Learning scene: `TennisIL`.

## [Push Block](https://youtu.be/jKdw216ZgoE)

![Push](images/push.png)

* Set-up: A platforming environment where the agent can push a block around.
* Goal: The agent must push the block to the goal.
* Agents: The environment contains one agent linked to a single brain.
* Agent Reward Function:
  * -0.0025 for every step.
  * +1.0 if the block touches the goal.
* Brains: One brain with the following observation/action space.
  * Vector Observation space: (Continuous) 70 variables corresponding to 14
    ray-casts each detecting one of three possible objects (wall, goal, or
    block).
  * Vector Action space: (Continuous) Size of 2, corresponding to movement in X
    and Z directions.
  * Visual Observations (Optional): One first-person camera. Use
    `VisualPushBlock` scene.
* Reset Parameters: None.
* Benchmark Mean Reward: 4.5
* Optional Imitation Learning scene: `PushBlockIL`.

## [Wall Jump](https://youtu.be/NITLug2DIWQ)

![Wall](images/wall.png)

* Set-up: A platforming environment where the agent can jump over a wall.
* Goal: The agent must use the block to scale the wall and reach the goal.
* Agents: The environment contains one agent linked to two different brains. The
  brain the agent is linked to changes depending on the height of the wall.
* Agent Reward Function:
  * -0.0005 for every step.
  * +1.0 if the agent touches the goal.
  * -1.0 if the agent falls off the platform.
* Brains: Two brains, each with the following observation/action space.
  * Vector Observation space: Size of 74, corresponding to 14 raycasts each
    detecting 4 possible objects. plus the global position of the agent and
    whether or not the agent is grounded.
  * Vector Action space: (Discrete) 4 Branches:
    * Forward Motion (3 possible actions: Forward, Backwards, No Action)
    * Rotation (3 possible acions: Rotate Left, Rotate Right, No Action)
    * Side Motion (3 possible acions: Left, Right, No Action)
    * Jump (2 possible actions: Jump, No Action)
  * Visual Observations: None.
* Reset Parameters: 4, corresponding to the height of the possible walls.
* Benchmark Mean Reward (Big & Small Wall Brain): 0.8

## [Reacher](https://youtu.be/2N9EoF6pQyE)

![Reacher](images/reacher.png)

* Set-up: Double-jointed arm which can move to target locations.
* Goal: The agents must move it's hand to the goal location, and keep it there.
* Agents: The environment contains 10 agent linked to a single brain.
* Agent Reward Function (independent):
  * +0.1 Each step agent's hand is in goal location.
* Brains: One brain with the following observation/action space.
  * Vector Observation space: 26 variables corresponding to position, rotation,
    velocity, and angular velocities of the two arm Rigidbodies.
  * Vector Action space: (Continuous) Size of 4, corresponding to torque
    applicable to two joints.
  * Visual Observations: None.
* Reset Parameters: Two, corresponding to goal size, and goal movement speed.
* Benchmark Mean Reward: 30

## [Crawler](https://youtu.be/ftLliaeooYI)

![Crawler](images/crawler.png)

* Set-up: A creature with 4 arms and 4 forearms.
* Goal: The agents must move its body toward the goal direction without falling.
  * `CrawlerStaticTarget` - Goal direction is always forward.
  * `CrawlerDynamicTarget`- Goal direction is randomized.
* Agents: The environment contains 3 agent linked to a single brain.
* Agent Reward Function (independent):
  * +0.03 times body velocity in the goal direction.
  * +0.01 times body direction alignment with goal direction.
* Brains: One brain with the following observation/action space.
  * Vector Observation space: 117 variables corresponding to position, rotation,
    velocity, and angular velocities of each limb plus the acceleration and
    angular acceleration of the body.
  * Vector Action space: (Continuous) Size of 20, corresponding to target
    rotations for joints.
  * Visual Observations: None.
* Reset Parameters: None
* Benchmark Mean Reward: 2000

## [Banana Collector](https://youtu.be/heVMs3t9qSk)

![Banana](images/banana.png)

* Set-up: A multi-agent environment where agents compete to collect bananas.
* Goal: The agents must learn to move to as many yellow bananas as possible
  while avoiding blue bananas.
* Agents: The environment contains 5 agents linked to a single brain.
* Agent Reward Function (independent):
  * +1 for interaction with yellow banana
  * -1 for interaction with blue banana.
* Brains: One brain with the following observation/action space.
  * Vector Observation space: 53 corresponding to velocity of agent (2), whether
    agent is frozen and/or shot its laser (2), plus ray-based perception of
    objects around agent's forward direction (49; 7 raycast angles with 7
    measurements for each).
  * Vector Action space: (Discrete) 4 Branches:
    * Forward Motion (3 possible actions: Forward, Backwards, No Action)
    * Side Motion (3 possible acions: Left, Right, No Action)
    * Rotation (3 possible acions: Rotate Left, Rotate Right, No Action)
    * Laser (2 possible actions: Laser, No Action)
  * Visual Observations (Optional): First-person camera per-agent. Use
    `VisualBanana` scene.
* Reset Parameters: None.
* Benchmark Mean Reward: 10
* Optional Imitation Learning scene: `BananaIL`.

## [Hallway](https://youtu.be/53GyfpPQRUQ)

![Hallway](images/hallway.png)

* Set-up: Environment where the agent needs to find information in a room,
  remember it, and use it to move to the correct goal.
* Goal: Move to the goal which corresponds to the color of the block in the
  room.
* Agents: The environment contains one agent linked to a single brain.
* Agent Reward Function (independent):
  * +1 For moving to correct goal.
  * -0.1 For moving to incorrect goal.
  * -0.0003 Existential penalty.
* Brains: One brain with the following observation/action space:
  * Vector Observation space: 30 corresponding to local ray-casts detecting
    objects, goals, and walls.
  * Vector Action space: (Discrete) 1 Branch, 4 actions corresponding to agent
    rotation and forward/backward movement.
  * Visual Observations (Optional): First-person view for the agent. Use
    `VisualHallway` scene.
* Reset Parameters: None.
* Benchmark Mean Reward: 0.7
* Optional Imitation Learning scene: `HallwayIL`.

## [Bouncer](https://youtu.be/Tkv-c-b1b2I)

![Bouncer](images/bouncer.png)

* Set-up: Environment where the agent needs on-demand decision making. The agent
  must decide how perform its next bounce only when it touches the ground.
* Goal: Catch the floating banana. Only has a limited number of jumps.
* Agents: The environment contains one agent linked to a single brain.
* Agent Reward Function (independent):
  * +1 For catching the banana.
  * -1 For bouncing out of bounds.
  * -0.05 Times the action squared. Energy expenditure penalty.
* Brains: One brain with the following observation/action space:
  * Vector Observation space: 6 corresponding to local position of agent and
    banana.
  * Vector Action space: (Continuous) 3 corresponding to agent force applied for
    the jump.
  * Visual Observations: None.
* Reset Parameters: None.
* Benchmark Mean Reward: 2.5

## [Soccer Twos](https://youtu.be/Hg3nmYD3DjQ)

![SoccerTwos](images/soccer.png)

* Set-up: Environment where four agents compete in a 2 vs 2 toy soccer game.
* Goal:
  * Striker: Get the ball into the opponent's goal.
  * Goalie: Prevent the ball from entering its own goal.
* Agents: The environment contains four agents, with two linked to one brain
  (strikers) and two linked to another (goalies).
* Agent Reward Function (dependent):
  * Striker:
    * +1 When ball enters opponent's goal.
    * -0.1 When ball enters own team's goal.
    * -0.001 Existential penalty.
  * Goalie:
    * -1 When ball enters team's goal.
    * +0.1 When ball enters opponents goal.
    * +0.001 Existential bonus.
* Brains: Two brain with the following observation/action space:
  * Vector Observation space: 112 corresponding to local 14 ray casts, each
    detecting 7 possible object types, along with the object's distance.
    Perception is in 180 degree view from front of agent.
  * Vector Action space: (Discrete) One Branch
    * Striker: 6 actions corresponding to forward, backward, sideways movement,
      as well as rotation.
    * Goalie: 4 actions corresponding to forward, backward, sideways movement.
  * Visual Observations: None.
* Reset Parameters: None
* Benchmark Mean Reward (Striker & Goalie Brain): 0 (the means will be inverse
  of each other and criss crosses during training)

## Walker

![Walker](images/walker.png)

* Set-up: Physics-based Humanoids agents with 26 degrees of freedom. These DOFs
  correspond to articulation of the following body-parts: hips, chest, spine,
  head, thighs, shins, feets, arms, forearms and hands.
* Goal: The agents must move its body toward the goal direction as quickly as
  possible without falling.
* Agents: The environment contains 11 independent agent linked to a single
  brain.
* Agent Reward Function (independent):
  * +0.03 times body velocity in the goal direction.
  * +0.01 times head y position.
  * +0.01 times body direction alignment with goal direction.
  * -0.01 times head velocity difference from body velocity.
* Brains: One brain with the following observation/action space.
  * Vector Observation space: 215 variables corresponding to position, rotation,
    velocity, and angular velocities of each limb, along with goal direction.
  * Vector Action space: (Continuous) Size of 39, corresponding to target
    rotations applicable to the joints.
  * Visual Observations: None.
* Reset Parameters: None.
* Benchmark Mean Reward: 1000

## Pyramids

![Pyramids](images/pyramids.png)

* Set-up: Environment where the agent needs to press a button to spawn a
  pyramid, then navigate to the pyramid, knock it over, and move to the gold
  brick at the top.
* Goal: Move to the golden brick on top of the spawned pyramid.
* Agents: The environment contains one agent linked to a single brain.
* Agent Reward Function (independent):
  * +2 For moving to golden brick (minus 0.001 per step).
* Brains: One brain with the following observation/action space:
  * Vector Observation space: 148 corresponding to local ray-casts detecting
    switch, bricks, golden brick, and walls, plus variable indicating switch
    state.
  * Vector Action space: (Discrete) 4 corresponding to agent rotation and
    forward/backward movement.
  * Visual Observations (Optional): First-person camera per-agent. Us
    `VisualPyramids` scene.
* Reset Parameters: None.
* Optional Imitation Learning scene: `PyramidsIL`.
* Benchmark Mean Reward: 1.75