This tutorial walks through the process of creating a Unity Environment from scratch. We recommend first reading the [Getting Started](Getting-Started.md) guide to understand the concepts presented here first in an already-built environment.
This tutorial walks through the process of creating a Unity Environment from
scratch. We recommend first reading the [Getting Started](Getting-Started.md)
guide to understand the concepts presented here first in an already-built
environment.
In this example, we will create an agent capable of controlling a ball on a platform. We will then train the agent to roll the ball toward the cube while avoiding falling off the platform.
In this example, we will create an agent capable of controlling a ball on a
platform. We will then train the agent to roll the ball toward the cube while
avoiding falling off the platform.
Using the ML-Agents toolkit in a Unity project involves the following basic
Using the ML-Agents Toolkit in a Unity project involves the following basic
from a simple physical simulation containing a few objects to an entire game
or ecosystem.
2. Implement your Agent subclasses. An Agent subclass defines the code an Agent
uses to observe its environment, to carry out assigned actions, and to
calculate the rewards used for reinforcement training. You can also implement
optional methods to reset the Agent when it has finished or failed its task.
3. Add your Agent subclasses to appropriate GameObjects, typically, the object
in the scene that represents the Agent in the simulation.
from a simple physical simulation containing a few objects to an entire game
or ecosystem.
1. Implement your Agent subclasses. An Agent subclass defines the code an Agent
uses to observe its environment, to carry out assigned actions, and to
calculate the rewards used for reinforcement training. You can also implement
optional methods to reset the Agent when it has finished or failed its task.
1. Add your Agent subclasses to appropriate GameObjects, typically, the object
in the scene that represents the Agent in the simulation.
**Note:** If you are unfamiliar with Unity, refer to
[Learning the interface](https://docs.unity3d.com/Manual/LearningtheInterface.html)
The first task to accomplish is simply creating a new Unity project and
importing the ML-Agents assets into it:
1. Launch the Unity Editor and create a new project named "RollerBall".
2. Make sure that the Scripting Runtime Version for the project is set to use
**.NET 4.x Equivalent** (This is an experimental option in Unity 2017,
but is the default as of 2018.3.)
3. In a file system window, navigate to the folder containing your cloned
ML-Agents repository.
4. Open the `manifest.json` file in the `Packages` directory of your project.
Add the following line to your project's package dependencies:
The C# editor code and python trainer code are not compatible between releases. This means that if you upgrade one, you *must* upgrade the other as well. If you experience new errors or unable to connect to training after updating, please double-check that the versions are in the same.
The versions can be found in
* `Academy.k_ApiVersion` in Academy.cs ([example](https://github.com/Unity-Technologies/ml-agents/blob/b255661084cb8f701c716b040693069a3fb9a257/UnitySDK/Assets/ML-Agents/Scripts/Academy.cs#L95))
* `UnityEnvironment.API_VERSION` in environment.py ([example](https://github.com/Unity-Technologies/ml-agents/blob/b255661084cb8f701c716b040693069a3fb9a257/ml-agents-envs/mlagents/envs/environment.py#L45))
The C# editor code and python trainer code are not compatible between releases.
This means that if you upgrade one, you _must_ upgrade the other as well. If you
experience new errors or unable to connect to training after updating, please
double-check that the versions are in the same. The versions can be found in
* The `--load` and `--train` command-line flags have been deprecated and replaced with `--resume` and `--inference`.
* Running with the same `--run-id` twice will now throw an error.
* Removed the multi-agent gym option from the gym wrapper. For multi-agent scenarios, use the [Low Level Python API](Python-API.md).
* The low level Python API has changed. You can look at the document [Low Level Python API documentation](Python-API.md) for more information. If you use `mlagents-learn` for training, this should be a transparent change.
* The obsolete `Agent` methods `GiveModel`, `Done`, `InitializeAgent`, `AgentAction` and `AgentReset` have been removed.
* The signature of `Agent.Heuristic()` was changed to take a `float[]` as a parameter, instead of returning the array. This was done to prevent a common source of error where users would return arrays of the wrong size.
* `num_updates` and `train_interval` for SAC have been replaced with `steps_per_update`.
- The `--load` and `--train` command-line flags have been deprecated and
replaced with `--resume` and `--inference`.
- Running with the same `--run-id` twice will now throw an error.
- The `play_against_current_self_ratio` self-play trainer hyperparameter has
been renamed to `play_against_latest_model_ratio`
- Removed the multi-agent gym option from the gym wrapper. For multi-agent
scenarios, use the [Low Level Python API](Python-API.md).
- The low level Python API has changed. You can look at the document
[Low Level Python API documentation](Python-API.md) for more information. If
you use `mlagents-learn` for training, this should be a transparent change.
- The obsolete `Agent` methods `GiveModel`, `Done`, `InitializeAgent`,
`AgentAction` and `AgentReset` have been removed.
- The signature of `Agent.Heuristic()` was changed to take a `float[]` as a
parameter, instead of returning the array. This was done to prevent a common
source of error where users would return arrays of the wrong size.
- `num_updates` and `train_interval` for SAC have been replaced with `steps_per_update`.
* Replace the `--load` flag with `--resume` when calling `mlagents-learn`, and don't use the `--train` flag as training
will happen by default. To run with inference instead of training, use `--inference`.
* To force-overwrite files from a pre-existing run, add the `--force` command-line flag.
* The Jupyter notebooks have been removed from the repository.
* `Academy.FloatProperties` was removed.
* `Academy.RegisterSideChannel` and `Academy.UnregisterSideChannel` were removed.
* Replace `Academy.FloatProperties` with `SideChannelUtils.GetSideChannel<FloatPropertiesChannel>()`.
* Replace `Academy.RegisterSideChannel` with `SideChannelUtils.RegisterSideChannel()`.
* Replace `Academy.UnregisterSideChannel` with `SideChannelUtils.UnregisterSideChannel`.
* If your Agent class overrides `Heuristic()`, change the signature to `public override void Heuristic(float[] actionsOut)` and assign values to `actionsOut` instead of returning an array.
* `steps_per_update` should be around equal to the number of agents in your environment, times `num_updates`
and divided by `train_interval`.
- Replace the `--load` flag with `--resume` when calling `mlagents-learn`, and
don't use the `--train` flag as training will happen by default. To run
without training, use `--inference`.
- To force-overwrite files from a pre-existing run, add the `--force`
command-line flag.
- The Jupyter notebooks have been removed from the repository.
- `Academy.FloatProperties` was removed.
- `Academy.RegisterSideChannel` and `Academy.UnregisterSideChannel` were
- If your Agent class overrides `Heuristic()`, change the signature to
`public override void Heuristic(float[] actionsOut)` and assign values to
`actionsOut` instead of returning an array.
- Set `steps_per_update` to be around equal to the number of agents in your environment,
times `num_updates` and divided by `train_interval`.
* The `Agent.CollectObservations()` virtual method now takes as input a `VectorSensor` sensor as argument. The `Agent.AddVectorObs()` methods were removed.
* The `SetMask` was renamed to `SetMask` method must now be called on the `DiscreteActionMasker` argument of the `CollectDiscreteActionMasks` virtual method.
* We consolidated our API for `DiscreteActionMasker`. `SetMask` takes two arguments : the branch index and the list of masked actions for that branch.
* The `Monitor` class has been moved to the Examples Project. (It was prone to errors during testing)
* The `MLAgents.Sensors` namespace has been introduced. All sensors classes are part of the `MLAgents.Sensors` namespace.
* The `MLAgents.SideChannels` namespace has been introduced. All side channel classes are part of the `MLAgents.SideChannels` namespace.
* The interface for `RayPerceptionSensor.PerceiveStatic()` was changed to take an input class and write to an output class, and the method was renamed to `Perceive()`.
* The `SetMask` method must now be called on the `DiscreteActionMasker` argument of the `CollectDiscreteActionMasks` method.
* The method `GetStepCount()` on the Agent class has been replaced with the property getter `StepCount`
* The `--multi-gpu` option has been removed temporarily.
* `AgentInfo.actionMasks` has been renamed to `AgentInfo.discreteActionMasks`.
* `BrainParameters` and `SpaceType` have been removed from the public API
* `BehaviorParameters` have been removed from the public API.
* `DecisionRequester` has been made internal (you can still use the DecisionRequesterComponent from the inspector). `RepeatAction` was renamed `TakeActionsBetweenDecisions` for clarity.
* The following methods in the `Agent` class have been renamed. The original method names will be removed in a later release:
* `InitializeAgent()` was renamed to `Initialize()`
* `AgentAction()` was renamed to `OnActionReceived()`
* `AgentReset()` was renamed to `OnEpsiodeBegin()`
* `Done()` was renamed to `EndEpisode()`
* `GiveModel()` was renamed to `SetModel()`
* The `IFloatProperties` interface has been removed.
* The interface for SideChannels was changed:
* In C#, `OnMessageReceived` now takes a `IncomingMessage` argument, and `QueueMessageToSend` takes an `OutgoingMessage` argument.
* In python, `on_message_received` now takes a `IncomingMessage` argument, and `queue_message_to_send` takes an `OutgoingMessage` argument.
* Automatic stepping for Academy is now controlled from the AutomaticSteppingEnabled property.
- The `Agent.CollectObservations()` virtual method now takes as input a
`VectorSensor` sensor as argument. The `Agent.AddVectorObs()` methods were
removed.
- The `SetMask` was renamed to `SetMask` method must now be called on the
`DiscreteActionMasker` argument of the `CollectDiscreteActionMasks` virtual
method.
- We consolidated our API for `DiscreteActionMasker`. `SetMask` takes two
arguments : the branch index and the list of masked actions for that branch.
- The `Monitor` class has been moved to the Examples Project. (It was prone to
errors during testing)
- The `MLAgents.Sensors` namespace has been introduced. All sensors classes are
part of the `MLAgents.Sensors` namespace.
- The `MLAgents.SideChannels` namespace has been introduced. All side channel
classes are part of the `MLAgents.SideChannels` namespace.
- The interface for `RayPerceptionSensor.PerceiveStatic()` was changed to take
an input class and write to an output class, and the method was renamed to
`Perceive()`.
- The `SetMask` method must now be called on the `DiscreteActionMasker` argument
of the `CollectDiscreteActionMasks` method.
- The method `GetStepCount()` on the Agent class has been replaced with the
property getter `StepCount`
- The `--multi-gpu` option has been removed temporarily.
- `AgentInfo.actionMasks` has been renamed to `AgentInfo.discreteActionMasks`.
- `BrainParameters` and `SpaceType` have been removed from the public API
- `BehaviorParameters` have been removed from the public API.
- `DecisionRequester` has been made internal (you can still use the
DecisionRequesterComponent from the inspector). `RepeatAction` was renamed
`TakeActionsBetweenDecisions` for clarity.
- The following methods in the `Agent` class have been renamed. The original
method names will be removed in a later release:
- `InitializeAgent()` was renamed to `Initialize()`
- `AgentAction()` was renamed to `OnActionReceived()`
- `AgentReset()` was renamed to `OnEpsiodeBegin()`
- `Done()` was renamed to `EndEpisode()`
- `GiveModel()` was renamed to `SetModel()`
- The `IFloatProperties` interface has been removed.
- The interface for SideChannels was changed:
- In C#, `OnMessageReceived` now takes a `IncomingMessage` argument, and
`QueueMessageToSend` takes an `OutgoingMessage` argument.
- In python, `on_message_received` now takes a `IncomingMessage` argument, and
`queue_message_to_send` takes an `OutgoingMessage` argument.
- Automatic stepping for Academy is now controlled from the
AutomaticSteppingEnabled property.
* Add the `using MLAgents.Sensors;` in addition to `using MLAgents;` on top of your Agent's script.
* Replace your Agent's implementation of `CollectObservations()` with `CollectObservations(VectorSensor sensor)`. In addition, replace all calls to `AddVectorObs()` with `sensor.AddObservation()` or `sensor.AddOneHotObservation()` on the `VectorSensor` passed as argument.
* Replace your calls to `SetActionMask` on your Agent to `DiscreteActionMasker.SetActionMask` in `CollectDiscreteActionMasks`.
* If you call `RayPerceptionSensor.PerceiveStatic()` manually, add your inputs to a `RayPerceptionInput`. To get the previous float array output,
iterate through `RayPerceptionOutput.rayOutputs` and call `RayPerceptionOutput.RayOutput.ToFloatArray()`.
* Replace all calls to `Agent.GetStepCount()` with `Agent.StepCount`
* We strongly recommend replacing the following methods with their new equivalent as they will be removed in a later release:
* `InitializeAgent()` to `Initialize()`
* `AgentAction()` to `OnActionReceived()`
* `AgentReset()` to `OnEpisodeBegin()`
* `Done()` to `EndEpisode()`
* `GiveModel()` to `SetModel()`
* Replace `IFloatProperties` variables with `FloatPropertiesChannel` variables.
* If you implemented custom `SideChannels`, update the signatures of your methods, and add your data to the `OutgoingMessage` or read it from the `IncomingMessage`.
* Replace calls to Academy.EnableAutomaticStepping()/DisableAutomaticStepping() with Academy.AutomaticSteppingEnabled = true/false.
- Add the `using MLAgents.Sensors;` in addition to `using MLAgents;` on top of
your Agent's script.
- Replace your Agent's implementation of `CollectObservations()` with
`CollectObservations(VectorSensor sensor)`. In addition, replace all calls to
`AddVectorObs()` with `sensor.AddObservation()` or
`sensor.AddOneHotObservation()` on the `VectorSensor` passed as argument.
- Replace your calls to `SetActionMask` on your Agent to
`DiscreteActionMasker.SetActionMask` in `CollectDiscreteActionMasks`.
- If you call `RayPerceptionSensor.PerceiveStatic()` manually, add your inputs
to a `RayPerceptionInput`. To get the previous float array output, iterate
through `RayPerceptionOutput.rayOutputs` and call
`RayPerceptionOutput.RayOutput.ToFloatArray()`.
- Replace all calls to `Agent.GetStepCount()` with `Agent.StepCount`
- We strongly recommend replacing the following methods with their new
equivalent as they will be removed in a later release:
- `InitializeAgent()` to `Initialize()`
- `AgentAction()` to `OnActionReceived()`
- `AgentReset()` to `OnEpisodeBegin()`
- `Done()` to `EndEpisode()`
- `GiveModel()` to `SetModel()`
- Replace `IFloatProperties` variables with `FloatPropertiesChannel` variables.
- If you implemented custom `SideChannels`, update the signatures of your
methods, and add your data to the `OutgoingMessage` or read it from the
`IncomingMessage`.
- Replace calls to Academy.EnableAutomaticStepping()/DisableAutomaticStepping()
with Academy.AutomaticSteppingEnabled = true/false.
* The `UnitySDK` folder has been split into a Unity Package (`com.unity.ml-agents`) and an examples project (`Project`). Please follow the [Installation Guide](Installation.md) to get up and running with this new repo structure.
* Several changes were made to how agents are reset and marked as done:
* Calling `Done()` on the Agent will now reset it immediately and call the `AgentReset` virtual method. (This is to simplify the previous logic in which the Agent had to wait for the next `EnvironmentStep` to reset)
* The "Reset on Done" setting in AgentParameters was removed; this is now effectively always true. `AgentOnDone` virtual method on the Agent has been removed.
* The `Decision Period` and `On Demand decision` checkbox have been removed from the Agent. On demand decision is now the default (calling `RequestDecision` on the Agent manually.)
* The Academy class was changed to a singleton, and its virtual methods were removed.
* Trainer steps are now counted per-Agent, not per-environment as in previous versions. For instance, if you have 10 Agents in the scene, 20 environment steps now corresponds to 200 steps as printed in the terminal and in Tensorboard.
* Curriculum config files are now YAML formatted and all curricula for a training run are combined into a single file.
* The `--num-runs` command-line option has been removed from `mlagents-learn`.
* Several fields on the Agent were removed or made private in order to simplify the interface.
* The `agentParameters` field of the Agent has been removed. (Contained only `maxStep` information)
* `maxStep` is now a public field on the Agent. (Was moved from `agentParameters`)
* The `Info` field of the Agent has been made private. (Was only used internally and not meant to be modified outside of the Agent)
* The `GetReward()` method on the Agent has been removed. (It was being confused with `GetCumulativeReward()`)
* The `AgentAction` struct no longer contains a `value` field. (Value estimates were not set during inference)
* The `GetValueEstimate()` method on the Agent has been removed.
* The `UpdateValueAction()` method on the Agent has been removed.
* The deprecated `RayPerception3D` and `RayPerception2D` classes were removed, and the `legacyHitFractionBehavior` argument was removed from `RayPerceptionSensor.PerceiveStatic()`.
* RayPerceptionSensor was inconsistent in how it handle scale on the Agent's transform. It now scales the ray length and sphere size for casting as the transform's scale changes.
- The `UnitySDK` folder has been split into a Unity Package
(`com.unity.ml-agents`) and an examples project (`Project`). Please follow the
[Installation Guide](Installation.md) to get up and running with this new repo
structure.
- Several changes were made to how agents are reset and marked as done:
- Calling `Done()` on the Agent will now reset it immediately and call the
`AgentReset` virtual method. (This is to simplify the previous logic in
which the Agent had to wait for the next `EnvironmentStep` to reset)
- The "Reset on Done" setting in AgentParameters was removed; this is now
effectively always true. `AgentOnDone` virtual method on the Agent has been
removed.
- The `Decision Period` and `On Demand decision` checkbox have been removed from
the Agent. On demand decision is now the default (calling `RequestDecision` on
the Agent manually.)
- The Academy class was changed to a singleton, and its virtual methods were
removed.
- Trainer steps are now counted per-Agent, not per-environment as in previous
versions. For instance, if you have 10 Agents in the scene, 20 environment
steps now corresponds to 200 steps as printed in the terminal and in
Tensorboard.
- Curriculum config files are now YAML formatted and all curricula for a
training run are combined into a single file.
- The `--num-runs` command-line option has been removed from `mlagents-learn`.
- Several fields on the Agent were removed or made private in order to simplify
the interface.
- The `agentParameters` field of the Agent has been removed. (Contained only
`maxStep` information)
- `maxStep` is now a public field on the Agent. (Was moved from
`agentParameters`)
- The `Info` field of the Agent has been made private. (Was only used
internally and not meant to be modified outside of the Agent)
- The `GetReward()` method on the Agent has been removed. (It was being
confused with `GetCumulativeReward()`)
- The `AgentAction` struct no longer contains a `value` field. (Value
estimates were not set during inference)
- The `GetValueEstimate()` method on the Agent has been removed.
- The `UpdateValueAction()` method on the Agent has been removed.
- The deprecated `RayPerception3D` and `RayPerception2D` classes were removed,
and the `legacyHitFractionBehavior` argument was removed from
`RayPerceptionSensor.PerceiveStatic()`.
- RayPerceptionSensor was inconsistent in how it handle scale on the Agent's
transform. It now scales the ray length and sphere size for casting as the
transform's scale changes.
* Follow the instructions on how to install the `com.unity.ml-agents` package into your project in the [Installation Guide](Installation.md).
* If your Agent implemented `AgentOnDone` and did not have the checkbox `Reset On Done` checked in the inspector, you must call the code that was in `AgentOnDone` manually.
* If you give your Agent a reward or penalty at the end of an episode (e.g. for reaching a goal or falling off of a platform), make sure you call `AddReward()` or `SetReward()`*before* calling `Done()`. Previously, the order didn't matter.
* If you were not using `On Demand Decision` for your Agent, you **must** add a `DecisionRequester` component to your Agent GameObject and set its `Decision Period` field to the old `Decision Period` of the Agent.
* If you have a class that inherits from Academy:
* If the class didn't override any of the virtual methods and didn't store any additional data, you can just remove the old script from the scene.
* If the class had additional data, create a new MonoBehaviour and store the data in the new MonoBehaviour instead.
* If the class overrode the virtual methods, create a new MonoBehaviour and move the logic to it:
* Move the InitializeAcademy code to MonoBehaviour.OnAwake
* Move the AcademyStep code to MonoBehaviour.FixedUpdate
* Move the OnDestroy code to MonoBehaviour.OnDestroy.
* Move the AcademyReset code to a new method and add it to the Academy.OnEnvironmentReset action.
* Multiply `max_steps` and `summary_freq` in your `trainer_config.yaml` by the number of Agents in the scene.
* Combine curriculum configs into a single file. See [the WallJump curricula](../config/curricula/wall_jump.yaml) for an example of the new curriculum config format.
A tool like https://www.json2yaml.com may be useful to help with the conversion.
* If you have a model trained which uses RayPerceptionSensor and has non-1.0 scale in the Agent's transform, it must be retrained.
- Follow the instructions on how to install the `com.unity.ml-agents` package
into your project in the [Installation Guide](Installation.md).
- If your Agent implemented `AgentOnDone` and did not have the checkbox
`Reset On Done` checked in the inspector, you must call the code that was in
`AgentOnDone` manually.
- If you give your Agent a reward or penalty at the end of an episode (e.g. for
reaching a goal or falling off of a platform), make sure you call
`AddReward()` or `SetReward()`_before_ calling `Done()`. Previously, the
order didn't matter.
- If you were not using `On Demand Decision` for your Agent, you **must** add a
`DecisionRequester` component to your Agent GameObject and set its
`Decision Period` field to the old `Decision Period` of the Agent.
- If you have a class that inherits from Academy:
- If the class didn't override any of the virtual methods and didn't store any
additional data, you can just remove the old script from the scene.
- If the class had additional data, create a new MonoBehaviour and store the
data in the new MonoBehaviour instead.
- If the class overrode the virtual methods, create a new MonoBehaviour and
move the logic to it:
- Move the InitializeAcademy code to MonoBehaviour.OnAwake
- Move the AcademyStep code to MonoBehaviour.FixedUpdate
- Move the OnDestroy code to MonoBehaviour.OnDestroy.
- Move the AcademyReset code to a new method and add it to the
Academy.OnEnvironmentReset action.
- Multiply `max_steps` and `summary_freq` in your `trainer_config.yaml` by the
number of Agents in the scene.
- Combine curriculum configs into a single file. See
[the WallJump curricula](../config/curricula/wall_jump.yaml) for an example of
the new curriculum config format. A tool like https://www.json2yaml.com may be
useful to help with the conversion.
- If you have a model trained which uses RayPerceptionSensor and has non-1.0
scale in the Agent's transform, it must be retrained.
## Migrating from ML-Agents toolkit v0.12.0 to v0.13.0
## Migrating from ML-Agents Toolkit v0.12.0 to v0.13.0
* The low level Python API has changed. You can look at the document [Low Level Python API documentation](Python-API.md) for more information. This should only affect you if you're writing a custom trainer; if you use `mlagents-learn` for training, this should be a transparent change.
* `reset()` on the Low-Level Python API no longer takes a `train_mode` argument. To modify the performance/speed of the engine, you must use an `EngineConfigurationChannel`
* `reset()` on the Low-Level Python API no longer takes a `config` argument. `UnityEnvironment` no longer has a `reset_parameters` field. To modify float properties in the environment, you must use a `FloatPropertiesChannel`. For more information, refer to the [Low Level Python API documentation](Python-API.md)
* `CustomResetParameters` are now removed.
* The Academy no longer has a `Training Configuration` nor `Inference Configuration` field in the inspector. To modify the configuration from the Low-Level Python API, use an `EngineConfigurationChannel`.
To modify it during training, use the new command line arguments `--width`, `--height`, `--quality-level`, `--time-scale` and `--target-frame-rate` in `mlagents-learn`.
* The Academy no longer has a `Default Reset Parameters` field in the inspector. The Academy class no longer has a `ResetParameters`. To access shared float properties with Python, use the new `FloatProperties` field on the Academy.
* Offline Behavioral Cloning has been removed. To learn from demonstrations, use the GAIL and
Behavioral Cloning features with either PPO or SAC. See [Imitation Learning](Training-Imitation-Learning.md) for more information.
* `mlagents.envs` was renamed to `mlagents_envs`. The previous repo layout depended on [PEP420](https://www.python.org/dev/peps/pep-0420/), which caused problems with some of our tooling such as mypy and pylint.
* The official version of Unity ML-Agents supports is now 2018.4 LTS. If you run into issues, please consider deleting your library folder and reponening your projects. You will need to install the Barracuda package into your project in order to ML-Agents to compile correctly.
- The low level Python API has changed. You can look at the document
[Low Level Python API documentation](Python-API.md) for more information. This
should only affect you if you're writing a custom trainer; if you use
`mlagents-learn` for training, this should be a transparent change.
- `reset()` on the Low-Level Python API no longer takes a `train_mode`
argument. To modify the performance/speed of the engine, you must use an
`EngineConfigurationChannel`
- `reset()` on the Low-Level Python API no longer takes a `config` argument.
`UnityEnvironment` no longer has a `reset_parameters` field. To modify float
properties in the environment, you must use a `FloatPropertiesChannel`. For
more information, refer to the
[Low Level Python API documentation](Python-API.md)
- `CustomResetParameters` are now removed.
- The Academy no longer has a `Training Configuration` nor
`Inference Configuration` field in the inspector. To modify the configuration
from the Low-Level Python API, use an `EngineConfigurationChannel`. To modify
it during training, use the new command line arguments `--width`, `--height`,
`--quality-level`, `--time-scale` and `--target-frame-rate` in
`mlagents-learn`.
- The Academy no longer has a `Default Reset Parameters` field in the inspector.
The Academy class no longer has a `ResetParameters`. To access shared float
properties with Python, use the new `FloatProperties` field on the Academy.
- Offline Behavioral Cloning has been removed. To learn from demonstrations, use
the GAIL and Behavioral Cloning features with either PPO or SAC. See
[Imitation Learning](Training-Imitation-Learning.md) for more information.
- `mlagents.envs` was renamed to `mlagents_envs`. The previous repo layout
depended on [PEP420](https://www.python.org/dev/peps/pep-0420/), which caused
problems with some of our tooling such as mypy and pylint.
- The official version of Unity ML-Agents supports is now 2018.4 LTS. If you run
into issues, please consider deleting your library folder and reponening your
projects. You will need to install the Barracuda package into your project in
order to ML-Agents to compile correctly.
* If you had a custom `Training Configuration` in the Academy inspector, you will need to pass your custom configuration at every training run using the new command line arguments `--width`, `--height`, `--quality-level`, `--time-scale` and `--target-frame-rate`.
* If you were using `--slow` in `mlagents-learn`, you will need to pass your old `Inference Configuration` of the Academy inspector with the new command line arguments `--width`, `--height`, `--quality-level`, `--time-scale` and `--target-frame-rate` instead.
* Any imports from `mlagents.envs` should be replaced with `mlagents_envs`.
## Migrating from ML-Agents toolkit v0.11.0 to v0.12.0
- If you had a custom `Training Configuration` in the Academy inspector, you
will need to pass your custom configuration at every training run using the
new command line arguments `--width`, `--height`, `--quality-level`,
`--time-scale` and `--target-frame-rate`.
- If you were using `--slow` in `mlagents-learn`, you will need to pass your old
`Inference Configuration` of the Academy inspector with the new command line
arguments `--width`, `--height`, `--quality-level`, `--time-scale` and
`--target-frame-rate` instead.
- Any imports from `mlagents.envs` should be replaced with `mlagents_envs`.
## Migrating from ML-Agents Toolkit v0.11.0 to v0.12.0
* Text actions and observations, and custom action and observation protos have been removed.
* RayPerception3D and RayPerception2D are marked deprecated, and will be removed in a future release. They can be replaced by RayPerceptionSensorComponent3D and RayPerceptionSensorComponent2D.
* The `Use Heuristic` checkbox in Behavior Parameters has been replaced with a `Behavior Type` dropdown menu. This has the following options:
* `Default` corresponds to the previous unchecked behavior, meaning that Agents will train if they connect to a python trainer, otherwise they will perform inference.
* `Heuristic Only` means the Agent will always use the `Heuristic()` method. This corresponds to having "Use Heuristic" selected in 0.11.0.
* `Inference Only` means the Agent will always perform inference.
* Barracuda was upgraded to 0.3.2, and it is now installed via the Unity Package Manager.
- Text actions and observations, and custom action and observation protos have
been removed.
- RayPerception3D and RayPerception2D are marked deprecated, and will be removed
in a future release. They can be replaced by RayPerceptionSensorComponent3D
and RayPerceptionSensorComponent2D.
- The `Use Heuristic` checkbox in Behavior Parameters has been replaced with a
`Behavior Type` dropdown menu. This has the following options:
- `Default` corresponds to the previous unchecked behavior, meaning that
Agents will train if they connect to a python trainer, otherwise they will
perform inference.
- `Heuristic Only` means the Agent will always use the `Heuristic()` method.
This corresponds to having "Use Heuristic" selected in 0.11.0.
- `Inference Only` means the Agent will always perform inference.
- Barracuda was upgraded to 0.3.2, and it is now installed via the Unity Package
Manager.
* We [fixed a bug](https://github.com/Unity-Technologies/ml-agents/pull/2823) in `RayPerception3d.Perceive()` that was causing the `endOffset` to be used incorrectly. However this may produce different behavior from previous versions if you use a non-zero `startOffset`.
To reproduce the old behavior, you should increase the the value of `endOffset` by `startOffset`.
You can verify your raycasts are performing as expected in scene view using the debug rays.
* If you use RayPerception3D, replace it with RayPerceptionSensorComponent3D (and similarly for 2D). The settings, such as ray angles and detectable tags, are configured on the component now.
RayPerception3D would contribute `(# of rays) * (# of tags + 2)` to the State Size in Behavior Parameters, but this is no longer necessary, so you should reduce the State Size by this amount.
Making this change will require retraining your model, since the observations that RayPerceptionSensorComponent3D produces are different from the old behavior.
* If you see messages such as `The type or namespace 'Barracuda' could not be found` or `The type or namespace 'Google' could not be found`, you will need to [install the Barracuda preview package](Installation.md#package-installation).
- We [fixed a bug](https://github.com/Unity-Technologies/ml-agents/pull/2823) in
`RayPerception3d.Perceive()` that was causing the `endOffset` to be used
incorrectly. However this may produce different behavior from previous
versions if you use a non-zero `startOffset`. To reproduce the old behavior,
you should increase the the value of `endOffset` by `startOffset`. You can
verify your raycasts are performing as expected in scene view using the debug
rays.
- If you use RayPerception3D, replace it with RayPerceptionSensorComponent3D
(and similarly for 2D). The settings, such as ray angles and detectable tags,
are configured on the component now. RayPerception3D would contribute
`(# of rays) * (# of tags + 2)` to the State Size in Behavior Parameters, but
this is no longer necessary, so you should reduce the State Size by this
amount. Making this change will require retraining your model, since the
observations that RayPerceptionSensorComponent3D produces are different from
the old behavior.
- If you see messages such as
`The type or namespace 'Barracuda' could not be found` or
`The type or namespace 'Google' could not be found`, you will need to
[install the Barracuda preview package](Installation.md#package-installation).
## Migrating from ML-Agents toolkit v0.10 to v0.11.0
## Migrating from ML-Agents Toolkit v0.10 to v0.11.0
* The definition of the gRPC service has changed.
* The online BC training feature has been removed.
* The BroadcastHub has been deprecated. If there is a training Python process, all LearningBrains in the scene will automatically be trained. If there is no Python process, inference will be used.
* The Brain ScriptableObjects have been deprecated. The Brain Parameters are now on the Agent and are referred to as Behavior Parameters. Make sure the Behavior Parameters is attached to the Agent GameObject.
* To use a heuristic behavior, implement the `Heuristic()` method in the Agent class and check the `use heuristic` checkbox in the Behavior Parameters.
* Several changes were made to the setup for visual observations (i.e. using Cameras or RenderTextures):
* Camera resolutions are no longer stored in the Brain Parameters.
* AgentParameters no longer stores lists of Cameras and RenderTextures
* To add visual observations to an Agent, you must now attach a CameraSensorComponent or RenderTextureComponent to the agent. The corresponding Camera or RenderTexture can be added to these in the editor, and the resolution and color/grayscale is configured on the component itself.
- The definition of the gRPC service has changed.
- The online BC training feature has been removed.
- The BroadcastHub has been deprecated. If there is a training Python process,
all LearningBrains in the scene will automatically be trained. If there is no
Python process, inference will be used.
- The Brain ScriptableObjects have been deprecated. The Brain Parameters are now
on the Agent and are referred to as Behavior Parameters. Make sure the
Behavior Parameters is attached to the Agent GameObject.
- To use a heuristic behavior, implement the `Heuristic()` method in the Agent
class and check the `use heuristic` checkbox in the Behavior Parameters.
- Several changes were made to the setup for visual observations (i.e. using
Cameras or RenderTextures):
- Camera resolutions are no longer stored in the Brain Parameters.
- AgentParameters no longer stores lists of Cameras and RenderTextures
- To add visual observations to an Agent, you must now attach a
CameraSensorComponent or RenderTextureComponent to the agent. The
corresponding Camera or RenderTexture can be added to these in the editor,
and the resolution and color/grayscale is configured on the component
itself.
* In order to be able to train, make sure both your ML-Agents Python package and UnitySDK code come from the v0.11 release. Training will not work, for example, if you update the ML-Agents Python package, and only update the API Version in UnitySDK.
* If your Agents used visual observations, you must add a CameraSensorComponent corresponding to each old Camera in the Agent's camera list (and similarly for RenderTextures).
* Since Brain ScriptableObjects have been removed, you will need to delete all the Brain ScriptableObjects from your `Assets` folder. Then, add a `Behavior Parameters` component to each `Agent` GameObject.
You will then need to complete the fields on the new `Behavior Parameters` component with the BrainParameters of the old Brain.
## Migrating from ML-Agents toolkit v0.9 to v0.10
- In order to be able to train, make sure both your ML-Agents Python package and
UnitySDK code come from the v0.11 release. Training will not work, for
example, if you update the ML-Agents Python package, and only update the API
Version in UnitySDK.
- If your Agents used visual observations, you must add a CameraSensorComponent
corresponding to each old Camera in the Agent's camera list (and similarly for
RenderTextures).
- Since Brain ScriptableObjects have been removed, you will need to delete all
the Brain ScriptableObjects from your `Assets` folder. Then, add a
`Behavior Parameters` component to each `Agent` GameObject. You will then need
to complete the fields on the new `Behavior Parameters` component with the
BrainParameters of the old Brain.
## Migrating from ML-Agents Toolkit v0.9 to v0.10
* We have updated the C# code in our repository to be in line with Unity Coding Conventions. This has changed the name of some public facing classes and enums.
* The example environments have been updated. If you were using these environments to benchmark your training, please note that the resulting rewards may be slightly different in v0.10.
- We have updated the C# code in our repository to be in line with Unity Coding
Conventions. This has changed the name of some public facing classes and
enums.
- The example environments have been updated. If you were using these
environments to benchmark your training, please note that the resulting
rewards may be slightly different in v0.10.
* `UnitySDK/Assets/ML-Agents/Scripts/Communicator.cs` and its class `Communicator` have been renamed to `UnitySDK/Assets/ML-Agents/Scripts/ICommunicator.cs` and `ICommunicator` respectively.
* The `SpaceType` Enums `discrete`, and `continuous` have been renamed to `Discrete` and `Continuous`.
* We have removed the `Done` call as well as the capacity to set `Max Steps` on the Academy. Therefore an AcademyReset will never be triggered from C# (only from Python). If you want to reset the simulation after a
fixed number of steps, or when an event in the simulation occurs, we recommend looking at our multi-agent example environments (such as FoodCollector).
In our examples, groups of Agents can be reset through an "Area" that can reset groups of Agents.
* The import for `mlagents.envs.UnityEnvironment` was removed. If you are using the Python API, change `from mlagents_envs import UnityEnvironment` to `from mlagents_envs.environment import UnityEnvironment`.
- `UnitySDK/Assets/ML-Agents/Scripts/Communicator.cs` and its class
`Communicator` have been renamed to
`UnitySDK/Assets/ML-Agents/Scripts/ICommunicator.cs` and `ICommunicator`
respectively.
- The `SpaceType` Enums `discrete`, and `continuous` have been renamed to
`Discrete` and `Continuous`.
- We have removed the `Done` call as well as the capacity to set `Max Steps` on
the Academy. Therefore an AcademyReset will never be triggered from C# (only
from Python). If you want to reset the simulation after a fixed number of
steps, or when an event in the simulation occurs, we recommend looking at our
multi-agent example environments (such as FoodCollector). In our examples,
groups of Agents can be reset through an "Area" that can reset groups of
Agents.
- The import for `mlagents.envs.UnityEnvironment` was removed. If you are using
the Python API, change `from mlagents_envs import UnityEnvironment` to
* We have changed the way reward signals (including Curiosity) are defined in the
`trainer_config.yaml`.
* When using multiple environments, every "step" is recorded in TensorBoard.
* The steps in the command line console corresponds to a single step of a single environment.
Previously, each step corresponded to one step for all environments (i.e., `num_envs` steps).
- We have changed the way reward signals (including Curiosity) are defined in
the `trainer_config.yaml`.
- When using multiple environments, every "step" is recorded in TensorBoard.
- The steps in the command line console corresponds to a single step of a single
environment. Previously, each step corresponded to one step for all
environments (i.e., `num_envs` steps).
* If you were overriding any of these following parameters in your config file, remove them
from the top-level config and follow the steps below:
* `gamma`: Define a new `extrinsic` reward signal and set it's `gamma` to your new gamma.
* `use_curiosity`, `curiosity_strength`, `curiosity_enc_size`: Define a `curiosity` reward signal
and set its `strength` to `curiosity_strength`, and `encoding_size` to `curiosity_enc_size`. Give it
the same `gamma` as your `extrinsic` signal to mimic previous behavior.
See [Reward Signals](Reward-Signals.md) for more information on defining reward signals.
* TensorBoards generated when running multiple environments in v0.8 are not comparable to those generated in
v0.9 in terms of step count. Multiply your v0.8 step count by `num_envs` for an approximate comparison.
You may need to change `max_steps` in your config as appropriate as well.
- If you were overriding any of these following parameters in your config file,
remove them from the top-level config and follow the steps below:
- `gamma`: Define a new `extrinsic` reward signal and set it's `gamma` to your
new gamma.
- `use_curiosity`, `curiosity_strength`, `curiosity_enc_size`: Define a
`curiosity` reward signal and set its `strength` to `curiosity_strength`,
and `encoding_size` to `curiosity_enc_size`. Give it the same `gamma` as
your `extrinsic` signal to mimic previous behavior. See
[Reward Signals](Reward-Signals.md) for more information on defining reward
signals.
- TensorBoards generated when running multiple environments in v0.8 are not
comparable to those generated in v0.9 in terms of step count. Multiply your
v0.8 step count by `num_envs` for an approximate comparison. You may need to
change `max_steps` in your config as appropriate as well.
## Migrating from ML-Agents toolkit v0.7 to v0.8
## Migrating from ML-Agents Toolkit v0.7 to v0.8
* We have split the Python packages into two separate packages `ml-agents` and `ml-agents-envs`.
* `--worker-id` option of `learn.py` has been removed, use `--base-port` instead if you'd like to run multiple instances of `learn.py`.
- We have split the Python packages into two separate packages `ml-agents` and
`ml-agents-envs`.
- `--worker-id` option of `learn.py` has been removed, use `--base-port` instead
if you'd like to run multiple instances of `learn.py`.
* If you are installing via PyPI, there is no change.
* If you intend to make modifications to `ml-agents` or `ml-agents-envs` please check the Installing for Development in the [Installation documentation](Installation.md).
## Migrating from ML-Agents toolkit v0.6 to v0.7
- If you are installing via PyPI, there is no change.
- If you intend to make modifications to `ml-agents` or `ml-agents-envs` please
check the Installing for Development in the
[Installation documentation](Installation.md).
## Migrating from ML-Agents Toolkit v0.6 to v0.7
* We no longer support TFS and are now using the [Unity Inference Engine](Unity-Inference-Engine.md)
A symmetric game is one in which opposing agents are equal in form, function and objective. Examples of symmetric games
are our Tennis and Soccer example environments. In reinforcement learning, this means both agents have the same observation and
action spaces and learn from the same reward function and so *they can share the same policy*. In asymmetric games,
this is not the case. An example of an asymmetric games are Hide and Seek. Agents in these
this is not the case. An example of an asymmetric game is our Strikers Vs Goalie example environment. Agents in these
types of games do not always have the same observation or action spaces and so sharing policy networks is not
necessarily ideal.
Note, in asymmetric games, the agents must have both different Behavior Names *and* different team IDs! Then, specify the trainer configuration
for each Behavior Name in your scene as you would normally, and remember to include the self-play hyperparameter hierarchy!
For examples of how to use this feature, you can see the trainer configurations and agent prefabs for our Tennis and Soccer environments.
Tennis and Soccer provide examples of symmetric games. To train an asymmetric game, specify trainer configurations for each of your behavior names
and include the self-play hyperparameter hierarchy in both.
For examples of how to use this feature, you can see the trainer configurations and agent prefabs for our Tennis, Soccer, and
Strikers Vs Goalie environments.
Tennis and Soccer provide examples of symmetric games and Strikers Vs Goalie provides an example of an asymmetric game.
## Best Practices Training with Self-Play
### Reward Signals
We make the assumption that the final reward in a trajectory corresponds to the outcome of an episode.
A final reward of +1 indicates winning, -1 indicates losing and 0 indicates a draw.
The ELO calculation (discussed below) depends on this final reward being either +1, 0, -1.
A final reward greater than 0 indicates winning, less than 0 indicates losing and 0 indicates a draw.
The final reward determines the result of an episode (win, loss, or draw) in the ELO calculation.
In problems that are too challenging to be solved by sparse rewards, it may be necessary to provide intermediate rewards to encourage useful instrumental behaviors.
For example, it may be difficult for a soccer agent to learn that kicking a ball into the net receives a reward because this sequence has a low probability
of occurring randomly. However, it will have a higher probability of occurring if the agent learns generally that kicking the ball has utility. So, we may be able
to speed up training by giving the agent intermediate reward for kicking the ball. However, we must be careful that the agent doesn't learn to undermine
its original objective of scoring goals e.g. if it scores a goal, the episode ends and it can no longer receive reward for kicking the ball. The behavior
that receives the most reward may be to keep the ball out of the net and to kick it indefinitely! To address this, we suggest
using a curriculum that allows the agents to learn the necessary intermediate behavior (i.e. colliding with a ball) and then
decays this reward signal to allow training on just the rewards of winning and losing. Please see our documentation on
how to use curriculum learning [here](./Training-Curriculum-Learning.md) and our SoccerTwos example environment.
### Save Steps
The `save_steps` parameter corresponds to the number of *trainer steps* between snapshots. For example, if `save_steps=10000` then a snapshot of the current policy will be saved every `10000` trainer steps. Note, trainer steps are counted per agent. For more information, please see the [migration doc](Migrating.md) after v0.13.
The `swap_steps` parameter corresponds to the number of *ghost steps* (not trainer steps) between swapping the opponents policy with a different snapshot.
A 'ghost step' refers to a step taken by an agent *that is following a fixed policy and not learning*. The reason for this distinction is that in asymmetric games,
we may have teams with an unequal number of agents e.g. a 2v1 scenario. The team with two agents collects
we may have teams with an unequal number of agents e.g. a 2v1 scenario like our Strikers Vs Goalie example environment. The team with two agents collects
twice as many agent steps per environment step as the team with one agent. Thus, these two values will need to be distinct to ensure that the same number
of trainer steps corresponds to the same number of opponent swaps for each team. The formula for `swap_steps` if
a user desires `x` swaps of a team with `num_agents` agents against an opponent team with `num_opponent_agents`
Ervin Teng
5 年前
