unity-tech-cn
/
ml-agents
45
14
派生 38
代码 合并请求 版本发布
浏览代码

Merge remote-tracking branch 'origin/master' into develop-add-fire

/develop/add-fire
Arthur Juliani 4 年前
当前提交
3769d943
共有 15 个文件被更改，包括 569 次插入 和 352 次删除
  1. 24
      Project/Assets/ML-Agents/Examples/Basic/Scripts/BasicController.cs
  2. 6
      com.unity.ml-agents/Runtime/Academy.cs
  3. 5
      com.unity.ml-agents/Runtime/Agent.cs
  4. 56
      docs/Background-Machine-Learning.md
  5. 33
      docs/Background-TensorFlow.md
  6. 23
      docs/Background-Unity.md
  7. 38
      docs/Glossary.md
  8. 22
      docs/Installation-Anaconda-Windows.md
  9. 2
      docs/Learning-Environment-Create-New.md
  10. 9
      docs/Limitations.md
  11. 687
      docs/Migrating.md
  12. 2
      docs/Training-on-Amazon-Web-Service.md
  13. 2
      docs/Training-on-Microsoft-Azure.md
  14. 4
      ml-agents/mlagents/trainers/ghost/trainer.py
  15. 8
      ml-agents/mlagents/trainers/tests/test_simple_rl.py

24
Project/Assets/ML-Agents/Examples/Basic/Scripts/BasicController.cs
查看文件


using UnityEngine;
using UnityEngine.SceneManagement;
using MLAgents;
/// <summary>

Agent m_Agent;
ResetAgent();
m_Position = 10;
transform.position = new Vector3(m_Position - 10f, 0f, 0f);
smallGoal.transform.position = new Vector3(k_SmallGoalPosition - 10f, 0f, 0f);
largeGoal.transform.position = new Vector3(k_LargeGoalPosition - 10f, 0f, 0f);
}
/// <summary>

}
public void ResetAgent()
{
m_Position = 10;
smallGoal.transform.position = new Vector3(k_SmallGoalPosition - 10f, 0f, 0f);
largeGoal.transform.position = new Vector3(k_LargeGoalPosition - 10f, 0f, 0f);
{
// This is a very inefficient way to reset the scene. Used here for testing.
SceneManager.LoadScene(SceneManager.GetActiveScene().name);
m_Agent = null; // LoadScene only takes effect at the next Update.
// We set the Agent to null to avoid using the Agent before the reload
}
public void FixedUpdate()

void WaitTimeInference()
{
if (m_Agent == null)
{
return;
}
m_Agent.RequestDecision();
m_Agent?.RequestDecision();
}
else
{

ApplyAction(m_Agent.GetAction());
m_TimeSinceDecision = 0f;
m_Agent.RequestDecision();
m_Agent?.RequestDecision();
}
else
{

6
com.unity.ml-agents/Runtime/Academy.cs
查看文件


// Don't show this object in the hierarchy
m_StepperObject.hideFlags = HideFlags.HideInHierarchy;
m_FixedUpdateStepper = m_StepperObject.AddComponent<AcademyFixedUpdateStepper>();
try
{
// This try-catch is because DontDestroyOnLoad cannot be used in Editor Tests
GameObject.DontDestroyOnLoad(m_StepperObject);
}
catch {}
}
/// <summary>

5
com.unity.ml-agents/Runtime/Agent.cs
查看文件


void NotifyAgentDone(DoneReason doneReason)
{
if (m_Info.done)
{
// The Agent was already marked as Done and should not be notified again
return;
}
m_Info.episodeId = m_EpisodeId;
m_Info.reward = m_Reward;
m_Info.done = true;

56
docs/Background-Machine-Learning.md
查看文件


# Background: Machine Learning
Given that a number of users of the ML-Agents toolkit might not have a formal
Given that a number of users of the ML-Agents Toolkit might not have a formal
understanding of the ML-Agents toolkit. However, we will not attempt to provide
understanding of the ML-Agents Toolkit. However, we will not attempt to provide
a thorough treatment of machine learning as there are fantastic resources
online.

## Unsupervised Learning
The goal of [unsupervised
learning](https://en.wikipedia.org/wiki/Unsupervised_learning) is to group or
cluster similar items in a data set. For example, consider the players of a
game. We may want to group the players depending on how engaged they are with
the game. This would enable us to target different groups (e.g. for
highly-engaged players we might invite them to be beta testers for new features,
while for unengaged players we might email them helpful tutorials). Say that we
wish to split our players into two groups. We would first define basic
attributes of the players, such as the number of hours played, total money spent
on in-app purchases and number of levels completed. We can then feed this data
set (three attributes for every player) to an unsupervised learning algorithm
where we specify the number of groups to be two. The algorithm would then split
the data set of players into two groups where the players within each group
would be similar to each other. Given the attributes we used to describe each
player, in this case, the output would be a split of all the players into two
groups, where one group would semantically represent the engaged players and the
second group would semantically represent the unengaged players.
The goal of
[unsupervised learning](https://en.wikipedia.org/wiki/Unsupervised_learning) is
to group or cluster similar items in a data set. For example, consider the
players of a game. We may want to group the players depending on how engaged
they are with the game. This would enable us to target different groups (e.g.
for highly-engaged players we might invite them to be beta testers for new
features, while for unengaged players we might email them helpful tutorials).
Say that we wish to split our players into two groups. We would first define
basic attributes of the players, such as the number of hours played, total money
spent on in-app purchases and number of levels completed. We can then feed this
data set (three attributes for every player) to an unsupervised learning
algorithm where we specify the number of groups to be two. The algorithm would
then split the data set of players into two groups where the players within each
group would be similar to each other. Given the attributes we used to describe
each player, in this case, the output would be a split of all the players into
two groups, where one group would semantically represent the engaged players and
the second group would semantically represent the unengaged players.
With unsupervised learning, we did not provide specific examples of which
players are considered engaged and which are considered unengaged. We just

to achieve good performance.
We now switch to reinforcement learning, the third class of machine learning
algorithms, and arguably the one most relevant for the ML-Agents toolkit.
algorithms, and arguably the one most relevant for the ML-Agents Toolkit.
## Reinforcement Learning

one can view a non-playable character (NPC) as a virtual robot, with its own
observations about the environment, its own set of actions and a specific
objective. Thus it is natural to explore how we can train behaviors within Unity
using reinforcement learning. This is precisely what the ML-Agents toolkit
using reinforcement learning. This is precisely what the ML-Agents Toolkit
training character behaviors using the ML-Agents toolkit.
training character behaviors using the ML-Agents Toolkit.
<p align="center">
<a href="http://www.youtube.com/watch?feature=player_embedded&v=fiQsmdwEGT8" target="_blank">

data, while the inference phase involves applying this model to new, previously
unseen, data. More specifically:
* For our unsupervised learning example, the training phase learns the optimal
- For our unsupervised learning example, the training phase learns the optimal
* For our supervised learning example, the training phase learns the mapping
- For our supervised learning example, the training phase learns the mapping
* For our reinforcement learning example, the training phase learns the optimal
- For our reinforcement learning example, the training phase learns the optimal
policy through guided trials, and in the inference phase, the agent observes
and tales actions in the wild using its learned policy.

More specifically, they can be used to solve both attribute and model selection
tasks. Deep learning has gained popularity in recent years due to its
outstanding performance on several challenging machine learning tasks. One
example is [AlphaGo](https://en.wikipedia.org/wiki/AlphaGo), a [computer
Go](https://en.wikipedia.org/wiki/Computer_Go) program, that leverages deep
learning, that was able to beat Lee Sedol (a Go world champion).
example is [AlphaGo](https://en.wikipedia.org/wiki/AlphaGo), a
[computer Go](https://en.wikipedia.org/wiki/Computer_Go) program, that leverages
deep learning, that was able to beat Lee Sedol (a Go world champion).
A key characteristic of deep learning algorithms is their ability learn very
complex functions from large amounts of training data. This makes them a natural

33
docs/Background-TensorFlow.md
查看文件


# Background: TensorFlow
As discussed in our
[machine learning background page](Background-Machine-Learning.md),
many of the algorithms we provide in the
ML-Agents toolkit leverage some form of deep learning. More specifically, our
implementations are built on top of the open-source library
[TensorFlow](https://www.tensorflow.org/). This means that the models produced
by the ML-Agents toolkit are (currently) in a format only understood by
TensorFlow. In this page we provide a brief overview of TensorFlow, in addition
to TensorFlow-related tools that we leverage within the ML-Agents toolkit.
[machine learning background page](Background-Machine-Learning.md), many of the
algorithms we provide in the ML-Agents Toolkit leverage some form of deep
learning. More specifically, our implementations are built on top of the
open-source library [TensorFlow](https://www.tensorflow.org/). In this page we
provide a brief overview of TensorFlow, in addition to TensorFlow-related tools
that we leverage within the ML-Agents Toolkit.
## TensorFlow

a desktop, server, or mobile device. Within the ML-Agents toolkit, when you
train the behavior of an agent, the output is a TensorFlow model (.nn) file
that you can then associate with an Agent. Unless you implement a new
algorithm, the use of TensorFlow is mostly abstracted away and behind the
scenes.
a desktop, server, or mobile device. Within the ML-Agents Toolkit, when you
train the behavior of an agent, the output is a TensorFlow model (.nn) file that
you can then associate with an Agent. Unless you implement a new algorithm, the
use of TensorFlow is mostly abstracted away and behind the scenes.
## TensorBoard

It allows the visualization of certain agent attributes (e.g. reward) throughout
training which can be helpful in both building intuitions for the different
hyperparameters and setting the optimal values for your Unity environment. We
provide more details on setting the hyperparameters in later parts of the
documentation, but, in the meantime, if you are unfamiliar with TensorBoard we
recommend our guide on [using TensorBoard with ML-Agents](Using-Tensorboard.md) or
this [tutorial](https://github.com/dandelionmane/tf-dev-summit-tensorboard-tutorial).
provide more details on setting the hyperparameters in the
[Training ML-Agents](Training-ML-Agents.md) page. If you are unfamiliar with
TensorBoard we recommend our guide on
[using TensorBoard with ML-Agents](Using-Tensorboard.md) or this
[tutorial](https://github.com/dandelionmane/tf-dev-summit-tensorboard-tutorial).

23
docs/Background-Unity.md
查看文件


and [Tutorials page](https://unity3d.com/learn/tutorials). The
[Roll-a-ball tutorial](https://unity3d.com/learn/tutorials/s/roll-ball-tutorial)
is a fantastic resource to learn all the basic concepts of Unity to get started
with the ML-Agents toolkit:
with the ML-Agents Toolkit:
* [Editor](https://docs.unity3d.com/Manual/UsingTheEditor.html)
* [Interface](https://docs.unity3d.com/Manual/LearningtheInterface.html)
* [Scene](https://docs.unity3d.com/Manual/CreatingScenes.html)
* [GameObject](https://docs.unity3d.com/Manual/GameObjects.html)
* [Rigidbody](https://docs.unity3d.com/ScriptReference/Rigidbody.html)
* [Camera](https://docs.unity3d.com/Manual/Cameras.html)
* [Scripting](https://docs.unity3d.com/Manual/ScriptingSection.html)
* [Physics](https://docs.unity3d.com/Manual/PhysicsSection.html)
* [Ordering of event functions](https://docs.unity3d.com/Manual/ExecutionOrder.html)
- [Editor](https://docs.unity3d.com/Manual/UsingTheEditor.html)
- [Interface](https://docs.unity3d.com/Manual/LearningtheInterface.html)
- [Scene](https://docs.unity3d.com/Manual/CreatingScenes.html)
- [GameObject](https://docs.unity3d.com/Manual/GameObjects.html)
- [Rigidbody](https://docs.unity3d.com/ScriptReference/Rigidbody.html)
- [Camera](https://docs.unity3d.com/Manual/Cameras.html)
- [Scripting](https://docs.unity3d.com/Manual/ScriptingSection.html)
- [Physics](https://docs.unity3d.com/Manual/PhysicsSection.html)
- [Ordering of event functions](https://docs.unity3d.com/Manual/ExecutionOrder.html)
* [Prefabs](https://docs.unity3d.com/Manual/Prefabs.html)
- [Prefabs](https://docs.unity3d.com/Manual/Prefabs.html)

38
docs/Glossary.md
查看文件


# ML-Agents Toolkit Glossary
* **Academy** - Singleton object which controls timing, reset, and
- **Academy** - Singleton object which controls timing, reset, and
* **Action** - The carrying-out of a decision on the part of an agent within the
- **Action** - The carrying-out of a decision on the part of an agent within the
* **Agent** - Unity Component which produces observations and takes actions in
- **Agent** - Unity Component which produces observations and takes actions in
* **Policy** - The decision making mechanism, typically a neural network model.
* **Decision** - The specification produced by a Policy for an action to be
- **Policy** - The decision making mechanism, typically a neural network model.
- **Decision** - The specification produced by a Policy for an action to be
* **Editor** - The Unity Editor, which may include any pane (e.g. Hierarchy,
- **Editor** - The Unity Editor, which may include any pane (e.g. Hierarchy,
* **Environment** - The Unity scene which contains Agents.
* **FixedUpdate** - Unity method called each time the game engine is
stepped. ML-Agents logic should be placed here.
* **Frame** - An instance of rendering the main camera for the display.
- **Environment** - The Unity scene which contains Agents.
- **FixedUpdate** - Unity method called each time the game engine is stepped.
ML-Agents logic should be placed here.
- **Frame** - An instance of rendering the main camera for the display.
* **Observation** - Partial information describing the state of the environment
- **Observation** - Partial information describing the state of the environment
* **Policy** - Function for producing decisions from observations.
* **Reward** - Signal provided at every step used to indicate desirability of an
- **Policy** - Function for producing decisions from observations.
- **Reward** - Signal provided at every step used to indicate desirability of an
* **State** - The underlying properties of the environment (including all agents
- **State** - The underlying properties of the environment (including all agents
* **Step** - Corresponds to each `FixedUpdate` call of the game engine. Is the
- **Step** - Corresponds to each `FixedUpdate` call of the game engine. Is the
* **Update** - Unity function called each time a frame is rendered. ML-Agents
- **Update** - Unity function called each time a frame is rendered. ML-Agents
* **External Coordinator** - ML-Agents class responsible for communication with
- **External Coordinator** - ML-Agents class responsible for communication with
* **Trainer** - Python class which is responsible for training a given
group of Agents.
- **Trainer** - Python class which is responsible for training a given group of
Agents.

22
docs/Installation-Anaconda-Windows.md
查看文件


:warning: **Note:** We no longer use this guide ourselves and so it may not work
correctly. We've decided to keep it up just in case it is helpful to you.
The ML-Agents toolkit supports Windows 10. While it might be possible to run the
ML-Agents toolkit using other versions of Windows, it has not been tested on
other versions. Furthermore, the ML-Agents toolkit has not been tested on a
The ML-Agents Toolkit supports Windows 10. While it might be possible to run the
ML-Agents Toolkit using other versions of Windows, it has not been tested on
other versions. Furthermore, the ML-Agents Toolkit has not been tested on a
To use the ML-Agents toolkit, you install Python and the required Python
To use the ML-Agents Toolkit, you install Python and the required Python
ML-Agents toolkit. However, training on a GPU might be required by future
ML-Agents Toolkit. However, training on a GPU might be required by future
versions and features.
## Step 1: Install Python via Anaconda

## Step 2: Setup and Activate a New Conda Environment
You will create a new [Conda environment](https://conda.io/docs/) to be used
with the ML-Agents toolkit. This means that all the packages that you install
with the ML-Agents Toolkit. This means that all the packages that you install
are localized to just this environment. It will not affect any other
installation of Python or other environments. Whenever you want to run
ML-Agents, you will need activate this Conda environment.

## Step 3: Install Required Python Packages
The ML-Agents toolkit depends on a number of Python packages. Use `pip` to
The ML-Agents Toolkit depends on a number of Python packages. Use `pip` to
install these Python dependencies.
If you haven't already, clone the ML-Agents Toolkit Github repository to your

```
This will complete the installation of all the required Python packages to run
the ML-Agents toolkit.
the ML-Agents Toolkit.
Sometimes on Windows, when you use pip to install certain Python packages, the
pip will get stuck when trying to read the cache of the package. If you see

## (Optional) Step 4: GPU Training using The ML-Agents Toolkit
GPU is not required for the ML-Agents toolkit and won't speed up the PPO
GPU is not required for the ML-Agents Toolkit and won't speed up the PPO
Currently for the ML-Agents toolkit, only CUDA v9.0 and cuDNN v7.0.5 is
Currently for the ML-Agents Toolkit, only CUDA v9.0 and cuDNN v7.0.5 is
supported.
### Install Nvidia CUDA toolkit

libraries, debugging and optimization tools, a C/C++ (Step Visual Studio 2017)
compiler and a runtime library and is needed to run the ML-Agents toolkit. In
compiler and a runtime library and is needed to run the ML-Agents Toolkit. In
this guide, we are using version
[9.0.176](https://developer.nvidia.com/compute/cuda/9.0/Prod/network_installers/cuda_9.0.176_win10_network-exe)).

2
docs/Learning-Environment-Create-New.md
查看文件


## Overview
Using the ML-Agents toolkit in a Unity project involves the following basic
Using the ML-Agents Toolkit in a Unity project involves the following basic
steps:
1. Create an environment for your agents to live in. An environment can range

9
docs/Limitations.md
查看文件


# Limitations
See the package-specific Limitations pages:
* [Unity `com.unity.mlagents` package](../com.unity.ml-agents/Documentation~/com.unity.ml-agents.md)
* [`mlagents` Python package](../ml-agents/README.md)
* [`mlagents_envs` Python package](../ml-agents-envs/README.md)
* [`gym_unity` Python package](../gym-unity/README.md)
- [Unity `com.unity.mlagents` package](../com.unity.ml-agents/Documentation~/com.unity.ml-agents.md)
- [`mlagents` Python package](../ml-agents/README.md)
- [`mlagents_envs` Python package](../ml-agents-envs/README.md)
- [`gym_unity` Python package](../gym-unity/README.md)

687
docs/Migrating.md
查看文件


# Upgrading
The C# editor code and python trainer code are not compatible between releases. This means that if you upgrade one, you *must* upgrade the other as well. If you experience new errors or unable to connect to training after updating, please double-check that the versions are in the same.
The versions can be found in
* `Academy.k_ApiVersion` in Academy.cs ([example](https://github.com/Unity-Technologies/ml-agents/blob/b255661084cb8f701c716b040693069a3fb9a257/UnitySDK/Assets/ML-Agents/Scripts/Academy.cs#L95))
* `UnityEnvironment.API_VERSION` in environment.py ([example](https://github.com/Unity-Technologies/ml-agents/blob/b255661084cb8f701c716b040693069a3fb9a257/ml-agents-envs/mlagents/envs/environment.py#L45))
The C# editor code and python trainer code are not compatible between releases.
This means that if you upgrade one, you _must_ upgrade the other as well. If you
experience new errors or unable to connect to training after updating, please
double-check that the versions are in the same. The versions can be found in
- `Academy.k_ApiVersion` in Academy.cs
([example](https://github.com/Unity-Technologies/ml-agents/blob/b255661084cb8f701c716b040693069a3fb9a257/UnitySDK/Assets/ML-Agents/Scripts/Academy.cs#L95))
- `UnityEnvironment.API_VERSION` in environment.py
([example](https://github.com/Unity-Technologies/ml-agents/blob/b255661084cb8f701c716b040693069a3fb9a257/ml-agents-envs/mlagents/envs/environment.py#L45))
# Migrating

* The `--load` and `--train` command-line flags have been deprecated and replaced with `--resume` and `--inference`.
* Running with the same `--run-id` twice will now throw an error.
* The `play_against_current_self_ratio` self-play trainer hyperparameter has been renamed to `play_against_latest_model_ratio`
* Removed the multi-agent gym option from the gym wrapper. For multi-agent scenarios, use the [Low Level Python API](Python-API.md).
* The low level Python API has changed. You can look at the document [Low Level Python API documentation](Python-API.md) for more information. If you use `mlagents-learn` for training, this should be a transparent change.
* The obsolete `Agent` methods `GiveModel`, `Done`, `InitializeAgent`, `AgentAction` and `AgentReset` have been removed.
* The signature of `Agent.Heuristic()` was changed to take a `float[]` as a parameter, instead of returning the array. This was done to prevent a common source of error where users would return arrays of the wrong size.
- The `--load` and `--train` command-line flags have been deprecated and
replaced with `--resume` and `--inference`.
- Running with the same `--run-id` twice will now throw an error.
- The `play_against_current_self_ratio` self-play trainer hyperparameter has
been renamed to `play_against_latest_model_ratio`
- Removed the multi-agent gym option from the gym wrapper. For multi-agent
scenarios, use the [Low Level Python API](Python-API.md).
- The low level Python API has changed. You can look at the document
[Low Level Python API documentation](Python-API.md) for more information. If
you use `mlagents-learn` for training, this should be a transparent change.
- The obsolete `Agent` methods `GiveModel`, `Done`, `InitializeAgent`,
`AgentAction` and `AgentReset` have been removed.
- The signature of `Agent.Heuristic()` was changed to take a `float[]` as a
parameter, instead of returning the array. This was done to prevent a common
source of error where users would return arrays of the wrong size.
* Replace the `--load` flag with `--resume` when calling `mlagents-learn`, and don't use the `--train` flag as training
will happen by default. To run without training, use `--inference`.
* To force-overwrite files from a pre-existing run, add the `--force` command-line flag.
* The Jupyter notebooks have been removed from the repository.
* `Academy.FloatProperties` was removed.
* `Academy.RegisterSideChannel` and `Academy.UnregisterSideChannel` were removed.
* Replace `Academy.FloatProperties` with `SideChannelUtils.GetSideChannel<FloatPropertiesChannel>()`.
* Replace `Academy.RegisterSideChannel` with `SideChannelUtils.RegisterSideChannel()`.
* Replace `Academy.UnregisterSideChannel` with `SideChannelUtils.UnregisterSideChannel`.
* If your Agent class overrides `Heuristic()`, change the signature to `public override void Heuristic(float[] actionsOut)` and assign values to `actionsOut` instead of returning an array.
- Replace the `--load` flag with `--resume` when calling `mlagents-learn`, and
don't use the `--train` flag as training will happen by default. To run
without training, use `--inference`.
- To force-overwrite files from a pre-existing run, add the `--force`
command-line flag.
- The Jupyter notebooks have been removed from the repository.
- `Academy.FloatProperties` was removed.
- `Academy.RegisterSideChannel` and `Academy.UnregisterSideChannel` were
removed.
- Replace `Academy.FloatProperties` with
`SideChannelUtils.GetSideChannel<FloatPropertiesChannel>()`.
- Replace `Academy.RegisterSideChannel` with
`SideChannelUtils.RegisterSideChannel()`.
- Replace `Academy.UnregisterSideChannel` with
`SideChannelUtils.UnregisterSideChannel`.
- If your Agent class overrides `Heuristic()`, change the signature to
`public override void Heuristic(float[] actionsOut)` and assign values to
`actionsOut` instead of returning an array.
* The `Agent.CollectObservations()` virtual method now takes as input a `VectorSensor` sensor as argument. The `Agent.AddVectorObs()` methods were removed.
* The `SetMask` was renamed to `SetMask` method must now be called on the `DiscreteActionMasker` argument of the `CollectDiscreteActionMasks` virtual method.
* We consolidated our API for `DiscreteActionMasker`. `SetMask` takes two arguments : the branch index and the list of masked actions for that branch.
* The `Monitor` class has been moved to the Examples Project. (It was prone to errors during testing)
* The `MLAgents.Sensors` namespace has been introduced. All sensors classes are part of the `MLAgents.Sensors` namespace.
* The `MLAgents.SideChannels` namespace has been introduced. All side channel classes are part of the `MLAgents.SideChannels` namespace.
* The interface for `RayPerceptionSensor.PerceiveStatic()` was changed to take an input class and write to an output class, and the method was renamed to `Perceive()`.
* The `SetMask` method must now be called on the `DiscreteActionMasker` argument of the `CollectDiscreteActionMasks` method.
* The method `GetStepCount()` on the Agent class has been replaced with the property getter `StepCount`
* The `--multi-gpu` option has been removed temporarily.
* `AgentInfo.actionMasks` has been renamed to `AgentInfo.discreteActionMasks`.
* `BrainParameters` and `SpaceType` have been removed from the public API
* `BehaviorParameters` have been removed from the public API.
* `DecisionRequester` has been made internal (you can still use the DecisionRequesterComponent from the inspector). `RepeatAction` was renamed `TakeActionsBetweenDecisions` for clarity.
* The following methods in the `Agent` class have been renamed. The original method names will be removed in a later release:
* `InitializeAgent()` was renamed to `Initialize()`
* `AgentAction()` was renamed to `OnActionReceived()`
* `AgentReset()` was renamed to `OnEpsiodeBegin()`
* `Done()` was renamed to `EndEpisode()`
* `GiveModel()` was renamed to `SetModel()`
* The `IFloatProperties` interface has been removed.
* The interface for SideChannels was changed:
* In C#, `OnMessageReceived` now takes a `IncomingMessage` argument, and `QueueMessageToSend` takes an `OutgoingMessage` argument.
* In python, `on_message_received` now takes a `IncomingMessage` argument, and `queue_message_to_send` takes an `OutgoingMessage` argument.
* Automatic stepping for Academy is now controlled from the AutomaticSteppingEnabled property.
- The `Agent.CollectObservations()` virtual method now takes as input a
`VectorSensor` sensor as argument. The `Agent.AddVectorObs()` methods were
removed.
- The `SetMask` was renamed to `SetMask` method must now be called on the
`DiscreteActionMasker` argument of the `CollectDiscreteActionMasks` virtual
method.
- We consolidated our API for `DiscreteActionMasker`. `SetMask` takes two
arguments : the branch index and the list of masked actions for that branch.
- The `Monitor` class has been moved to the Examples Project. (It was prone to
errors during testing)
- The `MLAgents.Sensors` namespace has been introduced. All sensors classes are
part of the `MLAgents.Sensors` namespace.
- The `MLAgents.SideChannels` namespace has been introduced. All side channel
classes are part of the `MLAgents.SideChannels` namespace.
- The interface for `RayPerceptionSensor.PerceiveStatic()` was changed to take
an input class and write to an output class, and the method was renamed to
`Perceive()`.
- The `SetMask` method must now be called on the `DiscreteActionMasker` argument
of the `CollectDiscreteActionMasks` method.
- The method `GetStepCount()` on the Agent class has been replaced with the
property getter `StepCount`
- The `--multi-gpu` option has been removed temporarily.
- `AgentInfo.actionMasks` has been renamed to `AgentInfo.discreteActionMasks`.
- `BrainParameters` and `SpaceType` have been removed from the public API
- `BehaviorParameters` have been removed from the public API.
- `DecisionRequester` has been made internal (you can still use the
DecisionRequesterComponent from the inspector). `RepeatAction` was renamed
`TakeActionsBetweenDecisions` for clarity.
- The following methods in the `Agent` class have been renamed. The original
method names will be removed in a later release:
- `InitializeAgent()` was renamed to `Initialize()`
- `AgentAction()` was renamed to `OnActionReceived()`
- `AgentReset()` was renamed to `OnEpsiodeBegin()`
- `Done()` was renamed to `EndEpisode()`
- `GiveModel()` was renamed to `SetModel()`
- The `IFloatProperties` interface has been removed.
- The interface for SideChannels was changed:
- In C#, `OnMessageReceived` now takes a `IncomingMessage` argument, and
`QueueMessageToSend` takes an `OutgoingMessage` argument.
- In python, `on_message_received` now takes a `IncomingMessage` argument, and
`queue_message_to_send` takes an `OutgoingMessage` argument.
- Automatic stepping for Academy is now controlled from the
AutomaticSteppingEnabled property.
* Add the `using MLAgents.Sensors;` in addition to `using MLAgents;` on top of your Agent's script.
* Replace your Agent's implementation of `CollectObservations()` with `CollectObservations(VectorSensor sensor)`. In addition, replace all calls to `AddVectorObs()` with `sensor.AddObservation()` or `sensor.AddOneHotObservation()` on the `VectorSensor` passed as argument.
* Replace your calls to `SetActionMask` on your Agent to `DiscreteActionMasker.SetActionMask` in `CollectDiscreteActionMasks`.
* If you call `RayPerceptionSensor.PerceiveStatic()` manually, add your inputs to a `RayPerceptionInput`. To get the previous float array output,
iterate through `RayPerceptionOutput.rayOutputs` and call `RayPerceptionOutput.RayOutput.ToFloatArray()`.
* Replace all calls to `Agent.GetStepCount()` with `Agent.StepCount`
* We strongly recommend replacing the following methods with their new equivalent as they will be removed in a later release:
* `InitializeAgent()` to `Initialize()`
* `AgentAction()` to `OnActionReceived()`
* `AgentReset()` to `OnEpisodeBegin()`
* `Done()` to `EndEpisode()`
* `GiveModel()` to `SetModel()`
* Replace `IFloatProperties` variables with `FloatPropertiesChannel` variables.
* If you implemented custom `SideChannels`, update the signatures of your methods, and add your data to the `OutgoingMessage` or read it from the `IncomingMessage`.
* Replace calls to Academy.EnableAutomaticStepping()/DisableAutomaticStepping() with Academy.AutomaticSteppingEnabled = true/false.
- Add the `using MLAgents.Sensors;` in addition to `using MLAgents;` on top of
your Agent's script.
- Replace your Agent's implementation of `CollectObservations()` with
`CollectObservations(VectorSensor sensor)`. In addition, replace all calls to
`AddVectorObs()` with `sensor.AddObservation()` or
`sensor.AddOneHotObservation()` on the `VectorSensor` passed as argument.
- Replace your calls to `SetActionMask` on your Agent to
`DiscreteActionMasker.SetActionMask` in `CollectDiscreteActionMasks`.
- If you call `RayPerceptionSensor.PerceiveStatic()` manually, add your inputs
to a `RayPerceptionInput`. To get the previous float array output, iterate
through `RayPerceptionOutput.rayOutputs` and call
`RayPerceptionOutput.RayOutput.ToFloatArray()`.
- Replace all calls to `Agent.GetStepCount()` with `Agent.StepCount`
- We strongly recommend replacing the following methods with their new
equivalent as they will be removed in a later release:
- `InitializeAgent()` to `Initialize()`
- `AgentAction()` to `OnActionReceived()`
- `AgentReset()` to `OnEpisodeBegin()`
- `Done()` to `EndEpisode()`
- `GiveModel()` to `SetModel()`
- Replace `IFloatProperties` variables with `FloatPropertiesChannel` variables.
- If you implemented custom `SideChannels`, update the signatures of your
methods, and add your data to the `OutgoingMessage` or read it from the
`IncomingMessage`.
- Replace calls to Academy.EnableAutomaticStepping()/DisableAutomaticStepping()
with Academy.AutomaticSteppingEnabled = true/false.
* The `UnitySDK` folder has been split into a Unity Package (`com.unity.ml-agents`) and an examples project (`Project`). Please follow the [Installation Guide](Installation.md) to get up and running with this new repo structure.
* Several changes were made to how agents are reset and marked as done:
* Calling `Done()` on the Agent will now reset it immediately and call the `AgentReset` virtual method. (This is to simplify the previous logic in which the Agent had to wait for the next `EnvironmentStep` to reset)
* The "Reset on Done" setting in AgentParameters was removed; this is now effectively always true. `AgentOnDone` virtual method on the Agent has been removed.
* The `Decision Period` and `On Demand decision` checkbox have been removed from the Agent. On demand decision is now the default (calling `RequestDecision` on the Agent manually.)
* The Academy class was changed to a singleton, and its virtual methods were removed.
* Trainer steps are now counted per-Agent, not per-environment as in previous versions. For instance, if you have 10 Agents in the scene, 20 environment steps now corresponds to 200 steps as printed in the terminal and in Tensorboard.
* Curriculum config files are now YAML formatted and all curricula for a training run are combined into a single file.
* The `--num-runs` command-line option has been removed from `mlagents-learn`.
* Several fields on the Agent were removed or made private in order to simplify the interface.
* The `agentParameters` field of the Agent has been removed. (Contained only `maxStep` information)
* `maxStep` is now a public field on the Agent. (Was moved from `agentParameters`)
* The `Info` field of the Agent has been made private. (Was only used internally and not meant to be modified outside of the Agent)
* The `GetReward()` method on the Agent has been removed. (It was being confused with `GetCumulativeReward()`)
* The `AgentAction` struct no longer contains a `value` field. (Value estimates were not set during inference)
* The `GetValueEstimate()` method on the Agent has been removed.
* The `UpdateValueAction()` method on the Agent has been removed.
* The deprecated `RayPerception3D` and `RayPerception2D` classes were removed, and the `legacyHitFractionBehavior` argument was removed from `RayPerceptionSensor.PerceiveStatic()`.
* RayPerceptionSensor was inconsistent in how it handle scale on the Agent's transform. It now scales the ray length and sphere size for casting as the transform's scale changes.
- The `UnitySDK` folder has been split into a Unity Package
(`com.unity.ml-agents`) and an examples project (`Project`). Please follow the
[Installation Guide](Installation.md) to get up and running with this new repo
structure.
- Several changes were made to how agents are reset and marked as done:
- Calling `Done()` on the Agent will now reset it immediately and call the
`AgentReset` virtual method. (This is to simplify the previous logic in
which the Agent had to wait for the next `EnvironmentStep` to reset)
- The "Reset on Done" setting in AgentParameters was removed; this is now
effectively always true. `AgentOnDone` virtual method on the Agent has been
removed.
- The `Decision Period` and `On Demand decision` checkbox have been removed from
the Agent. On demand decision is now the default (calling `RequestDecision` on
the Agent manually.)
- The Academy class was changed to a singleton, and its virtual methods were
removed.
- Trainer steps are now counted per-Agent, not per-environment as in previous
versions. For instance, if you have 10 Agents in the scene, 20 environment
steps now corresponds to 200 steps as printed in the terminal and in
Tensorboard.
- Curriculum config files are now YAML formatted and all curricula for a
training run are combined into a single file.
- The `--num-runs` command-line option has been removed from `mlagents-learn`.
- Several fields on the Agent were removed or made private in order to simplify
the interface.
- The `agentParameters` field of the Agent has been removed. (Contained only
`maxStep` information)
- `maxStep` is now a public field on the Agent. (Was moved from
`agentParameters`)
- The `Info` field of the Agent has been made private. (Was only used
internally and not meant to be modified outside of the Agent)
- The `GetReward()` method on the Agent has been removed. (It was being
confused with `GetCumulativeReward()`)
- The `AgentAction` struct no longer contains a `value` field. (Value
estimates were not set during inference)
- The `GetValueEstimate()` method on the Agent has been removed.
- The `UpdateValueAction()` method on the Agent has been removed.
- The deprecated `RayPerception3D` and `RayPerception2D` classes were removed,
and the `legacyHitFractionBehavior` argument was removed from
`RayPerceptionSensor.PerceiveStatic()`.
- RayPerceptionSensor was inconsistent in how it handle scale on the Agent's
transform. It now scales the ray length and sphere size for casting as the
transform's scale changes.
* Follow the instructions on how to install the `com.unity.ml-agents` package into your project in the [Installation Guide](Installation.md).
* If your Agent implemented `AgentOnDone` and did not have the checkbox `Reset On Done` checked in the inspector, you must call the code that was in `AgentOnDone` manually.
* If you give your Agent a reward or penalty at the end of an episode (e.g. for reaching a goal or falling off of a platform), make sure you call `AddReward()` or `SetReward()` *before* calling `Done()`. Previously, the order didn't matter.
* If you were not using `On Demand Decision` for your Agent, you **must** add a `DecisionRequester` component to your Agent GameObject and set its `Decision Period` field to the old `Decision Period` of the Agent.
* If you have a class that inherits from Academy:
* If the class didn't override any of the virtual methods and didn't store any additional data, you can just remove the old script from the scene.
* If the class had additional data, create a new MonoBehaviour and store the data in the new MonoBehaviour instead.
* If the class overrode the virtual methods, create a new MonoBehaviour and move the logic to it:
* Move the InitializeAcademy code to MonoBehaviour.OnAwake
* Move the AcademyStep code to MonoBehaviour.FixedUpdate
* Move the OnDestroy code to MonoBehaviour.OnDestroy.
* Move the AcademyReset code to a new method and add it to the Academy.OnEnvironmentReset action.
* Multiply `max_steps` and `summary_freq` in your `trainer_config.yaml` by the number of Agents in the scene.
* Combine curriculum configs into a single file. See [the WallJump curricula](../config/curricula/wall_jump.yaml) for an example of the new curriculum config format.
A tool like https://www.json2yaml.com may be useful to help with the conversion.
* If you have a model trained which uses RayPerceptionSensor and has non-1.0 scale in the Agent's transform, it must be retrained.
- Follow the instructions on how to install the `com.unity.ml-agents` package
into your project in the [Installation Guide](Installation.md).
- If your Agent implemented `AgentOnDone` and did not have the checkbox
`Reset On Done` checked in the inspector, you must call the code that was in
`AgentOnDone` manually.
- If you give your Agent a reward or penalty at the end of an episode (e.g. for
reaching a goal or falling off of a platform), make sure you call
`AddReward()` or `SetReward()` _before_ calling `Done()`. Previously, the
order didn't matter.
- If you were not using `On Demand Decision` for your Agent, you **must** add a
`DecisionRequester` component to your Agent GameObject and set its
`Decision Period` field to the old `Decision Period` of the Agent.
- If you have a class that inherits from Academy:
- If the class didn't override any of the virtual methods and didn't store any
additional data, you can just remove the old script from the scene.
- If the class had additional data, create a new MonoBehaviour and store the
data in the new MonoBehaviour instead.
- If the class overrode the virtual methods, create a new MonoBehaviour and
move the logic to it:
- Move the InitializeAcademy code to MonoBehaviour.OnAwake
- Move the AcademyStep code to MonoBehaviour.FixedUpdate
- Move the OnDestroy code to MonoBehaviour.OnDestroy.
- Move the AcademyReset code to a new method and add it to the
Academy.OnEnvironmentReset action.
- Multiply `max_steps` and `summary_freq` in your `trainer_config.yaml` by the
number of Agents in the scene.
- Combine curriculum configs into a single file. See
[the WallJump curricula](../config/curricula/wall_jump.yaml) for an example of
the new curriculum config format. A tool like https://www.json2yaml.com may be
useful to help with the conversion.
- If you have a model trained which uses RayPerceptionSensor and has non-1.0
scale in the Agent's transform, it must be retrained.
## Migrating from ML-Agents toolkit v0.12.0 to v0.13.0
## Migrating from ML-Agents Toolkit v0.12.0 to v0.13.0
* The low level Python API has changed. You can look at the document [Low Level Python API documentation](Python-API.md) for more information. This should only affect you if you're writing a custom trainer; if you use `mlagents-learn` for training, this should be a transparent change.
* `reset()` on the Low-Level Python API no longer takes a `train_mode` argument. To modify the performance/speed of the engine, you must use an `EngineConfigurationChannel`
* `reset()` on the Low-Level Python API no longer takes a `config` argument. `UnityEnvironment` no longer has a `reset_parameters` field. To modify float properties in the environment, you must use a `FloatPropertiesChannel`. For more information, refer to the [Low Level Python API documentation](Python-API.md)
* `CustomResetParameters` are now removed.
* The Academy no longer has a `Training Configuration` nor `Inference Configuration` field in the inspector. To modify the configuration from the Low-Level Python API, use an `EngineConfigurationChannel`.
To modify it during training, use the new command line arguments `--width`, `--height`, `--quality-level`, `--time-scale` and `--target-frame-rate` in `mlagents-learn`.
* The Academy no longer has a `Default Reset Parameters` field in the inspector. The Academy class no longer has a `ResetParameters`. To access shared float properties with Python, use the new `FloatProperties` field on the Academy.
* Offline Behavioral Cloning has been removed. To learn from demonstrations, use the GAIL and
Behavioral Cloning features with either PPO or SAC. See [Imitation Learning](Training-Imitation-Learning.md) for more information.
* `mlagents.envs` was renamed to `mlagents_envs`. The previous repo layout depended on [PEP420](https://www.python.org/dev/peps/pep-0420/), which caused problems with some of our tooling such as mypy and pylint.
* The official version of Unity ML-Agents supports is now 2018.4 LTS. If you run into issues, please consider deleting your library folder and reponening your projects. You will need to install the Barracuda package into your project in order to ML-Agents to compile correctly.
- The low level Python API has changed. You can look at the document
[Low Level Python API documentation](Python-API.md) for more information. This
should only affect you if you're writing a custom trainer; if you use
`mlagents-learn` for training, this should be a transparent change.
- `reset()` on the Low-Level Python API no longer takes a `train_mode`
argument. To modify the performance/speed of the engine, you must use an
`EngineConfigurationChannel`
- `reset()` on the Low-Level Python API no longer takes a `config` argument.
`UnityEnvironment` no longer has a `reset_parameters` field. To modify float
properties in the environment, you must use a `FloatPropertiesChannel`. For
more information, refer to the
[Low Level Python API documentation](Python-API.md)
- `CustomResetParameters` are now removed.
- The Academy no longer has a `Training Configuration` nor
`Inference Configuration` field in the inspector. To modify the configuration
from the Low-Level Python API, use an `EngineConfigurationChannel`. To modify
it during training, use the new command line arguments `--width`, `--height`,
`--quality-level`, `--time-scale` and `--target-frame-rate` in
`mlagents-learn`.
- The Academy no longer has a `Default Reset Parameters` field in the inspector.
The Academy class no longer has a `ResetParameters`. To access shared float
properties with Python, use the new `FloatProperties` field on the Academy.
- Offline Behavioral Cloning has been removed. To learn from demonstrations, use
the GAIL and Behavioral Cloning features with either PPO or SAC. See
[Imitation Learning](Training-Imitation-Learning.md) for more information.
- `mlagents.envs` was renamed to `mlagents_envs`. The previous repo layout
depended on [PEP420](https://www.python.org/dev/peps/pep-0420/), which caused
problems with some of our tooling such as mypy and pylint.
- The official version of Unity ML-Agents supports is now 2018.4 LTS. If you run
into issues, please consider deleting your library folder and reponening your
projects. You will need to install the Barracuda package into your project in
order to ML-Agents to compile correctly.
* If you had a custom `Training Configuration` in the Academy inspector, you will need to pass your custom configuration at every training run using the new command line arguments `--width`, `--height`, `--quality-level`, `--time-scale` and `--target-frame-rate`.
* If you were using `--slow` in `mlagents-learn`, you will need to pass your old `Inference Configuration` of the Academy inspector with the new command line arguments `--width`, `--height`, `--quality-level`, `--time-scale` and `--target-frame-rate` instead.
* Any imports from `mlagents.envs` should be replaced with `mlagents_envs`.
- If you had a custom `Training Configuration` in the Academy inspector, you
will need to pass your custom configuration at every training run using the
new command line arguments `--width`, `--height`, `--quality-level`,
`--time-scale` and `--target-frame-rate`.
- If you were using `--slow` in `mlagents-learn`, you will need to pass your old
`Inference Configuration` of the Academy inspector with the new command line
arguments `--width`, `--height`, `--quality-level`, `--time-scale` and
`--target-frame-rate` instead.
- Any imports from `mlagents.envs` should be replaced with `mlagents_envs`.
## Migrating from ML-Agents toolkit v0.11.0 to v0.12.0
## Migrating from ML-Agents Toolkit v0.11.0 to v0.12.0
* Text actions and observations, and custom action and observation protos have been removed.
* RayPerception3D and RayPerception2D are marked deprecated, and will be removed in a future release. They can be replaced by RayPerceptionSensorComponent3D and RayPerceptionSensorComponent2D.
* The `Use Heuristic` checkbox in Behavior Parameters has been replaced with a `Behavior Type` dropdown menu. This has the following options:
* `Default` corresponds to the previous unchecked behavior, meaning that Agents will train if they connect to a python trainer, otherwise they will perform inference.
* `Heuristic Only` means the Agent will always use the `Heuristic()` method. This corresponds to having "Use Heuristic" selected in 0.11.0.
* `Inference Only` means the Agent will always perform inference.
* Barracuda was upgraded to 0.3.2, and it is now installed via the Unity Package Manager.
- Text actions and observations, and custom action and observation protos have
been removed.
- RayPerception3D and RayPerception2D are marked deprecated, and will be removed
in a future release. They can be replaced by RayPerceptionSensorComponent3D
and RayPerceptionSensorComponent2D.
- The `Use Heuristic` checkbox in Behavior Parameters has been replaced with a
`Behavior Type` dropdown menu. This has the following options:
- `Default` corresponds to the previous unchecked behavior, meaning that
Agents will train if they connect to a python trainer, otherwise they will
perform inference.
- `Heuristic Only` means the Agent will always use the `Heuristic()` method.
This corresponds to having "Use Heuristic" selected in 0.11.0.
- `Inference Only` means the Agent will always perform inference.
- Barracuda was upgraded to 0.3.2, and it is now installed via the Unity Package
Manager.
* We [fixed a bug](https://github.com/Unity-Technologies/ml-agents/pull/2823) in `RayPerception3d.Perceive()` that was causing the `endOffset` to be used incorrectly. However this may produce different behavior from previous versions if you use a non-zero `startOffset`.
To reproduce the old behavior, you should increase the the value of `endOffset` by `startOffset`.
You can verify your raycasts are performing as expected in scene view using the debug rays.
* If you use RayPerception3D, replace it with RayPerceptionSensorComponent3D (and similarly for 2D). The settings, such as ray angles and detectable tags, are configured on the component now.
RayPerception3D would contribute `(# of rays) * (# of tags + 2)` to the State Size in Behavior Parameters, but this is no longer necessary, so you should reduce the State Size by this amount.
Making this change will require retraining your model, since the observations that RayPerceptionSensorComponent3D produces are different from the old behavior.
* If you see messages such as `The type or namespace 'Barracuda' could not be found` or `The type or namespace 'Google' could not be found`, you will need to [install the Barracuda preview package](Installation.md#package-installation).
## Migrating from ML-Agents toolkit v0.10 to v0.11.0
- We [fixed a bug](https://github.com/Unity-Technologies/ml-agents/pull/2823) in
`RayPerception3d.Perceive()` that was causing the `endOffset` to be used
incorrectly. However this may produce different behavior from previous
versions if you use a non-zero `startOffset`. To reproduce the old behavior,
you should increase the the value of `endOffset` by `startOffset`. You can
verify your raycasts are performing as expected in scene view using the debug
rays.
- If you use RayPerception3D, replace it with RayPerceptionSensorComponent3D
(and similarly for 2D). The settings, such as ray angles and detectable tags,
are configured on the component now. RayPerception3D would contribute
`(# of rays) * (# of tags + 2)` to the State Size in Behavior Parameters, but
this is no longer necessary, so you should reduce the State Size by this
amount. Making this change will require retraining your model, since the
observations that RayPerceptionSensorComponent3D produces are different from
the old behavior.
- If you see messages such as
`The type or namespace 'Barracuda' could not be found` or
`The type or namespace 'Google' could not be found`, you will need to
[install the Barracuda preview package](Installation.md#package-installation).
## Migrating from ML-Agents Toolkit v0.10 to v0.11.0
* The definition of the gRPC service has changed.
* The online BC training feature has been removed.
* The BroadcastHub has been deprecated. If there is a training Python process, all LearningBrains in the scene will automatically be trained. If there is no Python process, inference will be used.
* The Brain ScriptableObjects have been deprecated. The Brain Parameters are now on the Agent and are referred to as Behavior Parameters. Make sure the Behavior Parameters is attached to the Agent GameObject.
* To use a heuristic behavior, implement the `Heuristic()` method in the Agent class and check the `use heuristic` checkbox in the Behavior Parameters.
* Several changes were made to the setup for visual observations (i.e. using Cameras or RenderTextures):
* Camera resolutions are no longer stored in the Brain Parameters.
* AgentParameters no longer stores lists of Cameras and RenderTextures
* To add visual observations to an Agent, you must now attach a CameraSensorComponent or RenderTextureComponent to the agent. The corresponding Camera or RenderTexture can be added to these in the editor, and the resolution and color/grayscale is configured on the component itself.
- The definition of the gRPC service has changed.
- The online BC training feature has been removed.
- The BroadcastHub has been deprecated. If there is a training Python process,
all LearningBrains in the scene will automatically be trained. If there is no
Python process, inference will be used.
- The Brain ScriptableObjects have been deprecated. The Brain Parameters are now
on the Agent and are referred to as Behavior Parameters. Make sure the
Behavior Parameters is attached to the Agent GameObject.
- To use a heuristic behavior, implement the `Heuristic()` method in the Agent
class and check the `use heuristic` checkbox in the Behavior Parameters.
- Several changes were made to the setup for visual observations (i.e. using
Cameras or RenderTextures):
- Camera resolutions are no longer stored in the Brain Parameters.
- AgentParameters no longer stores lists of Cameras and RenderTextures
- To add visual observations to an Agent, you must now attach a
CameraSensorComponent or RenderTextureComponent to the agent. The
corresponding Camera or RenderTexture can be added to these in the editor,
and the resolution and color/grayscale is configured on the component
itself.
* In order to be able to train, make sure both your ML-Agents Python package and UnitySDK code come from the v0.11 release. Training will not work, for example, if you update the ML-Agents Python package, and only update the API Version in UnitySDK.
* If your Agents used visual observations, you must add a CameraSensorComponent corresponding to each old Camera in the Agent's camera list (and similarly for RenderTextures).
* Since Brain ScriptableObjects have been removed, you will need to delete all the Brain ScriptableObjects from your `Assets` folder. Then, add a `Behavior Parameters` component to each `Agent` GameObject.
You will then need to complete the fields on the new `Behavior Parameters` component with the BrainParameters of the old Brain.
## Migrating from ML-Agents toolkit v0.9 to v0.10
- In order to be able to train, make sure both your ML-Agents Python package and
UnitySDK code come from the v0.11 release. Training will not work, for
example, if you update the ML-Agents Python package, and only update the API
Version in UnitySDK.
- If your Agents used visual observations, you must add a CameraSensorComponent
corresponding to each old Camera in the Agent's camera list (and similarly for
RenderTextures).
- Since Brain ScriptableObjects have been removed, you will need to delete all
the Brain ScriptableObjects from your `Assets` folder. Then, add a
`Behavior Parameters` component to each `Agent` GameObject. You will then need
to complete the fields on the new `Behavior Parameters` component with the
BrainParameters of the old Brain.
## Migrating from ML-Agents Toolkit v0.9 to v0.10
* We have updated the C# code in our repository to be in line with Unity Coding Conventions. This has changed the name of some public facing classes and enums.
* The example environments have been updated. If you were using these environments to benchmark your training, please note that the resulting rewards may be slightly different in v0.10.
- We have updated the C# code in our repository to be in line with Unity Coding
Conventions. This has changed the name of some public facing classes and
enums.
- The example environments have been updated. If you were using these
environments to benchmark your training, please note that the resulting
rewards may be slightly different in v0.10.
* `UnitySDK/Assets/ML-Agents/Scripts/Communicator.cs` and its class `Communicator` have been renamed to `UnitySDK/Assets/ML-Agents/Scripts/ICommunicator.cs` and `ICommunicator` respectively.
* The `SpaceType` Enums `discrete`, and `continuous` have been renamed to `Discrete` and `Continuous`.
* We have removed the `Done` call as well as the capacity to set `Max Steps` on the Academy. Therefore an AcademyReset will never be triggered from C# (only from Python). If you want to reset the simulation after a
fixed number of steps, or when an event in the simulation occurs, we recommend looking at our multi-agent example environments (such as FoodCollector).
In our examples, groups of Agents can be reset through an "Area" that can reset groups of Agents.
* The import for `mlagents.envs.UnityEnvironment` was removed. If you are using the Python API, change `from mlagents_envs import UnityEnvironment` to `from mlagents_envs.environment import UnityEnvironment`.
- `UnitySDK/Assets/ML-Agents/Scripts/Communicator.cs` and its class
`Communicator` have been renamed to
`UnitySDK/Assets/ML-Agents/Scripts/ICommunicator.cs` and `ICommunicator`
respectively.
- The `SpaceType` Enums `discrete`, and `continuous` have been renamed to
`Discrete` and `Continuous`.
- We have removed the `Done` call as well as the capacity to set `Max Steps` on
the Academy. Therefore an AcademyReset will never be triggered from C# (only
from Python). If you want to reset the simulation after a fixed number of
steps, or when an event in the simulation occurs, we recommend looking at our
multi-agent example environments (such as FoodCollector). In our examples,
groups of Agents can be reset through an "Area" that can reset groups of
Agents.
- The import for `mlagents.envs.UnityEnvironment` was removed. If you are using
the Python API, change `from mlagents_envs import UnityEnvironment` to
`from mlagents_envs.environment import UnityEnvironment`.
## Migrating from ML-Agents toolkit v0.8 to v0.9
## Migrating from ML-Agents Toolkit v0.8 to v0.9
* We have changed the way reward signals (including Curiosity) are defined in the
`trainer_config.yaml`.
* When using multiple environments, every "step" is recorded in TensorBoard.
* The steps in the command line console corresponds to a single step of a single environment.
Previously, each step corresponded to one step for all environments (i.e., `num_envs` steps).
- We have changed the way reward signals (including Curiosity) are defined in
the `trainer_config.yaml`.
- When using multiple environments, every "step" is recorded in TensorBoard.
- The steps in the command line console corresponds to a single step of a single
environment. Previously, each step corresponded to one step for all
environments (i.e., `num_envs` steps).
* If you were overriding any of these following parameters in your config file, remove them
from the top-level config and follow the steps below:
* `gamma`: Define a new `extrinsic` reward signal and set it's `gamma` to your new gamma.
* `use_curiosity`, `curiosity_strength`, `curiosity_enc_size`: Define a `curiosity` reward signal
and set its `strength` to `curiosity_strength`, and `encoding_size` to `curiosity_enc_size`. Give it
the same `gamma` as your `extrinsic` signal to mimic previous behavior.
See [Reward Signals](Reward-Signals.md) for more information on defining reward signals.
* TensorBoards generated when running multiple environments in v0.8 are not comparable to those generated in
v0.9 in terms of step count. Multiply your v0.8 step count by `num_envs` for an approximate comparison.
You may need to change `max_steps` in your config as appropriate as well.
- If you were overriding any of these following parameters in your config file,
remove them from the top-level config and follow the steps below:
- `gamma`: Define a new `extrinsic` reward signal and set it's `gamma` to your
new gamma.
- `use_curiosity`, `curiosity_strength`, `curiosity_enc_size`: Define a
`curiosity` reward signal and set its `strength` to `curiosity_strength`,
and `encoding_size` to `curiosity_enc_size`. Give it the same `gamma` as
your `extrinsic` signal to mimic previous behavior. See
[Reward Signals](Reward-Signals.md) for more information on defining reward
signals.
- TensorBoards generated when running multiple environments in v0.8 are not
comparable to those generated in v0.9 in terms of step count. Multiply your
v0.8 step count by `num_envs` for an approximate comparison. You may need to
change `max_steps` in your config as appropriate as well.
## Migrating from ML-Agents toolkit v0.7 to v0.8
## Migrating from ML-Agents Toolkit v0.7 to v0.8
* We have split the Python packages into two separate packages `ml-agents` and `ml-agents-envs`.
* `--worker-id` option of `learn.py` has been removed, use `--base-port` instead if you'd like to run multiple instances of `learn.py`.
- We have split the Python packages into two separate packages `ml-agents` and
`ml-agents-envs`.
- `--worker-id` option of `learn.py` has been removed, use `--base-port` instead
if you'd like to run multiple instances of `learn.py`.
* If you are installing via PyPI, there is no change.
* If you intend to make modifications to `ml-agents` or `ml-agents-envs` please check the Installing for Development in the [Installation documentation](Installation.md).
- If you are installing via PyPI, there is no change.
- If you intend to make modifications to `ml-agents` or `ml-agents-envs` please
check the Installing for Development in the
[Installation documentation](Installation.md).
## Migrating from ML-Agents toolkit v0.6 to v0.7
## Migrating from ML-Agents Toolkit v0.6 to v0.7
* We no longer support TFS and are now using the [Unity Inference Engine](Unity-Inference-Engine.md)
- We no longer support TFS and are now using the
[Unity Inference Engine](Unity-Inference-Engine.md)
* Make sure to remove the `ENABLE_TENSORFLOW` flag in your Unity Project settings
- Make sure to remove the `ENABLE_TENSORFLOW` flag in your Unity Project
settings
## Migrating from ML-Agents toolkit v0.5 to v0.6
## Migrating from ML-Agents Toolkit v0.5 to v0.6
* Brains are now Scriptable Objects instead of MonoBehaviors.
* You can no longer modify the type of a Brain. If you want to switch
between `PlayerBrain` and `LearningBrain` for multiple agents,
you will need to assign a new Brain to each agent separately.
__Note:__ You can pass the same Brain to multiple agents in a scene by
leveraging Unity's prefab system or look for all the agents in a scene
using the search bar of the `Hierarchy` window with the word `Agent`.
- Brains are now Scriptable Objects instead of MonoBehaviors.
- You can no longer modify the type of a Brain. If you want to switch between
`PlayerBrain` and `LearningBrain` for multiple agents, you will need to assign
a new Brain to each agent separately. **Note:** You can pass the same Brain to
multiple agents in a scene by leveraging Unity's prefab system or look for all
the agents in a scene using the search bar of the `Hierarchy` window with the
word `Agent`.
* We replaced the **Internal** and **External** Brain with **Learning Brain**.
- We replaced the **Internal** and **External** Brain with **Learning Brain**.
* We removed the `Broadcast` checkbox of the Brain, to use the broadcast
- We removed the `Broadcast` checkbox of the Brain, to use the broadcast
* When training multiple Brains at the same time, each model is now stored
into a separate model file rather than in the same file under different
graph scopes.
* The **Learning Brain** graph scope, placeholder names, output names and custom
- When training multiple Brains at the same time, each model is now stored into
a separate model file rather than in the same file under different graph
scopes.
- The **Learning Brain** graph scope, placeholder names, output names and custom
* To update a scene from v0.5 to v0.6, you must:
* Remove the `Brain` GameObjects in the scene. (Delete all of the
Brain GameObjects under Academy in the scene.)
* Create new `Brain` Scriptable Objects using `Assets -> Create ->
ML-Agents` for each type of the Brain you plan to use, and put
the created files under a folder called Brains within your project.
* Edit their `Brain Parameters` to be the same as the parameters used
in the `Brain` GameObjects.
* Agents have a `Brain` field in the Inspector, you need to drag the
- To update a scene from v0.5 to v0.6, you must:
- Remove the `Brain` GameObjects in the scene. (Delete all of the Brain
GameObjects under Academy in the scene.)
- Create new `Brain` Scriptable Objects using `Assets -> Create -> ML-Agents`
for each type of the Brain you plan to use, and put the created files under
a folder called Brains within your project.
- Edit their `Brain Parameters` to be the same as the parameters used in the
`Brain` GameObjects.
- Agents have a `Brain` field in the Inspector, you need to drag the
* The Academy has a `Broadcast Hub` field in the inspector, which is
list of brains used in the scene. To train or control your Brain
from the `mlagents-learn` Python script, you need to drag the relevant
`LearningBrain` ScriptableObjects used in your scene into entries
into this list.
- The Academy has a `Broadcast Hub` field in the inspector, which is list of
brains used in the scene. To train or control your Brain from the
`mlagents-learn` Python script, you need to drag the relevant
`LearningBrain` ScriptableObjects used in your scene into entries into this
list.
## Migrating from ML-Agents toolkit v0.4 to v0.5
## Migrating from ML-Agents Toolkit v0.4 to v0.5
* The Unity project `unity-environment` has been renamed `UnitySDK`.
* The `python` folder has been renamed to `ml-agents`. It now contains two
- The Unity project `unity-environment` has been renamed `UnitySDK`.
- The `python` folder has been renamed to `ml-agents`. It now contains two
* The supported Unity version has changed from `2017.1 or later` to `2017.4
or later`. 2017.4 is an LTS (Long Term Support) version that helps us
- The supported Unity version has changed from `2017.1 or later` to
`2017.4 or later`. 2017.4 is an LTS (Long Term Support) version that helps us
maintain good quality and support. Earlier versions of Unity might still work,
but you may encounter an
[error](FAQ.md#instance-of-corebraininternal-couldnt-be-created) listed here.

* Discrete Actions now use [branches](https://arxiv.org/abs/1711.08946). You can
- Discrete Actions now use [branches](https://arxiv.org/abs/1711.08946). You can
now specify concurrent discrete actions. You will need to update the Brain
Parameters in the Brain Inspector in all your environments that use discrete
actions. Refer to the

### Python API
* In order to run a training session, you can now use the command
- In order to run a training session, you can now use the command
[here](Training-ML-Agents.md#training-with-mlagents-learn). For example,
if we previously ran
[here](Training-ML-Agents.md#training-with-mlagents-learn). For example, if we
previously ran
```sh
python3 learn.py 3DBall --train

from the root directory where we installed the ML-Agents Toolkit.
* It is now required to specify the path to the yaml trainer configuration file
- It is now required to specify the path to the yaml trainer configuration file
[trainer_config.yaml](../config/trainer_config.yaml). An example of passing
a trainer configuration to `mlagents-learn` is shown above.
* The environment name is now passed through the `--env` option.
* Curriculum learning has been changed. Refer to the
[curriculum learning documentation](Training-Curriculum-Learning.md)
for detailed information. In summary:
* Curriculum files for the same environment must now be placed into a folder.
[trainer_config.yaml](../config/trainer_config.yaml). An example of passing a
trainer configuration to `mlagents-learn` is shown above.
- The environment name is now passed through the `--env` option.
- Curriculum learning has been changed. Refer to the
[curriculum learning documentation](Training-Curriculum-Learning.md) for
detailed information. In summary:
- Curriculum files for the same environment must now be placed into a folder.
* `min_lesson_length` now specifies the minimum number of episodes in a lesson
- `min_lesson_length` now specifies the minimum number of episodes in a lesson
* It is no longer necessary to specify the `Max Steps` of the Academy to use
- It is no longer necessary to specify the `Max Steps` of the Academy to use
## Migrating from ML-Agents toolkit v0.3 to v0.4
## Migrating from ML-Agents Toolkit v0.3 to v0.4
* `using MLAgents;` needs to be added in all of the C# scripts that use
- `using MLAgents;` needs to be added in all of the C# scripts that use
* We've changed some of the Python packages dependencies in requirement.txt
- We've changed some of the Python packages dependencies in requirement.txt
folder
to update your Python packages.
folder to update your Python packages.
## Migrating from ML-Agents toolkit v0.2 to v0.3
## Migrating from ML-Agents Toolkit v0.2 to v0.3
There are a large number of new features and improvements in the ML-Agents
toolkit v0.3 which change both the training process and Unity API in ways which

### Important
* The ML-Agents toolkit is no longer compatible with Python 2.
- The ML-Agents Toolkit is no longer compatible with Python 2.
* The training script `ppo.py` and `PPO.ipynb` Python notebook have been
- The training script `ppo.py` and `PPO.ipynb` Python notebook have been
* Hyperparameters for training Brains are now stored in the
- Hyperparameters for training Brains are now stored in the
* Modifications to an Agent's rewards must now be done using either
- Modifications to an Agent's rewards must now be done using either
* Setting an Agent to done now requires the use of the `Done()` method.
* `CollectStates()` has been replaced by `CollectObservations()`, which now no
- Setting an Agent to done now requires the use of the `Done()` method.
- `CollectStates()` has been replaced by `CollectObservations()`, which now no
* To collect observations, call `AddVectorObs()` within `CollectObservations()`.
- To collect observations, call `AddVectorObs()` within `CollectObservations()`.
* `AgentStep()` has been replaced by `AgentAction()`.
* `WaitTime()` has been removed.
* The `Frame Skip` field of the Academy is replaced by the Agent's `Decision
Frequency` field, enabling the Agent to make decisions at different frequencies.
* The names of the inputs in the Internal Brain have been changed. You must
- `AgentStep()` has been replaced by `AgentAction()`.
- `WaitTime()` has been removed.
- The `Frame Skip` field of the Academy is replaced by the Agent's
`Decision Frequency` field, enabling the Agent to make decisions at different
frequencies.
- The names of the inputs in the Internal Brain have been changed. You must
replace `state` with `vector_observation` and `observation` with
`visual_observation`. In addition, you must remove the `epsilon` placeholder.

the concepts used in ML-Agents. The changes are highlighted in the table below.
| Old - v0.2 and earlier | New - v0.3 and later |
| --- | --- |
| State | Vector Observation |
| Observation | Visual Observation |
| Action | Vector Action |
| N/A | Text Observation |
| N/A | Text Action |
| ---------------------- | -------------------- |
| State | Vector Observation |
| Observation | Visual Observation |
| Action | Vector Action |
| N/A | Text Observation |
| N/A | Text Action |

2
docs/Training-on-Amazon-Web-Service.md
查看文件


[Deep Learning AMI (Ubuntu)](https://aws.amazon.com/marketplace/pp/B077GCH38C)
listed under AWS Marketplace with a p2.xlarge instance.
### Installing the ML-Agents toolkit on the instance
### Installing the ML-Agents Toolkit on the instance
After launching your EC2 instance using the ami and ssh into it:

2
docs/Training-on-Microsoft-Azure.md
查看文件


# Training on Microsoft Azure (works with ML-Agents toolkit v0.3)
# Training on Microsoft Azure (works with ML-Agents Toolkit v0.3)
:warning: **Note:** We no longer use this guide ourselves and so it may not work
correctly. We've decided to keep it up just in case it is helpful to you.

4
ml-agents/mlagents/trainers/ghost/trainer.py
查看文件


"""
policy = self.trainer.create_policy(parsed_behavior_id, brain_parameters)
policy.create_tf_graph()
policy.initialize_or_load()
policy.init_load_weights()
team_id = parsed_behavior_id.team_id
self.controller.subscribe_team_id(team_id, self)

internal_trainer_policy = self.trainer.create_policy(
parsed_behavior_id, brain_parameters
)
internal_trainer_policy.create_tf_graph()
self.trainer.add_policy(parsed_behavior_id, internal_trainer_policy)
internal_trainer_policy.init_load_weights()
self.current_policy_snapshot[
parsed_behavior_id.brain_name

self._save_snapshot() # Need to save after trainer initializes policy
self.trainer.add_policy(parsed_behavior_id, internal_trainer_policy)
self._learning_team = self.controller.get_learning_team
self.wrapped_trainer_team = team_id
return policy

8
ml-agents/mlagents/trainers/tests/test_simple_rl.py
查看文件


[BRAIN_NAME + "?team=0", brain_name_opp + "?team=1"], use_discrete=use_discrete
)
override_vals = {
"max_steps": 2000,
"max_steps": 4000,
"save_steps": 5000,
"swap_steps": 5000,
"team_change": 2000,
"save_steps": 10000,
"swap_steps": 10000,
"team_change": 4000,
},
}
config = generate_config(PPO_CONFIG, override_vals)

撰写 预览
正在加载...
取消
保存