浏览代码
Merge pull request #1083 from Unity-Technologies/develop-flat-code-restructure
Merge pull request #1083 from Unity-Technologies/develop-flat-code-restructure
ML-Agents Code Restructure/develop-generalizationTraining-TrainerController
GitHub
6 年前
当前提交
3900ed66
共有 170 个文件被更改,包括 1720 次插入 和 1506 次删除
-
49.gitignore
-
10Dockerfile
-
25docs/API-Reference.md
-
15docs/Background-Jupyter.md
-
146docs/Basic-Guide.md
-
113docs/FAQ.md
-
2docs/Feature-Memory.md
-
380docs/Getting-Started-with-Balance-Ball.md
-
68docs/Glossary.md
-
2docs/Installation-Windows.md
-
57docs/Installation.md
-
12docs/Learning-Environment-Create-New.md
-
2docs/Learning-Environment-Design-External-Internal-Brains.md
-
2docs/Learning-Environment-Examples.md
-
28docs/Learning-Environment-Executable.md
-
685docs/ML-Agents-Overview.md
-
138docs/Python-API.md
-
81docs/Readme.md
-
23docs/Training-Curriculum-Learning.md
-
4docs/Training-Imitation-Learning.md
-
34docs/Training-ML-Agents.md
-
2docs/Training-on-Amazon-Web-Service.md
-
6docs/Training-on-Microsoft-Azure.md
-
115docs/Using-Docker.md
-
6docs/Using-Tensorboard.md
-
8docs/dox-ml-agents.conf
-
16gym-unity/Readme.md
-
2gym-unity/gym_unity/envs/unity_env.py
-
2gym-unity/setup.py
-
2MLAgentsSDK/Assets/ML-Agents/Scripts/CommunicatorObjects/UnityToExternalGrpc.cs.meta
-
10MLAgentsSDK/Assets/ML-Agents/Scripts/CommunicatorObjects/UnityToExternalGrpc.cs
-
2MLAgentsSDK/Assets/ML-Agents/Scripts/CommunicatorObjects/UnityToExternal.cs.meta
-
19MLAgentsSDK/Assets/ML-Agents/Scripts/CommunicatorObjects/UnityToExternal.cs
-
2MLAgentsSDK/Assets/ML-Agents/Scripts/CommunicatorObjects/UnityRlOutput.cs.meta
-
27MLAgentsSDK/Assets/ML-Agents/Scripts/CommunicatorObjects/UnityRlOutput.cs
-
2MLAgentsSDK/Assets/ML-Agents/Scripts/CommunicatorObjects/UnityRlInput.cs.meta
-
41MLAgentsSDK/Assets/ML-Agents/Scripts/CommunicatorObjects/UnityRlInput.cs
-
2MLAgentsSDK/Assets/ML-Agents/Scripts/CommunicatorObjects/UnityRlInitializationOutput.cs.meta
-
31MLAgentsSDK/Assets/ML-Agents/Scripts/CommunicatorObjects/UnityRlInitializationOutput.cs
-
2MLAgentsSDK/Assets/ML-Agents/Scripts/CommunicatorObjects/UnityRlInitializationInput.cs.meta
-
14MLAgentsSDK/Assets/ML-Agents/Scripts/CommunicatorObjects/UnityRlInitializationInput.cs
-
2MLAgentsSDK/Assets/ML-Agents/Scripts/CommunicatorObjects/UnityOutput.cs.meta
-
29MLAgentsSDK/Assets/ML-Agents/Scripts/CommunicatorObjects/UnityOutput.cs
-
2MLAgentsSDK/Assets/ML-Agents/Scripts/CommunicatorObjects/UnityMessage.cs.meta
-
32MLAgentsSDK/Assets/ML-Agents/Scripts/CommunicatorObjects/UnityMessage.cs
-
2MLAgentsSDK/Assets/ML-Agents/Scripts/CommunicatorObjects/UnityInput.cs.meta
-
27MLAgentsSDK/Assets/ML-Agents/Scripts/CommunicatorObjects/UnityInput.cs
-
2MLAgentsSDK/Assets/ML-Agents/Scripts/CommunicatorObjects/SpaceTypeProto.cs.meta
-
17MLAgentsSDK/Assets/ML-Agents/Scripts/CommunicatorObjects/SpaceTypeProto.cs
-
2MLAgentsSDK/Assets/ML-Agents/Scripts/CommunicatorObjects/ResolutionProto.cs.meta
-
15MLAgentsSDK/Assets/ML-Agents/Scripts/CommunicatorObjects/ResolutionProto.cs
-
2MLAgentsSDK/Assets/ML-Agents/Scripts/CommunicatorObjects/Header.cs.meta
-
14MLAgentsSDK/Assets/ML-Agents/Scripts/CommunicatorObjects/Header.cs
-
2MLAgentsSDK/Assets/ML-Agents/Scripts/CommunicatorObjects/EnvironmentParametersProto.cs.meta
-
21MLAgentsSDK/Assets/ML-Agents/Scripts/CommunicatorObjects/EnvironmentParametersProto.cs
-
2MLAgentsSDK/Assets/ML-Agents/Scripts/CommunicatorObjects/EngineConfigurationProto.cs.meta
-
19MLAgentsSDK/Assets/ML-Agents/Scripts/CommunicatorObjects/EngineConfigurationProto.cs
-
2MLAgentsSDK/Assets/ML-Agents/Scripts/CommunicatorObjects/CommandProto.cs.meta
-
14MLAgentsSDK/Assets/ML-Agents/Scripts/CommunicatorObjects/CommandProto.cs
-
2MLAgentsSDK/Assets/ML-Agents/Scripts/CommunicatorObjects/BrainTypeProto.cs.meta
-
18MLAgentsSDK/Assets/ML-Agents/Scripts/CommunicatorObjects/BrainTypeProto.cs
-
2MLAgentsSDK/Assets/ML-Agents/Scripts/CommunicatorObjects/BrainParametersProto.cs.meta
-
36MLAgentsSDK/Assets/ML-Agents/Scripts/CommunicatorObjects/BrainParametersProto.cs
-
2MLAgentsSDK/Assets/ML-Agents/Scripts/CommunicatorObjects/AgentInfoProto.cs.meta
-
24MLAgentsSDK/Assets/ML-Agents/Scripts/CommunicatorObjects/AgentInfoProto.cs
-
2MLAgentsSDK/Assets/ML-Agents/Scripts/CommunicatorObjects/AgentActionProto.cs.meta
-
16MLAgentsSDK/Assets/ML-Agents/Scripts/CommunicatorObjects/AgentActionProto.cs
-
0MLAgentsSDK/Assets/ML-Agents/Scripts/ActionMasker.cs
-
2MLAgentsSDK/Assets/ML-Agents/Scripts/Academy.cs
-
4ml-agents/tests/trainers/test_curriculum.py
-
36ml-agents/tests/trainers/test_trainer_controller.py
-
5ml-agents/tests/mock_communicator.py
-
10ml-agents/mlagents/envs/communicator_objects/unity_to_external_pb2_grpc.py
-
18ml-agents/mlagents/envs/communicator_objects/unity_to_external_pb2.py
-
54ml-agents/mlagents/envs/communicator_objects/unity_rl_output_pb2.py
-
66ml-agents/mlagents/envs/communicator_objects/unity_rl_input_pb2.py
-
39ml-agents/mlagents/envs/communicator_objects/unity_rl_initialization_output_pb2.py
-
21ml-agents/mlagents/envs/communicator_objects/unity_rl_initialization_input_pb2.py
-
33ml-agents/mlagents/envs/communicator_objects/unity_output_pb2.py
-
39ml-agents/mlagents/envs/communicator_objects/unity_message_pb2.py
-
33ml-agents/mlagents/envs/communicator_objects/unity_input_pb2.py
-
25ml-agents/mlagents/envs/communicator_objects/space_type_proto_pb2.py
-
25ml-agents/mlagents/envs/communicator_objects/resolution_proto_pb2.py
-
23ml-agents/mlagents/envs/communicator_objects/header_pb2.py
-
36ml-agents/mlagents/envs/communicator_objects/environment_parameters_proto_pb2.py
-
31ml-agents/mlagents/envs/communicator_objects/engine_configuration_proto_pb2.py
-
23ml-agents/mlagents/envs/communicator_objects/command_proto_pb2.py
-
29ml-agents/mlagents/envs/communicator_objects/brain_type_proto_pb2.py
-
49ml-agents/mlagents/envs/communicator_objects/brain_parameters_proto_pb2.py
-
41ml-agents/mlagents/envs/communicator_objects/agent_info_proto_pb2.py
-
27ml-agents/mlagents/envs/communicator_objects/agent_action_proto_pb2.py
-
4ml-agents/mlagents/envs/socket_communicator.py
-
6ml-agents/mlagents/envs/rpc_communicator.py
-
4ml-agents/mlagents/envs/exception.py
-
4ml-agents/mlagents/envs/environment.py
-
4ml-agents/mlagents/envs/communicator.py
-
0ml-agents/mlagents/envs/brain.py
-
3ml-agents/mlagents/envs/__init__.py
-
21notebooks/getting-started.ipynb
-
16ml-agents/mlagents/trainers/trainer_controller.py
|
|||
# API Reference |
|||
|
|||
Our developer-facing C# classes (Academy, Agent, Decision and |
|||
Monitor) have been documented to be compatabile with |
|||
[Doxygen](http://www.stack.nl/~dimitri/doxygen/) for auto-generating HTML |
|||
Our developer-facing C# classes (Academy, Agent, Decision and Monitor) have been |
|||
documented to be compatabile with |
|||
[Doxygen](http://www.stack.nl/~dimitri/doxygen/) for auto-generating HTML |
|||
To generate the API reference, |
|||
[download Doxygen](http://www.stack.nl/~dimitri/doxygen/download.html) and run |
|||
the following command within the `docs/` directory: |
|||
To generate the API reference, [download |
|||
Doxygen](http://www.stack.nl/~dimitri/doxygen/download.html) and run the |
|||
following command within the `docs/` directory: |
|||
that includes the classes that have been properly formatted. |
|||
The generated HTML files will be placed |
|||
in the `html/` subdirectory. Open `index.html` within that subdirectory to |
|||
navigate to the API reference home. Note that `html/` is already included in |
|||
the repository's `.gitignore` file. |
|||
that includes the classes that have been properly formatted. The generated HTML |
|||
files will be placed in the `html/` subdirectory. Open `index.html` within that |
|||
subdirectory to navigate to the API reference home. Note that `html/` is already |
|||
included in the repository's `.gitignore` file. |
|||
In the near future, we aim to expand our documentation |
|||
to include all the Unity C# classes and Python API. |
|||
In the near future, we aim to expand our documentation to include all the Unity |
|||
C# classes and Python API. |
|
|||
# Background: Jupyter |
|||
|
|||
[Jupyter](https://jupyter.org) is a fantastic tool for writing code with |
|||
embedded visualizations. We provide one such notebook, `python/notebooks/getting-started.ipynb`, |
|||
for testing the Python control interface to a Unity build. This notebook is |
|||
introduced in the |
|||
[Getting Started with the 3D Balance Ball Environment](Getting-Started-with-Balance-Ball.md) |
|||
[Jupyter](https://jupyter.org) is a fantastic tool for writing code with |
|||
embedded visualizations. We provide one such notebook, |
|||
`notebooks/getting-started.ipynb`, for testing the Python control |
|||
interface to a Unity build. This notebook is introduced in the [Getting Started |
|||
with the 3D Balance Ball Environment](Getting-Started-with-Balance-Ball.md) |
|||
in the _Jupyter/IPython Quick Start Guide_. To launch Jupyter, run in the command line: |
|||
in the _Jupyter/IPython Quick Start Guide_. To launch Jupyter, run in the |
|||
command line: |
|||
`jupyter notebook` |
|||
jupyter notebook |
|||
|
|||
Then navigate to `localhost:8888` to access your notebooks. |
|
|||
# Frequently Asked Questions |
|||
|
|||
|
|||
### Scripting Runtime Environment not setup correctly |
|||
## Scripting Runtime Environment not setup correctly |
|||
If you haven't switched your scripting runtime version from .NET 3.5 to .NET 4.6 or .NET 4.x, you will see such error message: |
|||
If you haven't switched your scripting runtime version from .NET 3.5 to .NET 4.6 |
|||
or .NET 4.x, you will see such error message: |
|||
This is because .NET 3.5 doesn't support method Clear() for StringBuilder, refer to [Setting Up The ML-Agents Toolkit Within Unity](Installation.md#setting-up-ml-agent-within-unity) for solution. |
|||
This is because .NET 3.5 doesn't support method Clear() for StringBuilder, refer |
|||
to [Setting Up The ML-Agents Toolkit Within |
|||
Unity](Installation.md#setting-up-ml-agent-within-unity) for solution. |
|||
### TensorFlowSharp flag not turned on. |
|||
## TensorFlowSharp flag not turned on |
|||
If you have already imported the TensorFlowSharp plugin, but havn't set ENABLE_TENSORFLOW flag for your scripting define symbols, you will see the following error message: |
|||
If you have already imported the TensorFlowSharp plugin, but havn't set |
|||
ENABLE_TENSORFLOW flag for your scripting define symbols, you will see the |
|||
following error message: |
|||
You need to install and enable the TensorFlowSharp plugin in order to use the internal brain. |
|||
You need to install and enable the TensorFlowSharp plugin in order to use the internal brain. |
|||
This error message occurs because the TensorFlowSharp plugin won't be usage without the ENABLE_TENSORFLOW flag, refer to [Setting Up The ML-Agents Toolkit Within Unity](Installation.md#setting-up-ml-agent-within-unity) for solution. |
|||
This error message occurs because the TensorFlowSharp plugin won't be usage |
|||
without the ENABLE_TENSORFLOW flag, refer to [Setting Up The ML-Agents Toolkit |
|||
Within Unity](Installation.md#setting-up-ml-agent-within-unity) for solution. |
|||
### Tensorflow epsilon placeholder error |
|||
## Tensorflow epsilon placeholder error |
|||
If you have a graph placeholder set in the internal Brain inspector that is not present in the TensorFlow graph, you will see some error like this: |
|||
If you have a graph placeholder set in the internal Brain inspector that is not |
|||
present in the TensorFlow graph, you will see some error like this: |
|||
UnityAgentsException: One of the Tensorflow placeholder could not be found. In brain <some_brain_name>, there are no FloatingPoint placeholder named <some_placeholder_name>. |
|||
UnityAgentsException: One of the Tensorflow placeholder could not be found. In brain <some_brain_name>, there are no FloatingPoint placeholder named <some_placeholder_name>. |
|||
Solution: Go to all of your Brain object, find `Graph placeholders` and change its `size` to 0 to remove the `epsilon` placeholder. |
|||
Solution: Go to all of your Brain object, find `Graph placeholders` and change |
|||
its `size` to 0 to remove the `epsilon` placeholder. |
|||
Similarly, if you have a graph scope set in the internal Brain inspector that is not correctly set, you will see some error like this: |
|||
Similarly, if you have a graph scope set in the internal Brain inspector that is |
|||
not correctly set, you will see some error like this: |
|||
Solution: Make sure your Graph Scope field matches the corresponding brain object name in your Hierachy Inspector when there is multiple brain. |
|||
Solution: Make sure your Graph Scope field matches the corresponding brain |
|||
object name in your Hierachy Inspector when there is multiple brain. |
|||
### Environment Permission Error |
|||
## Environment Permission Error |
|||
If you directly import your Unity environment without building it in the |
|||
editor, you might need to give it additional permissions to execute it. |
|||
If you directly import your Unity environment without building it in the |
|||
editor, you might need to give it additional permissions to execute it. |
|||
`chmod -R 755 *.app` |
|||
```shell |
|||
chmod -R 755 *.app |
|||
``` |
|||
`chmod -R 755 *.x86_64` |
|||
```shell |
|||
chmod -R 755 *.x86_64 |
|||
``` |
|||
On Windows, you can find |
|||
On Windows, you can find |
|||
### Environment Connection Timeout |
|||
## Environment Connection Timeout |
|||
|
|||
If you are able to launch the environment from `UnityEnvironment` but then |
|||
receive a timeout error, there may be a number of possible causes. |
|||
If you are able to launch the environment from `UnityEnvironment` but |
|||
then receive a timeout error, there may be a number of possible causes. |
|||
* _Cause_: There may be no Brains in your environment which are set |
|||
to `External`. In this case, the environment will not attempt to |
|||
communicate with python. _Solution_: Set the Brains(s) you wish to |
|||
externally control through the Python API to `External` from the |
|||
Unity Editor, and rebuild the environment. |
|||
* _Cause_: On OSX, the firewall may be preventing communication with |
|||
the environment. _Solution_: Add the built environment binary to the |
|||
list of exceptions on the firewall by following |
|||
[instructions](https://support.apple.com/en-us/HT201642). |
|||
* _Cause_: An error happened in the Unity Environment preventing |
|||
communication. _Solution_: Look into the |
|||
[log files](https://docs.unity3d.com/Manual/LogFiles.html) |
|||
generated by the Unity Environment to figure what error happened. |
|||
* _Cause_: There may be no Brains in your environment which are set to |
|||
`External`. In this case, the environment will not attempt to communicate |
|||
with python. _Solution_: Set the Brains(s) you wish to externally control |
|||
through the Python API to `External` from the Unity Editor, and rebuild the |
|||
environment. |
|||
* _Cause_: On OSX, the firewall may be preventing communication with the |
|||
environment. _Solution_: Add the built environment binary to the list of |
|||
exceptions on the firewall by following |
|||
[instructions](https://support.apple.com/en-us/HT201642). |
|||
* _Cause_: An error happened in the Unity Environment preventing communication. |
|||
_Solution_: Look into the [log |
|||
files](https://docs.unity3d.com/Manual/LogFiles.html) generated by the Unity |
|||
Environment to figure what error happened. |
|||
### Communication port {} still in use |
|||
## Communication port {} still in use |
|||
If you receive an exception `"Couldn't launch new environment because |
|||
communication port {} is still in use. "`, you can change the worker |
|||
number in the Python script when calling |
|||
If you receive an exception `"Couldn't launch new environment because |
|||
communication port {} is still in use. "`, you can change the worker number in |
|||
the Python script when calling |
|||
`UnityEnvironment(file_name=filename, worker_id=X)` |
|||
```python |
|||
UnityEnvironment(file_name=filename, worker_id=X) |
|||
``` |
|||
### Mean reward : nan |
|||
## Mean reward : nan |
|||
If you receive a message `Mean reward : nan` when attempting to train a |
|||
model using PPO, this is due to the episodes of the learning environment |
|||
not terminating. In order to address this, set `Max Steps` for either |
|||
the Academy or Agents within the Scene Inspector to a value greater |
|||
than 0. Alternatively, it is possible to manually set `done` conditions |
|||
for episodes from within scripts for custom episode-terminating events. |
|||
If you receive a message `Mean reward : nan` when attempting to train a model |
|||
using PPO, this is due to the episodes of the learning environment not |
|||
terminating. In order to address this, set `Max Steps` for either the Academy or |
|||
Agents within the Scene Inspector to a value greater than 0. Alternatively, it |
|||
is possible to manually set `done` conditions for episodes from within scripts |
|||
for custom episode-terminating events. |
|
|||
# ML-Agents Toolkit Glossary |
|||
|
|||
* **Academy** - Unity Component which controls timing, reset, and |
|||
training/inference settings of the environment. |
|||
* **Action** - The carrying-out of a decision on the part of an |
|||
agent within the environment. |
|||
* **Agent** - Unity Component which produces observations and |
|||
takes actions in the environment. Agents actions are determined |
|||
by decisions produced by a linked Brain. |
|||
* **Brain** - Unity Component which makes decisions for the agents |
|||
linked to it. |
|||
* **Decision** - The specification produced by a Brain for an action |
|||
to be carried out given an observation. |
|||
* **Editor** - The Unity Editor, which may include any pane |
|||
(e.g. Hierarchy, Scene, Inspector). |
|||
* **Environment** - The Unity scene which contains Agents, Academy, |
|||
and Brains. |
|||
* **FixedUpdate** - Unity method called each time the the game engine |
|||
is stepped. ML-Agents logic should be placed here. |
|||
* **Frame** - An instance of rendering the main camera for the |
|||
display. Corresponds to each `Update` call of the game engine. |
|||
* **Observation** - Partial information describing the state of the |
|||
environment available to a given agent. (e.g. Vector, Visual, Text) |
|||
* **Policy** - Function for producing decisions from observations. |
|||
* **Reward** - Signal provided at every step used to indicate |
|||
desirability of an agent’s action within the current state |
|||
of the environment. |
|||
* **State** - The underlying properties of the environment |
|||
(including all agents within it) at a given time. |
|||
* **Step** - Corresponds to each `FixedUpdate` call of the game engine. |
|||
Is the smallest atomic change to the state possible. |
|||
* **Update** - Unity function called each time a frame is rendered. |
|||
ML-Agents logic should not be placed here. |
|||
* **External Coordinator** - ML-Agents class responsible for |
|||
communication with outside processes (in this case, the Python API). |
|||
* **Trainer** - Python class which is responsible for training a given |
|||
external brain. Contains TensorFlow graph which makes decisions |
|||
for external brain. |
|||
* **Academy** - Unity Component which controls timing, reset, and |
|||
training/inference settings of the environment. |
|||
* **Action** - The carrying-out of a decision on the part of an agent within the |
|||
environment. |
|||
* **Agent** - Unity Component which produces observations and takes actions in |
|||
the environment. Agents actions are determined by decisions produced by a |
|||
linked Brain. |
|||
* **Brain** - Unity Component which makes decisions for the agents linked to it. |
|||
* **Decision** - The specification produced by a Brain for an action to be |
|||
carried out given an observation. |
|||
* **Editor** - The Unity Editor, which may include any pane (e.g. Hierarchy, |
|||
Scene, Inspector). |
|||
* **Environment** - The Unity scene which contains Agents, Academy, and Brains. |
|||
* **FixedUpdate** - Unity method called each time the the game engine is |
|||
stepped. ML-Agents logic should be placed here. |
|||
* **Frame** - An instance of rendering the main camera for the display. |
|||
Corresponds to each `Update` call of the game engine. |
|||
* **Observation** - Partial information describing the state of the environment |
|||
available to a given agent. (e.g. Vector, Visual, Text) |
|||
* **Policy** - Function for producing decisions from observations. |
|||
* **Reward** - Signal provided at every step used to indicate desirability of an |
|||
agent’s action within the current state of the environment. |
|||
* **State** - The underlying properties of the environment (including all agents |
|||
within it) at a given time. |
|||
* **Step** - Corresponds to each `FixedUpdate` call of the game engine. Is the |
|||
smallest atomic change to the state possible. |
|||
* **Update** - Unity function called each time a frame is rendered. ML-Agents |
|||
logic should not be placed here. |
|||
* **External Coordinator** - ML-Agents class responsible for communication with |
|||
outside processes (in this case, the Python API). |
|||
* **Trainer** - Python class which is responsible for training a given external |
|||
brain. Contains TensorFlow graph which makes decisions for external brain. |