浏览代码
Merge pull request #35 from Unity-Technologies/fix-docs
Merge pull request #35 from Unity-Technologies/fix-docs
Move Wiki to Docs directory/develop-generalizationTraining-TrainerController
GitHub
7 年前
当前提交
daf205da
共有 15 个文件被更改，包括 737 次插入 和 8 次删除
-
10README.md
-
2python/README.md
-
4unity-environment/README.md
-
71docs/Agents-Editor-Interface.md
-
58docs/Example-Environments.md
-
135docs/Getting-Started-with-Balance-Ball.md
-
37docs/Limitations-&-Common-Issues.md
-
143docs/Making-a-new-Unity-Environment.md
-
33docs/Organizing-the-Scene.md
-
36docs/Training-on-Amazon-Web-Service.md
-
44docs/Unity-Agents---Python-API.md
-
43docs/Unity-Agents-Overview.md
-
17docs/Readme.md
-
112docs/Using-TensorFlow-Sharp-in-Unity-(Experimental).md
|
|||
# ML Agents Editor Interface |
|||
|
|||
This page contains an explanation of the use of each of the inspector panels relating to the `Academy`, `Brain`, and `Agent` objects. |
|||
|
|||
## Academy |
|||
|
|||
 |
|||
|
|||
* `Max Steps` - Total number of steps per-episode. `0` corresponds to episodes without a maximum number |
|||
of steps. Once the step counter reaches maximum, the environment will reset. |
|||
* `Frames To Skip` - How many steps of the environment to skip before asking Brains for decisions. |
|||
* `Wait Time` - How many seconds to wait between steps when running in `Inference`. |
|||
* `Configuration` - The engine-level settings which correspond to rendering quality and engine speed. |
|||
* `Width` - Width of the environment window in pixels. |
|||
* `Height` - Width of the environment window in pixels. |
|||
* `Quality Level` - Rendering quality of environment. (Higher is better) |
|||
* `Time Scale` - Speed at which environment is run. (Higher is faster) |
|||
* `Target Frame Rate` - FPS engine attempts to maintain. |
|||
* `Default Reset Parameters` - List of custom parameters that can be changed in the environment on reset. |
|||
|
|||
## Brain |
|||
|
|||
 |
|||
|
|||
* `Brain Parameters` - Define state, observation, and action spaces for the Brain. |
|||
* `State Size` - Length of state vector for brain (In _Continuous_ state space). Or number of possible |
|||
values (in _Discrete_ state space). |
|||
* `Action Size` - Length of action vector for brain (In _Continuous_ state space). Or number of possible |
|||
values (in _Discrete_ action space). |
|||
* `Memory Size` - Length of memory vector for brain. Used with Recurrent networks and frame-stacking CNNs. |
|||
* `Camera Resolution` - Describes height, width, and whether to greyscale visual observations for the Brain. |
|||
* `Action Descriptions` - A list of strings used to name the available actions for the Brain. |
|||
* `State Space Type` - Corresponds to whether state vector contains a single integer (Discrete) or a series of real-valued floats (Continuous). |
|||
* `Action Space Type` - Corresponds to whether action vector contains a single integer (Discrete) or a series of real-valued floats (Continuous). |
|||
* `Type of Brain` - Describes how Brain will decide actions. |
|||
* `External` - Actions are decided using Python API. |
|||
* `Internal` - Actions are decided using internal TensorflowSharp model. |
|||
* `Player` - Actions are decided using Player input mappings. |
|||
* `Heuristic` - Actions are decided using custom `Decision` script, which should be attached to the Brain game object. |
|||
|
|||
### Internal Brain |
|||
|
|||
 |
|||
|
|||
* `Graph Model` : This must be the `bytes` file corresponding to the pretrained Tensorflow graph. (You must first drag this file into your Resources folder and then from the Resources folder into the inspector) |
|||
* `Graph Scope` : If you set a scope while training your tensorflow model, all your placeholder name will have a prefix. You must specify that prefix here. |
|||
* `Batch Size Node Name` : If the batch size is one of the inputs of your graph, you must specify the name if the placeholder here. The brain will make the batch size equal to the number of agents connected to the brain automatically. |
|||
* `State Node Name` : If your graph uses the state as an input, you must specify the name if the placeholder here. |
|||
* `Recurrent Input Node Name` : If your graph uses a recurrent input / memory as input and outputs new recurrent input / memory, you must specify the name if the input placeholder here. |
|||
* `Recurrent Output Node Name` : If your graph uses a recurrent input / memory as input and outputs new recurrent input / memory, you must specify the name if the output placeholder here. |
|||
* `Observation Placeholder Name` : If your graph uses observations as input, you must specify it here. Note that the number of observations is equal to the length of `Camera Resolutions` in the brain parameters. |
|||
* `Action Node Name` : Specify the name of the placeholder corresponding to the actions of the brain in your graph. If the action space type is continuous, the output must be a one dimensional tensor of float of length `Action Space Size`, if the action space type is discrete, the output must be a one dimensional tensor of int of length 1. |
|||
* `Graph Placeholder` : If your graph takes additional inputs that are fixed (example: noise level) you can specify them here. Note that in your graph, these must correspond to one dimensional tensors of int or float of size 1. |
|||
* `Name` : Corresponds to the name of the placeholdder. |
|||
* `Value Type` : Either Integer or Floating Point. |
|||
* `Min Value` and `Max Value` : Specify the range of the value here. The value will be sampled from the uniform distribution ranging from `Min Value` to `Max Value` inclusive. |
|||
|
|||
|
|||
### Player Brain |
|||
|
|||
 |
|||
|
|||
If the action space is discrete, you must map input keys to their corresponding integer values. If the action space is continuous, you must map input keys to their corresponding indices and float values. |
|||
|
|||
## Agent |
|||
|
|||
 |
|||
|
|||
* `Brain` - The brain to register this agent to. Can be dragged into the inspector using the Editor. |
|||
* `Observations` - A list of `Cameras` which will be used to generate observations. |
|||
* `Max Step` - The per-agent maximum number of steps. Once this number is reached, the agent will be reset if `Reset On Done` is checked. |
|||
|
|||
# Example Learning Environments |
|||
|
|||
### About Example Environments |
|||
Unity ML Agents currently contains three example environments which demonstrate various features of the platform. In the coming months more will be added. We are also actively open to adding community contributed environments as examples, as long as they are small, simple, demonstrate a unique feature of the platform, and provide a unique non-trivial challenge to modern RL algorithms. Feel free to submit these environments with a Pull-Request explaining the nature of the environment and task. |
|||
|
|||
Environments are located in `unity-environment/ML-Agents/Examples`. |
|||
|
|||
## 3DBall |
|||
|
|||
 |
|||
|
|||
* Set-up: A balance-ball task, where the agent controls the platform. |
|||
* Goal: The agent must balance the platform in order to keep the ball on it for as long as possible. |
|||
* Agents: The environment contains 12 agents of the same kind, all linked to a single brain. |
|||
* Agent Reward Function: |
|||
* +0.1 for every step the ball remains on the platform. |
|||
* -1.0 if the ball falls from the platform. |
|||
* Brains: One brain with the following state/action space. |
|||
* State space: (Continuous) 8 variables corresponding to rotation of platform, and position, rotation, and velocity of ball. |
|||
* Action space: (Continuous) Size of 2, with one value corresponding to X-rotation, and the other to Z-rotation. |
|||
* Observations: 0 |
|||
* Reset Parameters: None |
|||
|
|||
## GridWorld |
|||
|
|||
 |
|||
|
|||
* Set-up: A version of the classic grid-world task. Scene contains agent, goal, and obstacles. |
|||
* Goal: The agent must navigate the grid to the goal while avoiding the obstacles. |
|||
* Agents: The environment contains one agent linked to a single brain. |
|||
* Agent Reward Function: |
|||
* -0.01 for every step. |
|||
* +1.0 if the agent navigates to the goal position of the grid (episode ends). |
|||
* -1.0 if the agent navigates to an obstacle (episode ends). |
|||
* Brains: One brain with the following state/action space. |
|||
* State space: (Continuous) 6 variables corresponding to position of agent and nearest goal and obstacle. |
|||
* Action space: (Discrete) Size of 4, corresponding to movement in cardinal directions. |
|||
* Observations: One corresponding to top-down view of GridWorld. |
|||
* Reset Parameters: Three, corresponding to grid size, number of obstacles, and number of goals. |
|||
|
|||
|
|||
## Tennis |
|||
|
|||
 |
|||
|
|||
* Set-up: Two-player game where agents control rackets to bounce ball over a net. |
|||
* Goal: The agents must bounce ball between one another while not dropping or sending ball out of bounds. |
|||
* Agents: The environment contains two agent linked to a single brain. |
|||
* Agent Reward Function (independent): |
|||
* -0.1 To last agent to hit ball before going out of bounds or hitting ground/net (episode ends). |
|||
* +0.1 To agent when hitting ball after ball was hit by the other agent. |
|||
* +0.1 To agent who didn't hit ball last when ball hits ground. |
|||
* Brains: One brain with the following state/action space. |
|||
* State space: (Continuous) 6 variables corresponding to position of agent and nearest goal and obstacle. |
|||
* Action space: (Discrete) Size of 4, corresponding to movement toward net, away from net, jumping, and no-movement. |
|||
* Observations: None |
|||
* Reset Parameters: One, corresponding to size of ball. |
|||
|
|||
|
|||
# Getting Started with the Balance Ball Example |
|||
|
|||
 |
|||
|
|||
This tutorial will walk through the end-to-end process of installing Unity Agents, building an example environment, training an agent in it, and finally embedding the trained model into the Unity environment. |
|||
|
|||
Unity ML Agents contains a number of example environments which can be used as templates for new environments, or as ways to test a new ML algorithm to ensure it is functioning correctly. |
|||
|
|||
In this walkthrough we will be using the **3D Balance Ball** environment. The environment contains a number of platforms and balls. Platforms can act to keep the ball up by rotating either horizontally or vertically. Each platform is an agent which is rewarded the longer it can keep a ball balanced on it, and provided a negative reward for dropping the ball. The goal of the training process is to have the platforms learn to never drop the ball. |
|||
|
|||
Let's get started! |
|||
|
|||
## Getting Unity ML Agents |
|||
### Start by installing **Unity 2017.1** or later (required) |
|||
|
|||
Download link available [here](https://store.unity.com/download?ref=update). |
|||
|
|||
If you are new to using the Unity Editor, you can find the general documentation [here](https://docs.unity3d.com/Manual/index.html). |
|||
|
|||
### Clone the repository |
|||
Once installed, you will want to clone the Agents GitHub repository. References will be made throughout to `unity-environment` and `python` directories. Both are located at the root of the repository. |
|||
|
|||
## Building Unity Environment |
|||
Launch the Unity Editor, and log in, if necessary. |
|||
|
|||
1. Open the `unity-environment` folder using the Unity editor. *(If this is not first time running Unity, you'll be able to skip most of these immediate steps, choose directly from the list of recently opened projects)* |
|||
- On the initial dialog, choose `Open` on the top options |
|||
- On the file dialog, choose `unity-environment` and click `Open` *(It is safe to ignore any warning message about non-matching editor installation)* |
|||
- Once the project is open, on the `Project` panel (bottom of the tool), navigate to the folder `Assets/ML-Agents/Examples/3DBall/` |
|||
- Double-click the `Scene` icon (Unity logo) to load all environment assets |
|||
2. Go to `Edit -> Project Settings -> Player` |
|||
- Ensure that `Resolution and Presentation -> Run in Background` is Checked. |
|||
- Ensure that `Resolution and Presentation -> Display Resolution Dialog` is set to Disabled. |
|||
3. Expand the `Ball3DAcademy` GameObject and locate its child object `Ball3DBrain` within the Scene hierarchy in the editor. Ensure Type of Brain for this object is set to `External`. |
|||
4. *File -> Build Settings* |
|||
5. Choose your target platform: |
|||
- (optional) Select “Developer Build” to log debug messages. |
|||
6. Click *Build*: |
|||
- Save environment binary to the `python` sub-directory of the cloned repository *(you may need to click on the down arrow on the file chooser to be able to select that folder)* |
|||
|
|||
## Installing Python API |
|||
In order to train an agent within the framework, you will need to install Python 2 or 3, and the dependencies described below. |
|||
|
|||
### Windows Users |
|||
|
|||
If you are a Windows user who is new to Python/TensorFlow, follow [this guide](https://nitishmutha.github.io/tensorflow/2017/01/22/TensorFlow-with-gpu-for-windows.html) to set up your Python environment. |
|||
|
|||
### Requirements |
|||
* Jupyter |
|||
* Matplotlib |
|||
* numpy |
|||
* Pillow |
|||
* Python (2 or 3) |
|||
* scipy |
|||
* TensorFlow (1.0+) |
|||
|
|||
### Installing Dependencies |
|||
To install dependencies, go into the `python` directory and run: |
|||
|
|||
`pip install .` |
|||
|
|||
or |
|||
|
|||
`pip3 install .` |
|||
|
|||
If your Python environment doesn't include `pip`, see these [instructions](https://packaging.python.org/guides/installing-using-linux-tools/#installing-pip-setuptools-wheel-with-linux-package-managers) on installing it. |
|||
|
|||
Once dependencies are installed, you are ready to test the Ball balance environment from Python. |
|||
|
|||
### Testing Python API |
|||
|
|||
To launch jupyter, run in the command line: |
|||
|
|||
`jupyter notebook` |
|||
|
|||
Then navigate to `localhost:8888` to access the notebooks. If you're new to jupyter, check out the [quick start guide](https://jupyter-notebook-beginner-guide.readthedocs.io/en/latest/execute.html) before you continue. |
|||
|
|||
To ensure that your environment and the Python API work as expected, you can use the `python/Basics` Jupyter notebook. This notebook contains a simple walkthrough of the functionality of the API. Within `Basics`, be sure to set `env_name` to the name of the environment file you built earlier. |
|||
|
|||
## Training the Brain with Reinforcement Learning |
|||
|
|||
### Training with PPO |
|||
In order to train an agent to correctly balance the ball, we will use a Reinforcement Learning algorithm called Proximal Policy Optimization (PPO). This is a method that has been shown to be safe, efficient, and more general purpose than many other RL algorithms, as such we have chosen it as the example algorithm for use with ML Agents. For more information on PPO, OpenAI has a recent [blog post](https://blog.openai.com/openai-baselines-ppo/) explaining it. |
|||
|
|||
In order to train the agents within the Ball Balance environment: |
|||
|
|||
1. Open `python/PPO.ipynb` notebook from Jupyter. |
|||
2. Set `env_name` to whatever you named your environment file. |
|||
3. (optional) Set `run_path` directory to your choice. |
|||
4. Run all cells of notebook except for final. |
|||
|
|||
### Observing Training Progress |
|||
In order to observe the training process in more detail, you can use Tensorboard. |
|||
In your command line, run : |
|||
|
|||
`tensorboard --logdir='summaries` |
|||
|
|||
Then navigate to `localhost:6006`. |
|||
|
|||
From Tensorboard, you will see the summary statistics of six variables: |
|||
* Cumulative Reward - The mean cumulative episode reward over all agents. Should increase during a successful training session. |
|||
* Value Loss - The mean loss of the value function update. Correlates to how well the model is able to predict the value of each state. This should decrease during a succesful training session. |
|||
* Policy Loss - The mean loss of the policy function update. Correlates to how much the policy (process for deciding actions) is changing. The magnitude of this should decrease during a succesful training session. |
|||
* Episode Length - The mean length of each episode in the environment for all agents. |
|||
* Value Estimates - The mean value estimate for all states visited by the agent. Should increase during a successful training session. |
|||
* Policy Entropy - How random the decisions of the model are. Should slowly decrease during a successful training process. If it decreases too quickly, the `beta` hyperparameter should be increased. |
|||
|
|||
## Embedding Trained Brain into Unity Environment _[Experimental]_ |
|||
Once the training process displays an average reward of ~75 or greater, and there has been a recently saved model (denoted by the `Saved Model` message) you can choose to stop the training process by stopping the cell execution. Once this is done, you now have a trained TensorFlow model. You must now convert the saved model to a Unity-ready format which can be embedded directly into the Unity project by following the steps below. |
|||
|
|||
### Setting up TensorFlowSharp Support |
|||
Because TensorFlowSharp support is still experimental, it is disabled by default. In order to enable it, you must follow these steps. Please note that the `Internal` Brain mode will only be available once completing these steps. |
|||
|
|||
1. Make sure you are using Unity 2017.1 or newer. |
|||
2. Make sure the TensorFlowSharp plugin is in your Asset folder. A Plugins folder which includes TF# can be downloaded [here](https://s3.amazonaws.com/unity-agents/TFSharpPlugin.unitypackage). |
|||
3. Go to `Edit` -> `Project Settings` -> `Player` |
|||
4. For each of the platforms you target (**`PC, Mac and Linux Standalone`**, **`iOS`** or **`Android`**): |
|||
1. Go into `Other Settings`. |
|||
2. Select `Scripting Runtime Version` to `Experimental (.NET 4.6 Equivalent)` |
|||
3. In `Scripting Defined Symbols`, add the flag `ENABLE_TENSORFLOW` |
|||
5. Restart the Unity Editor. |
|||
|
|||
### Embedding the trained model into Unity |
|||
|
|||
1. Run the final cell of the notebook under "Export the trained TensorFlow graph" to produce an `<env_name >.bytes` file. |
|||
2. Move `<env_name>.bytes` from `python/models/...` into `unity-environment/Assets/ML-Agents/Examples/3DBall/TFModels/`. |
|||
3. Open the Unity Editor, and select the `3DBall` scene as described above. |
|||
4. Select the `3DBallBrain` object from the Scene hierarchy. |
|||
5. Change the `Type of Brain` to `Internal`. |
|||
6. Drag the `<env_name>.bytes` file from the Project window of the Editor to the `Graph Model` placeholder in the `3DBallBrain` inspector window. |
|||
7. Set the `Graph Placeholder` size to 1. |
|||
8. Add a placeholder called `epsilon` with a type of `floating point` and a range of values from 0 to 0. |
|||
9. Press the Play button at the top of the editor. |
|||
|
|||
If you followed these steps correctly, you should now see the trained model being used to control the behavior of the balance ball within the Editor itself. From here you can re-build the Unity binary, and run it standalone with your agent's new learned behavior built right in. |
|||
|
|||
# Limitations and Common Issues |
|||
|
|||
## Unity SDK |
|||
### Headless Mode |
|||
Currently headless mode is disabled. We hope to address these in a future version of Unity. |
|||
|
|||
### Rendering Speed and Synchronization |
|||
Currently the speed of the game physics can only be increased to 100x real-time. The Academy also moves in time with FixedUpdate() rather than Update(), so game behavior tied to frame updates may be out of sync. |
|||
|
|||
### macOS Metal Support |
|||
When running a Unity Environment on macOS using Metal rendering, the application can crash when the lock-screen is open. The solution is to set rendering to OpenGL. This can be done by navigating: `Edit -> Project Settings -> Player`. Clicking on `Other Settings`. Unchecking `Auto Graphics API for Mac`. Setting `OpenGL Core` to be above `Metal` in the priority list. |
|||
|
|||
## Python API |
|||
|
|||
### Unity Environment Permission Error |
|||
|
|||
If you directly import your Unity environment without building it in the editor, you might need to give it additionnal permissions to execute it. |
|||
|
|||
If you receive such a permission error on macOS, run: |
|||
|
|||
`chmod -R 755 *.app` |
|||
|
|||
or on Linux: |
|||
|
|||
`chmod -R 755 *.x86_64` |
|||
|
|||
On Windows, you can find instructions [here](https://technet.microsoft.com/en-us/library/cc754344(v=ws.11).aspx). |
|||
|
|||
### Filename not found |
|||
|
|||
If you receive a file-not-found error while attempting to launch an environment, ensure that the environment files are in the root repository directory. For example, if there is a sub-folder containing the environment files, those files should be removed from the sub-folder and moved to the root. |
|||
|
|||
### Communication port {} still in use |
|||
|
|||
If you receive an exception `"Couldn't launch new environment because communication port {} is still in use. "`, you can change the worker number in the python script when calling |
|||
|
|||
`UnityEnvironment(file_name=filename, worker_num=X)` |
|||
|
|||
# Making a new Learning Environment |
|||
|
|||
This tutorial walks through the process of creating a Unity Environment. A Unity Environment is an application built using the Unity Engine which can be used to train Reinforcement Learning agents. |
|||
|
|||
## Setting up the Unity Project |
|||
|
|||
1. Open an existing Unity project, or create a new one and import the RL interface package: |
|||
* [ML-Agents package without TensorflowSharp](https://s3.amazonaws.com/unity-agents/ML-AgentsNoPlugin.unitypackage) |
|||
* [ML-Agents package with TensorflowSharp](https://s3.amazonaws.com/unity-agents/ML-AgentsWithPlugin.unitypackage) |
|||
|
|||
2. Rename `TemplateAcademy.cs` (and the contained class name) to the desired name of your new academy class. All Template files are in the folder `Assets -> Template -> Scripts`. Typical naming convention is `YourNameAcademy`. |
|||
|
|||
3. Attach `YourNameAcademy.cs` to a new empty game object in the currently opened scene (`Unity` -> `GameObject` -> `Create Empty`) and rename this game object to `YourNameAcademy`. Since `YourNameAcademy` will be used to control all the environment logic, ensure the attached-to object is one which will remain in the scene regardless of the environment resetting, or other within-environment behavior. |
|||
|
|||
4. Attach `Brain.cs` to a new empty game object and rename this game object to `YourNameBrain1`. Set this game object as a child of `YourNameAcademy` (Drag `YourNameBrain1` into `YourNameAcademy`). Note that you can have multiple brains in the Academy but they all must have different names. |
|||
|
|||
5. Disable Window Resolution dialogue box and Splash Screen. |
|||
1. Go to `Edit` -> `Project Settings` -> `Player` -> `Resolution and Presentation`. |
|||
2. Set `Display Resolution Dialogue` to `Disabled`. |
|||
3.Check `Run In Background`. |
|||
4. Click `Splash Image`. |
|||
5. Uncheck `Show Splash Screen` _(Unity Pro only)_. |
|||
|
|||
6. If you will be using Tensorflow Sharp in Unity, you must: |
|||
1. Make sure you are using Unity 2017.1 or newer. |
|||
2. Make sure the TensorflowSharp plugin is in your Asset folder. It can be downloaded [here](https://s3.amazonaws.com/unity-agents/TFSharpPlugin.unitypackage). |
|||
3. Go to `Edit` -> `Project Settings` -> `Player` |
|||
4. For each of the platforms you target (**`PC, Mac and Linux Standalone`**, **`iOS`** or **`Android`**): |
|||
1. Go into `Other Settings`. |
|||
2. Select `Scripting Runtime Version` to `Experimental (.NET 4.6 Equivalent)` |
|||
3. In `Scripting Defined Symbols`, add the flag `ENABLE_TENSORFLOW` |
|||
5. Note that some of these changes will require a Unity Restart |
|||
|
|||
# Implementing `YourNameAcademy` |
|||
|
|||
1. Click on the game object **`YourNameAcademy`**. |
|||
|
|||
2. In the inspector tab, you can modify the characteristics of the academy: |
|||
* **`Max Steps`** Maximum length of each episode (set to 0 if you want do not want the environment to reset after a certain time). |
|||
* **`Wait Time`** Real-time between steps when running environment in test-mode. Used only when steps happen in `Update()`. |
|||
* **`Frames To Skip`** Number of frames (or physics updates) to skip between steps. The agents will act at every frame but get new actions only at every step. |
|||
* **`Training Configuration`** and **`Inference Configuration`** The first defines the configuration of the Engine at training time and the second at test / inference time. The training mode corresponds only to external training when the reset parameter `train_model` was set to True. The adjustable parameters are as follows: |
|||
* `Width` and `Height` Correspond to the width and height in pixels of the window (must be both greater than 0). Typically set it to a small size during training, and a larger size for visualization during inference. |
|||
* `Quality Level` Determines how mush rendering is performed. Typically set to small value during training and higher value for visualization during inference. |
|||
* `Time Scale` Physics speed. If environment utilized physics calculations, increase this during training, and set to `1.0f` during inference. Otherwise, set it to `1.0f`. |
|||
* `Target Frame Rate` Frequency of frame rendering. If environment utilizes observations, increase this during training, and set to `60` during inference. If no observations are used, this can be set to `1` during training. |
|||
* **`Default Reset Parameters`** You can set the default configuration to be passed at reset. This will be a mapping from strings to float values that you can call in the academy with `resetParameters["YourDefaultParameter"]` |
|||
|
|||
3. Within **`InitializeAcademy()`**, you can define the initialization of the Academy. Note that this command is ran only once at the beginning of the training session. |
|||
|
|||
3. Within **`AcademyStep()`**, you can define the environment logic each step. Use this function to modify the environment for the agents that will live in it. |
|||
|
|||
4. Within **`AcademyReset()`**, you can reset the environment for a new episode. It should contain environment-specific code for setting up the environment. Note that `AcademyReset()` is called at the beginning of the training session to ensure the first episode is similar to the others. |
|||
|
|||
## Implementing `YourNameBrain` |
|||
For each Brain game object in your academy : |
|||
|
|||
1. Click on the game object `YourNameBrain` |
|||
|
|||
2. In the inspector tab, you can modify the characteristics of the brain in **`Brain Parameters`** |
|||
* `State Size` Number of variables within the state provided to the agent(s). |
|||
* `Action Size` The number of possible actions for each individual agent to take. |
|||
* `Memory Size` The number of floats the agents will remember each step. |
|||
* `Camera Resolutions` A list of flexible length that contains resolution parameters : `height` and `width` define the number dimensions of the camera outputs in pixels. Check `Black And White` if you want the camera outputs to be black and white. |
|||
* `Action Descriptions` A list describing in human-readable language the meaning of each available action. |
|||
* `State Space Type` and `Action Space Type`. Either `discrete` or `continuous`. |
|||
* `discrete` corresponds to describing the action space with an `int`. |
|||
* `continuous` corresponds to describing the action space with an array of `float`. |
|||
|
|||
3. If you want to collect data on you play, you can check the box **`Collect Data`**. This way, the states and actions passing through the brain will be saved in the folder `saved_plays`. |
|||
|
|||
4. You can chose what kind of brain you want `YourNameBrain` to be. There are four possibilities: |
|||
* `External` : You need at least one of your brains to be external if you wish to interact with your environment from python. |
|||
* `Player` : To control you agents manually. If the action space is discrete, you must map input keys to their corresponding integer values. If the action space is continuous, you must map input keys to their corresponding indices and float values. |
|||
* `Heuristic` : You can have your brain automatically react to the observations and states in a customizable way. You will need to drag a `Decision` script into `YourNameBrain`. To create a custom reaction, you must : |
|||
* Rename `TemplateDecision.cs` (and the contained class name) to the desired name of your new reaction. Typical naming convention is `YourNameDecision`. |
|||
* Implement `Decide`: Given the state, observation and memory of an agent, this function must return an array of floats corresponding to the actions taken by the agent. If the action space type is discrete, the array must be of size 1. |
|||
* Optionally, implement `MakeMemory`: Given the state, observation and memory of an agent, this function must return an array of floats corresponding to the new memories of the agent. |
|||
* `Internal` : Note that you must have Tensorflow Sharp setup (see top of this page). Here are the fields that must be completed: |
|||
* `Graph Model` : This must be the `bytes` file corresponding to the pretrained Tensorflow graph. (You must first drag this file into your Resources folder and then from the Resources folder into the inspector) |
|||
* `Graph Scope` : If you set a scope while training your tensorflow model, all your placeholder name will have a prefix. You must specify that prefix here. |
|||
* `Batch Size Node Name` : If the batch size is one of the inputs of your graph, you must specify the name if the placeholder here. The brain will make the batch size equal to the number of agents connected to the brain automatically. |
|||
* `State Node Name` : If your graph uses the state as an input, you must specify the name if the placeholder here. |
|||
* `Recurrent Input Node Name` : If your graph uses a recurrent input / memory as input and outputs new recurrent input / memory, you must specify the name if the input placeholder here. |
|||
* `Recurrent Output Node Name` : If your graph uses a recurrent input / memory as input and outputs new recurrent input / memory, you must specify the name if the output placeholder here. |
|||
* `Observation Placeholder Name` : If your graph uses observations as input, you must specify it here. Note that the number of observations is equal to the length of `Camera Resolutions` in the brain parameters. |
|||
* `Action Node Name` : Specify the name of the placeholder corresponding to the actions of the brain in your graph. If the action space type is continuous, the output must be a one dimensional tensor of float of length `Action Space Size`, if the action space type is discrete, the output must be a one dimensional tensor of int of length 1. |
|||
* `Graph Placeholder` : If your graph takes additional inputs that are fixed (example: noise level) you can specify them here. Note that in your graph, these must correspond to one dimensional tensors of int or float of size 1. |
|||
* `Name` : Corresponds to the name of the placeholdder. |
|||
* `Value Type` : Either Integer or Floating Point. |
|||
* `Min Value` and 'Max Value' : Specify the minimum and maximum values (included) the placeholder can take. The value will be sampled from the uniform distribution at each step. If you want this value to be fixed, set both `Min Value` and `Max Value` to the same number. |
|||
|
|||
## Implementing `YourNameAgent` |
|||
|
|||
1. Rename `TemplateAgent.cs` (and the contained class name) to the desired name of your new agent. Typical naming convention is `YourNameAgent`. |
|||
|
|||
2. Attach `YourNameAgent.cs` to the game object that represents your agent. (Example: if you want to make a self driving car, attach `YourNameAgent.cs` to a car looking game object) |
|||
|
|||
3. In the inspector menu of you agent, drag the brain game object you want to use with this agent into the corresponding `Brain` box. Please note that you can have multiple agents with the same brain. If you want to give an agent a brain or change his brain via script, please use the method `ChangeBrain()`. |
|||
|
|||
4. In the inspector menu of you agent, you can specify what cameras, your agent will use as its observations. To do so, drag the desired number of cameras into the `Observations` field. Note that if you want a camera to move along your agent, you can set this camera as a child of your agent |
|||
|
|||
5. If `Reset On Done` is checked, `Reset()` will be called when the agent is done. Else, `AgentOnDone()` will be called. Note that if `Reset On Done` is unchecked, the agent will remain "done" until the Academy resets. This means that it will not take actions in the environment. |
|||
|
|||
6. Implement the following functions in `YourNameAgent.cs` : |
|||
* `InitializeAgent()` : Use this method to initialize your agent. This method is called then the agent is created. |
|||
* `CollectState()` : Must return a list of floats corresponding to the state the agent is in. If the state space type is discrete, return a list of length 1 containing the float equivalent of your state. |
|||
* `AgentStep()` : This function will be called every frame, you must define what your agent will do given the input actions. You must also specify the rewards and whether or not the agent is done. To do so, modify the public fields of the agent `reward` and `done`. |
|||
* `AgentReset()` : This function is called at start, when the Academy resets and when the agent is done (if `Reset On Done` is checked). |
|||
* `AgentOnDone()` : If `Reset On Done` is not checked, this function will be called when the agent is done. `Reset()` will only be called when the Academy resets. |
|||
|
|||
If you create Agents via script, we recommend you save them as prefabs and instantiate them either during steps or resets. If you do, you can use `GiveBrain(brain)` to have the agent subscribe to a specific brain. You can also use `RemoveBrain()` to unsubscribe from a brain. |
|||
|
|||
# Defining the reward function |
|||
The reward function is the set of circumstances and event which we want to reward or punish the agent for making happen. Here are some examples of positive and negative rewards: |
|||
* Positive |
|||
* Reaching a goal |
|||
* Staying alive |
|||
* Defeating an enemy |
|||
* Gaining health |
|||
* Finishing a level |
|||
* Negative |
|||
* Taking damage |
|||
* Failing a level |
|||
* The agent’s death |
|||
|
|||
Small negative rewards are also typically used each step in scenarios where the optimal agent behavior is to complete an episode as quickly as possible. |
|||
|
|||
Note that the reward is reset to 0 at every step, you must add to the reward (`reward += rewardIncrement`). If you use `skipFrame` in the Academy and set your rewards instead of incrementing them, you might loose information since the reward is sent at every step, not at every frame. |
|||
|
|||
## Agent Monitor |
|||
* You can add the script `AgentMonitor.cs` to any gameObject with a component `YourNameAgent.cs`. In the inspector of this component, you will see: |
|||
* `Fixed Position` : If this box is checked, the monitor will be on the left corner of the screen and will remain here. Note that you can only have one agent with a fixed monitor or multiple monitors will overlap. |
|||
* `Vertical Offset`: If `Fixed Position` is unchecked, the monitor will follow the Agent on the screen. Use `Vertical Offset` to decide how far above the agent the monitor should be. |
|||
* `Display Brain Name` : If this box is checked, the name of the brain will appear in the monitor. (Can be useful if you have similar agents using different brains). |
|||
* `Display Brain Type` : If this box is checked, the type of the brain of the agent will be displayed. |
|||
* `Display FrameCount` : If this box is checked, the number of frames that elapsed since the agent was reset will be displayed. |
|||
* `Display Current Reward`: If this box is checked, the current reward of the agent will be displayed. |
|||
* `Display Max Reward` : If this box is checked, the maximum reward obtained during this training session will be displayed. |
|||
* `Display State` : If this box is checked, the current state of the agent will be displayed. |
|||
* `Display Action` : If this box is checked, the current action the agent performs will be displayed. |
|||
|
|||
If you passed a `value` from an external brain, the value will be displayed as a bar (green if value is positive / red if value is negative) above the monitor. The bar's maximum value is set to 1 by default but if the value of the agent is above this number, it becomes the new maximum. |
|||
|
|||
# Organizing the Scene Layout |
|||
|
|||
This tutorial will help you understand how to organize your scene when using Agents in your Unity environment. |
|||
|
|||
## ML-Agents Game Objects |
|||
|
|||
There are three kinds of game objects you need to include in your scene in order to use Unity ML-Agents: |
|||
* Academy |
|||
* Brain |
|||
* Agents |
|||
|
|||
#### Keep in mind : |
|||
* There can only be one Academy game object in a scene. |
|||
* You can have multiple Brain game objects but they must be child of the Academy game object. |
|||
|
|||
#### Here is an example of what your scene hierarchy should look like : |
|||
|
|||
 |
|||
|
|||
### Functionality |
|||
|
|||
#### The Academy |
|||
The Academy is responsible for: |
|||
* Synchronizing the environment and keeping all agent's steps in pace. As such, there can only be one per scene. |
|||
* Determining the speed of the engine, its quality, and the display's resolution. |
|||
* Modifying the environment at every step and every reset according to the logic defined in `AcademyStep()` and `AcademyReset()`. |
|||
* Coordingating the Brains which must be set as children of the Academy. |
|||
|
|||
#### Brains |
|||
Each brain corresponds to a specific Decision-making method. This often aligns with a specific neural network model. A Brains is responsible for deciding the action of all the Agents which are linked to it. There can be multiple brains in the same scene and multiple agents can subscribe to the same brain. |
|||
|
|||
#### Agents |
|||
Each agent within a scene takes actions according to the decisions provided by it's linked Brain. There can be as many Agents of as many types as you like in the scene. The state size and action size of each agent must match the brain's parameters in order for the Brain to decide actions for it. |
|||
|
|||
# Training on Amazon Web Service |
|||
|
|||
This page contains instructions for setting up an EC2 instance on Amazon Web Service for use in training ML-Agents environments. Current limitations of the Unity Engine require that a screen be available to render to. In order to make this possible when training on a remote server, a virtual screen is required. We can do this by installing Xorg and creating a virtual screen. Once installed and created, we can display the Unity environment in the virtual environment, and train as we would on a local machine. |
|||
|
|||
Instructions here are adapted from this [Medium post](https://medium.com/towards-data-science/how-to-run-unity-on-amazon-cloud-or-without-monitor-3c10ce022639) on running general Unity applications in the cloud. |
|||
|
|||
1. To begin with, you will need an EC2 instance which contains the latest Nvidia drivers, CUDA8, and cuDNN. There are a number of external tutorials which describe this, such as: |
|||
* [Getting CUDA 8 to Work With openAI Gym on AWS and Compiling Tensorflow for CUDA 8 Compatibility](https://davidsanwald.github.io/2016/11/13/building-tensorflow-with-gpu-support.html) |
|||
* [Installing TensorFlow on an AWS EC2 P2 GPU Instance](http://expressionflow.com/2016/10/09/installing-tensorflow-on-an-aws-ec2-p2-gpu-instance/) |
|||
* [Updating Nvidia CUDA to 8.0.x in Ubuntu 16.04 – EC2 Gx instance](https://aichamp.wordpress.com/2016/11/09/updating-nvidia-cuda-to-8-0-x-in-ubuntu-16-04-ec2-gx-instance/) |
|||
2. Move `python` to remote instance. |
|||
2. Install the required packages with `pip install .`. |
|||
3. Run the following commands to install Xorg: |
|||
``` |
|||
sudo apt-get update |
|||
sudo apt-get install -y xserver-xorg mesa-utils |
|||
sudo nvidia-xconfig -a --use-display-device=None --virtual=1280x1024 |
|||
``` |
|||
4. Restart the EC2 instance. |
|||
5. On start-up, run: |
|||
``` |
|||
sudo /usr/bin/X :0 & |
|||
export DISPLAY=:0 |
|||
``` |
|||
Depending on how Xorg is configured, you may need to run `sudo killall Xorg` before starting Xorg with the above command. |
|||
6. To ensure the installation was succesful, run `glxgears`. If there are no errors, then Xorg is correctly configured. |
|||
7. There is a bug in _Unity 2017.1_ which requires the uninstallation of `libxrandr2`, which can be removed with `apt-get remove --purge libxrandr2`. This is scheduled to be fixed in 2017.3. |
|||
|
|||
If all steps worked correctly, upload an example binary built for Linux to the instance, and test it from python with: |
|||
```python |
|||
from unityagents import UnityEnvironment |
|||
|
|||
env = UnityEnvironment(your_env) |
|||
``` |
|||
|
|||
You should receive a message confirming that the environment was loaded succesfully. |
|||
|
|||
# Python API |
|||
|
|||
_Notice: Currently communication between Unity and Python takes place over an open socket without authentication. As such, please make sure that the network where training takes place is secure. This will be addressed in a future release._ |
|||
|
|||
## Loading a Unity Environment |
|||
|
|||
Python-side communication happens through `UnityEnvironment` which is located in `python/unityagents`. To load a Unity environment from a built binary file, put the file in the same directory as `unityagents`. In python, run: |
|||
|
|||
|
|||
```python |
|||
from unityagents import UnityEnvironment |
|||
env = UnityEnvironment(file_name=filename, worker_num=0) |
|||
``` |
|||
|
|||
* `file_name` is the name of the environment binary (located in the root directory of the python project). |
|||
* `worker_num` indicates which port to use for communication with the environment. For use in parallel training regimes such as A3C. |
|||
|
|||
## Interacting with a Unity Environment |
|||
|
|||
A BrainInfo object contains the following fields: |
|||
|
|||
* **`observations`** : A list of 4 dimensional numpy arrays. Matrix n of the list corresponds to the n<sup>th</sup> observation of the brain. |
|||
* **`states`** : A two dimensional numpy array of dimension `(batch size, state size)` if the state space is continuous and `(batch size, state size)` if the state space is discrete. |
|||
* **`memories`** : A two dimensional numpy array of dimension `(batch size, memory size)` which corresponds to the memories sent at the previous step. |
|||
* **`rewards`** : A list as long as the number of agents using the brain containing the rewards they each obtained at the previous step. |
|||
* **`local_done`** : A list as long as the number of agents using the brain containing `done` flags (wether or not the agent is done). |
|||
* **`agents`** : A list of the unique ids of the agents using the brain. |
|||
|
|||
Once loaded, `env` can be used in the following way: |
|||
- **Print : `print(str(env))`** |
|||
Prints all parameters relevant to the loaded environment and the external brains. |
|||
- **Reset : `env.reset(train_model=True, config=None)`** |
|||
Send a reset signal to the environment, and provides a dictionary mapping brain names to BrainInfo objects. |
|||
- `train_model` indicates whether to run the environment in train (`True`) or test (`False`) mode. |
|||
- `config` is an optional dictionary of configuration flags specific to the environment. For more information on adding optional config flags to an environment, see [here](Making-a-new-Unity-Environment.md#implementing-yournameacademy). For generic environments, `config` can be ignored. `config` is a dictionary of strings to floats where the keys are the names of the `resetParameters` and the values are their corresponding float values. |
|||
- **Step : `env.step(action, memory=None, value = None)`** |
|||
Sends a step signal to the environment using the actions. Note that if you have more than one brain in the environment, you must provide a dictionary from brain names to actions. |
|||
- `action` can be one dimensional arrays or two dimensional arrays if you have multiple agents per brains. |
|||
- `memory` is an optional input that can be used to send a list of floats per agents to be retrieved at the next step. |
|||
- `value` is an optional input that be used to send a single float per agent to be displayed if and `AgentMonitor.cs` component is attached to the agent. |
|||
Returns a dictionary mapping brain names to BrainInfo objects. |
|||
- **Close : `env.close()`** |
|||
Sends a shutdown signal to the environment and closes the communication socket. |
|||
|
|||
|
|||
# Learning Environments Overview |
|||
|
|||
 |
|||
|
|||
A visual depiction of how an Learning Environment might be configured within ML-Agents. |
|||
|
|||
The three main kinds of objects within any Agents Learning Environment are: |
|||
|
|||
* Agent - Each Agent can have a unique set of states and observations, take unique actions within the environment, and can receive unique rewards for events within the environment. An agent's actions are decided by the brain it is linked to. |
|||
* Brain - Each Brain defines a specific state and action space, and is responsible for deciding which actions each of its linked agents will take. Brains can be set to one of four modes: |
|||
* External - Action decisions are made using TensorFlow (or your ML library of choice) through communication over an open socket with our Python API. |
|||
* Internal (Experimental) - Actions decisions are made using a trained model embedded into the project via TensorFlowSharp. |
|||
* Player - Action decisions are made using player input. |
|||
* Heuristic - Action decisions are made using hand-coded behavior. |
|||
* Academy - The Academy object within a scene also contains as children all Brains within the environment. Each environment contains a single Academy which defines the scope of the environment, in terms of: |
|||
* Engine Configuration - The speed and rendering quality of the game engine in both training and inference modes. |
|||
* Frameskip - How many engine steps to skip between each agent making a new decision. |
|||
* Global episode length - How long the the episode will last. When reached, all agents are set to done. |
|||
|
|||
The states and observations of all agents with brains set to External are collected by the External Communicator, and communicated via the Python API. By setting multiple agents to a single brain, actions can be decided in a batch fashion, taking advantage of the inherently parallel computations of neural networks. For more information on how these objects work together within a scene, see our wiki page. |
|||
|
|||
## Flexible Training Scenarios |
|||
|
|||
With the Unity ML-Agents, a variety of different kinds of training scenarios are possible, depending on how agents, brains, and rewards are connected. We are excited to see what kinds of novel and fun environments the community creates. For those new to training intelligent agents, below are a few examples that can serve as inspiration. Each is a prototypical environment configurations with a description of how it can be created using the ML-Agents SDK. |
|||
|
|||
* **Single-Agent** - A single agent linked to a single brain. The traditional way of training an agent. An example is any single-player game, such as Chicken. [Video Link](https://www.youtube.com/watch?v=fiQsmdwEGT8&feature=youtu.be). |
|||
* **Simultaneous Single-Agent** - Multiple independent agents with independent reward functions linked to a single brain. A parallelized version of the traditional training scenario, which can speed-up and stabilize the training process. An example might be training a dozen robot-arms to each open a door simultaneously. [Video Link](https://www.youtube.com/watch?v=fq0JBaiCYNA). |
|||
* **Adversarial Self-Play** - Two interacting agents with inverse reward functions linked to a single brain. In two-player games, adversarial self-play can allow an agent to become increasingly more skilled, while always having the perfectly matched opponent: itself. This was the strategy employed when training AlphaGo, and more recently used by OpenAI to train a human-beating 1v1 Dota 2 agent. |
|||
* **Cooperative Multi-Agent** - Multiple interacting agents with a shared reward function linked to either a single or multiple different brains. In this scenario, all agents must work together to accomplish a task than couldn’t be done alone. Examples include environments where each agent only has access to partial information, which needs to be shared in order to accomplish the task or collaboratively solve a puzzle. (Demo project coming soon) |
|||
* **Competitive Multi-Agent** - Multiple interacting agents with inverse reward function linked to either a single or multiple different brains. In this scenario, agents must compete with one another to either win a competition, or obtain some limited set of resources. All team sports would fall into this scenario. (Demo project coming soon) |
|||
* **Ecosystem** - Multiple interacting agents with independent reward function linked to either a single or multiple different brains. This scenario can be thought of as creating a small world in which animals with different goals all interact, such a savanna in which there might be zebras, elephants, and giraffes, or an autonomous driving simulation within an urban environment. (Demo project coming soon) |
|||
|
|||
## Additional Features |
|||
|
|||
Beyond the flexible training scenarios made possible by the Academy/Brain/Agent system, ML-Agents also includes other features which improve the flexibility and interpretability of the training process. |
|||
|
|||
* **Monitoring Agent’s Decision Making** - Since communication in ML-Agents is a two-way street, we provide an Agent Monitor class in Unity which can display aspects of the trained agent, such as policy and value output within the Unity environment itself. By providing these outputs in real-time, researchers and developers can more easily debug an agent’s behavior. |
|||
|
|||
* **Curriculum Learning** - It is often difficult for agents to learn a complex task at the beginning of the training process. Curriculum learning is the process of gradually increasing the difficulty of a task to allow more efficient learning. ML-Agents supports setting custom environment parameters every time the environment is reset. This allows elements of the environment related to difficulty or complexity to be dynamically adjusted based on training progress. |
|||
|
|||
* **Complex Visual Observations** - Unlike other platforms, where the agent’s observation might be limited to a single vector or image, ML-Agents allows multiple cameras to be used for observations per agent. This enables agents to learn to integrate information from multiple visual streams, as would be the case when training a self-driving car which required multiple cameras with different viewpoints, a navigational agent which might need to integrate aerial and first-person visuals, or an agent which takes both a raw visual input, as well as a depth-map or object-segmented image. |
|||
|
|||
* **Imitation Learning (Coming Soon)** - It is often more intuitive to simply demonstrate the behavior we want an agent to perform, rather than attempting to have it learn via trial-and-error methods. In a future release, ML-Agents will provide the ability to record all state/action/reward information for use in supervised learning scenarios, such as imitation learning. By utilizing imitation learning, a player can provide demonstrations of how an agent should behave in an environment, and then utilize those demonstrations to train an agent in either a standalone fashion, or as a first-step in a reinforcement learning process. |
|||
|
|||
# Unity ML Agents Documentation |
|||
|
|||
## Basic |
|||
* [Unity ML Agents Overview](Unity-Agents-Overview.md) |
|||
* [Getting Started with the Balance Ball Environment](Getting-Started-with-Balance-Ball.md) |
|||
* [Example Environments](Example-Environments.md) |
|||
|
|||
## Advanced |
|||
* [How to make a new Unity Environment](Making-a-new-Unity-Environment.md) |
|||
* [How to organize the Scene](Organizing-the-Scene.md) |
|||
* [How to use the Python API](Unity-Agents---Python-API.md) |
|||
* [How to use TensorflowSharp inside Unity [Experimental]](Using-TensorFlow-Sharp-in-Unity-(Experimental).md) |
|||
* [Agents SDK Inspector Descriptions](Agents-Editor-Interface.md) |
|||
* [Training on the Cloud with Amazon Web Services](Training-on-Amazon-Web-Service.md) |
|||
|
|||
## Help |
|||
* [Limitations & Common Issues](Limitations-&-Common-Issues.md) |
|||
|
|||
# Using TensorFlowSharp in Unity (Experimental) |
|||
|
|||
Unity now offers the possibility to use pretrained TensorFlow graphs inside of the game engine. This was made possible thanks to [the TensorFlowSharp project](https://github.com/migueldeicaza/TensorFlowSharp). |
|||
|
|||
_Notice: This feature is still experimental. While it is possible to embed trained models into Unity games, Unity Technologies does not officially support this use-case for production games at this time. As such, no guarantees are provided regarding the quality of experience. If you encounter issues regarding battery life, or general performance (especially on mobile), please let us know._ |
|||
|
|||
## Supported devices : |
|||
|
|||
* Linux 64 bits |
|||
* Mac OSX 64 bits |
|||
* Windows 64 bits |
|||
* iOS (Requires additional steps) |
|||
* Android |
|||
|
|||
## Requirements |
|||
|
|||
* Unity 2017.1 or above |
|||
* Unity Tensorflow Plugin ([Download here](https://s3.amazonaws.com/unity-agents/TFSharpPlugin.unitypackage)) |
|||
# Using TensorflowSharp with ML-Agents |
|||
|
|||
In order to bring a fully trained agent back into Unity, you will need to make sure the nodes of your graph have appropriate names. You can give names to nodes in Tensorflow : |
|||
```python |
|||
variable= tf.identity(variable, name="variable_name") |
|||
``` |
|||
We recommend using the following naming convention: |
|||
* Name the batch size input placeholder `batch_size` |
|||
* Name the input state placeholder `state` |
|||
* Name the output node `action` |
|||
* Name the recurrent vector (memory) input placeholder `recurrent_in` (if any) |
|||
* Name the recurrent vector (memory) output node `recurrent_out` (if any) |
|||
* Name the observations placeholders input placeholders `observation_i` where `i` is the index of the observation (starting at 0) |
|||
|
|||
You can have additional placeholders for float or integers but they must be placed in placeholders of dimension 1 and size 1. (Be sure to name them) |
|||
|
|||
It is important that the inputs and outputs of the graph are exactly the one you receive / give when training your model with an `External` brain. This means you cannot have any operations such as reshaping outside of the graph. |
|||
The object you get by calling `step` or `reset` has fields `states`, `observations` and `memories` which must correspond to the placeholders of your graph. Similarly, the arguments `action` and `memory` you pass to `step` must correspond to the output nodes of your graph. |
|||
|
|||
While training your Agent using the Python API, you can save your graph at any point of the training. Note that the argument `output_node_names` must be the name of the tensor your graph outputs (separated by a coma if multiple outputs). In this case, it will be either `action` or `action,recurrent_out` if you have recurrent outputs. |
|||
```python |
|||
from tensorflow.python.tools import freeze_graph |
|||
|
|||
freeze_graph.freeze_graph(input_graph = model_path +'/raw_graph_def.pb', |
|||
input_binary = True, |
|||
input_checkpoint = last_checkpoint, |
|||
output_node_names = "action", |
|||
output_graph = model_path +'/your_name_graph.bytes' , |
|||
clear_devices = True, initializer_nodes = "",input_saver = "", |
|||
restore_op_name = "save/restore_all", filename_tensor_name = "save/Const:0") |
|||
``` |
|||
Your model will be saved with the name `your_name_graph.bytes` and will contain both the graph and associated weights. Note that you must save your graph as a bytes file so Unity can load it. |
|||
|
|||
## Inside Unity |
|||
|
|||
Go to `Edit` -> `Player Settings` and add `ENABLE_TENSORFLOW` to the `Scripting Define Symbols` for each type of device you want to use (**`PC, Mac and Linux Standalone`**, **`iOS`** or **`Android`**). |
|||
|
|||
Set the Brain you used for training to `Internal`. Drag `your_name_graph.bytes` into Unity and then drag it into The `Graph Model` field in the Brain. If you used a scope when training you graph, specify it in the `Graph Scope` field. Specify the names of the nodes you used in your graph. If you followed these instructions well, the agents in your environment that use this brain will use you fully trained network to make decisions. |
|||
|
|||
# iOS additional instructions for building |
|||
|
|||
* Once you build for iOS in the editor, Xcode will launch. |
|||
* In `General` -> `Linked Frameworks and Libraries`: |
|||
* Add a framework called `Framework.accelerate` |
|||
* Remove the library `libtensorflow-core.a` |
|||
* In `Build Settings`->`Linking`->`Other Linker Flags`: |
|||
* Double Click on the flag list |
|||
* Type `-force_load` |
|||
* Drag the library `libtensorflow-core.a` from the `Project Navigator` on the left under `Libraries/ML-Agents/Plugins/iOS` into the flag list. |
|||
|
|||
# Using TensorflowSharp without ML-Agents |
|||
|
|||
Beyond controlling an in-game agent, you may desire to use TensorFlowSharp for more general computation. The below instructions describe how to generally embed Tensorflow models without using the ML-Agents framework. |
|||
|
|||
You must have a Tensorflow graph `your_name_graph.bytes` made using Tensorflow's `freeze_graph.py`. The process to create such graph is explained above. |
|||
|
|||
## Inside of Unity |
|||
|
|||
Put the file `your_name_graph.bytes` into Resources. |
|||
|
|||
In your C# script : |
|||
At the top, add the line |
|||
```csharp |
|||
using Tensorflow; |
|||
``` |
|||
If you will be building for android, you must add this block at the start of your code : |
|||
```csharp |
|||
#if UNITY_ANDROID |
|||
TensorFlowSharp.Android.NativeBinding.Init(); |
|||
#endif |
|||
``` |
|||
Put your graph as a text asset in the variable `graphModel`. You can do so in the inspector by making `graphModel` a public variable and dragging you asset in the inspector or load it from the Resources folder : |
|||
```csharp |
|||
TextAsset graphModel = Resources.Load (your_name_graph) as TextAsset; |
|||
``` |
|||
You then must recreate the graph in Unity by adding the code : |
|||
```csharp |
|||
graph = new TFGraph (); |
|||
graph.Import (graphModel.bytes); |
|||
session = new TFSession (graph); |
|||
``` |
|||
Your newly created graph need to get input tensors. Here is an example with a one dimensional tensor of size 2: |
|||
|
|||
```csharp |
|||
var runner = session.GetRunner (); |
|||
runner.AddInput (graph ["input_placeholder_name"] [0], new float[]{ placeholder_value1, placeholder_value2 }); |
|||
``` |
|||
You need to give all required inputs to the graph. There is one input per TensorFlow placeholder. |
|||
|
|||
To retrieve the output of your graph run the following code. Note that this is for an output that would be a two dimensional tensor of floats. Cast to a long array if your outputs are integers. |
|||
```csharp |
|||
runner.Fetch (graph["output_placeholder_name"][0]); |
|||
float[,] recurrent_tensor = runner.Run () [0].GetValue () as float[,]; |
|||
``` |
|||
撰写
预览
正在加载...
取消
保存
Reference in new issue