information on the gym interface, see [here ](https://github.com/openai/gym ).
We provide a gym wrapper and instructions for using it with existing machine
learning algorithms which utilize gym. Our wrapper provides interfaces on top
of our `UnityEnvironment` class, which is the default way of interfacing with a
learning algorithms which utilize gym. Our wrapper provides interfaces on top of
our `UnityEnvironment` class, which is the default way of interfacing with a
Unity environment via Python.
## Installation
```sh
pip install gym_unity
pip3 install gym_unity
pip install -e .
pip3 install -e .
```
## Using the Gym Wrapper
env = UnityToGymWrapper(unity_environment, worker_id, use_visual, uint8_visual)
```
* `unity_environment` refers to the Unity environment to be wrapped.
- `unity_environment` refers to the Unity environment to be wrapped.
* `use_visual` refers to whether to use visual observations (True) or vector
observations (False) as the default observation provided by the `reset` and
`step` functions. Defaults to `False` .
- `use_visual` refers to whether to use visual observations (True) or vector
observations (False) as the default observation provided by the `reset` and
`step` functions. Defaults to `False` .
* `uint8_visual` refers to whether to output visual observations as `uint8` values
(0-255). Many common Gym environments (e.g. Atari) do this. By default they
will be floats (0.0-1.0). Defaults to `False` .
- `uint8_visual` refers to whether to output visual observations as `uint8`
values (0-255). Many common Gym environments (e.g. Atari) do this. By default
they will be floats (0.0-1.0). Defaults to `False` .
* `flatten_branched` will flatten a branched discrete action space into a Gym Discrete.
Otherwise, it will be converted into a MultiDiscrete. Defaults to `False` .
- `flatten_branched` will flatten a branched discrete action space into a Gym
Discrete. Otherwise, it will be converted into a MultiDiscrete. Defaults to
`False` .
* `allow_multiple_visual_obs` will return a list of observation instead of only
one if disabled. Defaults to `False` .
- `allow_multiple_visual_obs` will return a list of observation instead of only
one if disabled. Defaults to `False` .
* It is only possible to use an environment with a **single** Agent.
* By default, the first visual observation is provided as the `observation` , if
present. Otherwise, vector observations are provided. You can receive all visual
observations by using the `allow_multiple_visual_obs=True` option in the gym
parameters. If set to `True` , you will receive a list of `observation` instead
of only the first one.
* The `TerminalSteps` or `DecisionSteps` output from the environment can still be
accessed from the `info` provided by `env.step(action)` .
* Stacked vector observations are not supported.
* Environment registration for use with `gym.make()` is currently not supported.
- It is only possible to use an environment with a **single** Agent.
- By default, the first visual observation is provided as the `observation` , if
present. Otherwise, vector observations are provided. You can receive all
visual observations by using the `allow_multiple_visual_obs=True` option in
the gym parameters. If set to `True` , you will receive a list of `observation`
instead of only the first one.
- The `TerminalSteps` or `DecisionSteps` output from the environment can still
be accessed from the `info` provided by `env.step(action)` .
- Stacked vector observations are not supported.
- Environment registration for use with `gym.make()` is currently not supported.
## Running OpenAI Baselines Algorithms
### Other Algorithms
Other algorithms in the Baselines repository can be run using scripts similar to
the examples from the baselines package. In most cases, the primary changes needed
to use a Unity environment are to import `UnityToGymWrapper` , and to replace the
environment creation code, typically `gym.make()` , with a call to
the examples from the baselines package. In most cases, the primary changes
needed to use a Unity environment are to import `UnityToGymWrapper` , and to
replace the environment creation code, typically `gym.make()` , with a call to
`UnityToGymWrapper(unity_environment)` passing the environment as input.
A typical rule of thumb is that for vision-based environments, modification
Some algorithms will make use of `make_env()` or `make_mujoco_env()`
functions. You can define a similar function for Unity environments. An example of
such a method using the PPO2 baseline:
Some algorithms will make use of `make_env()` or `make_mujoco_env()` functions.
You can define a similar function for Unity environments. An example of such a
method using the PPO2 baseline:
```python
from mlagents_envs.environment import UnityEnvironment
## Run Google Dopamine Algorithms
Google provides a framework [Dopamine ](https://github.com/google/dopamine ), and
implementations of algorithms, e.g. DQN, Rainbow, and the C51 variant of Rainbow.
Using the Gym wrapper, we can run Unity environments using Dopamine.
implementations of algorithms, e.g. DQN, Rainbow, and the C51 variant of
Rainbow. Using the Gym wrapper, we can run Unity environments using Dopamine.
First, after installing the Gym wrapper, clone the Dopamine repository.
Then, follow the appropriate install instructions as specified on
[Dopamine's homepage ](https://github.com/google/dopamine ). Note that the Dopamine
guide specifies using a virtualenv. If you choose to do so, make sure your unity_env
package is also installed within the same virtualenv as Dopamine.
[Dopamine's homepage ](https://github.com/google/dopamine ). Note that the
Dopamine guide specifies using a virtualenv. If you choose to do so, make sure
your unity_env package is also installed within the same virtualenv as Dopamine.
First, open `dopamine/atari/run_experiment.py` . Alternatively, copy the entire `atari`
folder, and name it something else (e.g. `unity` ). If you choose the copy approach,
be sure to change the package names in the import statements in `train.py` to your new
directory.
First, open `dopamine/atari/run_experiment.py` . Alternatively, copy the entire
`atari` `unity` ). If you choose the
copy approach, be sure to change the package names in the import statements in
`train.py` to your new
Within `run_experiment.py` , we will need to make changes to which environment is
instantiated, just as in the Baselines example. At the top of the file, insert
from gym_unity.envs import UnityToGymWrapper
```
to import the Gym Wrapper. Navigate to the `create_atari_environment` method
in the same file, and switch to instantiating a Unity environment by replacing
the method with the following code.
to import the Gym Wrapper. Navigate to the `create_atari_environment` method in
the same file, and switch to instantiating a Unity environment by replacing the
method with the following code.
```python
game_version = 'v0' if sticky_actions else 'v4'
return env
```
`./envs/GridWorld` is the path to your built Unity executable. For more information on
building Unity environments, see [here ](../docs/Learning-Environment-Executable.md ), and note
the Limitations section below.
`./envs/GridWorld` is the path to your built Unity executable. For more
information on building Unity environments, see
[here ](../docs/Learning-Environment-Executable.md ), and note the Limitations
section below.
Note that we are not using the preprocessor from Dopamine,
as it uses many Atari-specific calls. Furthermore, frame-skipping can be done from within Unity,
Note that we are not using the preprocessor from Dopamine, as it uses many
Atari-specific calls. Furthermore, frame-skipping can be done from within Unity,
Since Dopamine is designed around variants of DQN, it is only compatible
with discrete action spaces, and specifically the Discrete Gym space. For environments
that use branched discrete action spaces (e.g.
Since Dopamine is designed around variants of DQN, it is only compatible with
discrete action spaces, and specifically the Discrete Gym space. For
environments that use branched discrete action spaces (e.g.
`flatten_branched` parameter in `UnityToGymWrapper` , which treats each combination of branched
actions as separate actions.
`flatten_branched` parameter in `UnityToGymWrapper` , which treats each
combination of branched actions as separate actions.
Furthermore, when building your environments, ensure that your Agent is using visual
observations with greyscale enabled, and that the dimensions of the visual observations
is 84 by 84 (matches the parameter found in `dqn_agent.py` and `rainbow_agent.py` ).
Dopamine's agents currently do not automatically adapt to the observation
dimensions or number of channels.
Furthermore, when building your environments, ensure that your Agent is using
visual observations with greyscale enabled, and that the dimensions of the
visual observations is 84 by 84 (matches the parameter found in `dqn_agent.py`
and `rainbow_agent.py` ). Dopamine's agents currently do not automatically adapt
to the observation dimensions or number of channels.
The hyperparameters provided by Dopamine are tailored to the Atari games, and you will
likely need to adjust them for ML-Agents environments. Here is a sample
The hyperparameters provided by Dopamine are tailored to the Atari games, and
you will likely need to adjust them for ML-Agents environments. Here is a sample
`dopamine/agents/rainbow/configs/rainbow.gin` file that is known to work with
GridWorld.
This example assumed you copied `atari` to a separate folder named `unity` .
Replace `unity` in `import dopamine.unity.run_experiment` with the folder you
copied your `run_experiment.py` and `trainer.py` files to.
If you directly modified the existing files, then use `atari` here.
copied your `run_experiment.py` and `trainer.py` files to. If you directly
modified the existing files, then use `atari` here.
### Starting a Run
--gin_files='dopamine/agents/rainbow/configs/rainbow.gin'
```
Again, we assume that you've copied `atari` into a separate folder.
Remember to replace `unity` with the directory you copied your files into. If you
edited the Atari files directly, this should be `atari` .
Again, we assume that you've copied `atari` into a separate folder. Remember to
replace `unity` with the directory you copied your files into. If you edited the
Atari files directly, this should be `atari` .
Dopamine as run on the GridWorld example environment. All Dopamine (DQN, Rainbow,
C51) runs were done with the same epsilon, epsilon decay, replay history, training steps,
and buffer settings as specified above. Note that the first 20000 steps are used to pre-fill
the training buffer, and no learning happens.
Dopamine as run on the GridWorld example environment. All Dopamine (DQN,
Rainbow, C51) runs were done with the same epsilon, epsilon decay, replay
history, training steps, and buffer settings as specified above. Note that the
first 20000 steps are used to pre-fill the training buffer, and no learning
happens.
We provide results from our PPO implementation and the DQN from Baselines as reference.
Note that all runs used the same greyscale GridWorld as Dopamine. For PPO, `num_layers`
was set to 2, and all other hyperparameters are the default for GridWorld in `trainer_config.yaml` .
For Baselines DQN, the provided hyperparameters in the previous section are used. Note
that Baselines implements certain features (e.g. dueling-Q) that are not enabled
in Dopamine DQN.
We provide results from our PPO implementation and the DQN from Baselines as
reference. Note that all runs used the same greyscale GridWorld as Dopamine. For
PPO, `num_layers` was set to 2, and all other hyperparameters are the default
for GridWorld in `trainer_config.yaml` . For Baselines DQN, the provided
hyperparameters in the previous section are used. Note that Baselines implements
certain features (e.g. dueling-Q) that are not enabled in Dopamine DQN.
algorithm to train on the VisualBanana environment, and provide the results below.
The same hyperparameters were used as in the GridWorld case, except that
algorithm to train on the VisualBanana environment, and provide the results
below. The same hyperparameters were used as in the GridWorld case, except that
`replay_history` and `epsilon_decay` were increased to 100000.
GitHub
5 年前