浏览代码

[containerization] CPU based containerization to support all environments that don't use observations

/develop-generalizationTraining-TrainerController
eshvk 7 年前
当前提交
23981dbf
共有 10 个文件被更改,包括 149 次插入32 次删除
  1. 5
      .gitignore
  2. 4
      docs/installation.md
  3. 18
      python/learn.py
  4. 2
      python/trainer_config.yaml
  5. 3
      python/unityagents/environment.py
  6. 90
      python/unitytrainers/trainer_controller.py
  7. 12
      Dockerfile
  8. 44
      docs/Using-Docker.md
  9. 3
      unity-volume/.gitignore

5
.gitignore


/unity-environment/Assets/AssetStoreTools*
# Tensorflow Model Info
/python/models
/python/summaries
/models
/summaries
# Environemnt logfile
*unity-environment.log

*.egg-info*
*.eggs*
*.gitignore.swp
.DS_Store

4
docs/installation.md


* docopt (Training)
* TensorFlow (1.0+) (Training)
## Docker-based Installation (experimental)
If you’d like to use Docker for ML Agents, please follow [this guide](Using-Docker.md).
### Installing Dependencies
To install dependencies, go into the `python` sub-directory of the repositroy, and run (depending on your python version) from the command line:

18
python/learn.py


import logging
import os
from docopt import docopt
from unitytrainers.trainer_controller import TrainerController

--slow Whether to run the game at training speed [default: False].
--train Whether to train model, or only run inference [default: False].
--worker-id=<n> Number to add to communication port (5005). Used for multi-environment [default: 0].
--docker-target-name=<dt> Docker Volume to store curriculum, executable and model files [default: Empty].
# Docker Parameters
if options['--docker-target-name'] == 'Empty':
docker_target_name = ''
else:
docker_target_name = options['--docker-target-name']
# General parameters
run_id = options['--run-id']

save_freq = int(options['--save-freq'])
env_name = options['<env>']
env_path = options['<env>']
keep_checkpoints = int(options['--keep-checkpoints'])
worker_id = int(options['--worker-id'])
curriculum_file = str(options['--curriculum'])

fast_simulation = not bool(options['--slow'])
tc = TrainerController(env_name, run_id, save_freq, curriculum_file, fast_simulation, load_model, train_model,
worker_id, keep_checkpoints, lesson, seed)
# Constants
# Assumption that this yaml is present in same dir as this file
base_path = os.path.dirname(__file__)
TRAINER_CONFIG_PATH = os.path.abspath(os.path.join(base_path, "trainer_config.yaml"))
tc = TrainerController(env_path, run_id, save_freq, curriculum_file, fast_simulation, load_model, train_model,
worker_id, keep_checkpoints, lesson, seed, docker_target_name, TRAINER_CONFIG_PATH)
tc.start_learning()

2
python/trainer_config.yaml


summary_freq: 1000
normalize: true
batch_size: 1000
buffer_size: 10000
buffer_size: 10000
hidden_units: 64
max_steps: 1.0e4

3
python/unityagents/environment.py


:int base_port: Baseline port number to connect to Unity environment over. worker_id increments over this.
:int worker_id: Number to add to communication port (5005) [0]. Used for asynchronous agent scenarios.
"""
atexit.register(self.close)
self.port = base_port + worker_id
self._buffer_size = 12000

file_name = (file_name.strip()
.replace('.app', '').replace('.exe', '').replace('.x86_64', '').replace('.x86', ''))
true_filename = os.path.basename(os.path.normpath(file_name))
logger.info('The true file name is {}'.format(true_filename))
launch_string = None
if platform == "linux" or platform == "linux2":
candidates = glob.glob(os.path.join(cwd, file_name) + '.x86_64')

"Provided filename does not match any environments."
.format(true_filename))
else:
logger.info("This is the launch string {}".format(launch_string))
# Launch Unity environment
proc1 = subprocess.Popen(
[launch_string,

90
python/unitytrainers/trainer_controller.py


class TrainerController(object):
def __init__(self, env_name, run_id, save_freq, curriculum_file, fast_simulation, load, train,
worker_id, keep_checkpoints, lesson, seed):
self.model_path = './models/{}'.format(run_id)
def __init__(self, env_path, run_id, save_freq, curriculum_file, fast_simulation, load, train,
worker_id, keep_checkpoints, lesson, seed, docker_target_name, trainer_config_path):
"""
:param env_path: Location to the environment executable to be loaded.
:param run_id: The sub-directory name for model and summary statistics
:param save_freq: Frequency at which to save model
:param curriculum_file: Curriculum json file for environment
:param fast_simulation: Whether to run the game at training speed
:param load: Whether to load the model or randomly initialize
:param train: Whether to train model, or only run inference
:param worker_id: Number to add to communication port (5005). Used for multi-environment
:param keep_checkpoints: How many model checkpoints to keep
:param lesson: Start learning from this lesson
:param seed: Random seed used for training.
:param docker_target_name: Name of docker volume that will contain all data.
:param trainer_config_path: Fully qualified path to location of trainer configuration file
"""
self.trainer_config_path = trainer_config_path
env_path = (env_path.strip()
.replace('.app', '')
.replace('.exe', '')
.replace('.x86_64', '')
.replace('.x86', '')) # Strip out executable extensions if passed
# Recognize and use docker volume if one is passed as an argument
if docker_target_name == '':
self.model_path = './models/{run_id}'.format(run_id=run_id)
self.curriculum_file = curriculum_file
self.summaries_dir = './summaries'
else:
self.model_path = '/{docker_target_name}/models/{run_id}'.format(
docker_target_name=docker_target_name,
run_id=run_id)
env_path = '/{docker_target_name}/{env_name}'.format(docker_target_name=docker_target_name,
env_name=env_path)
if curriculum_file is None:
self.curriculum_file = None
else:
self.curriculum_file = '/{docker_target_name}/{curriculum_file}'.format(
docker_target_name=docker_target_name,
curriculum_file=curriculum_file)
self.summaries_dir = '/{docker_target_name}/summaries'.format(docker_target_name=docker_target_name)
self.curriculum_file = curriculum_file
self.lesson = lesson
self.fast_simulation = fast_simulation
self.load_model = load

self.seed = seed
np.random.seed(self.seed)
tf.set_random_seed(self.seed)
self.env = UnityEnvironment(file_name=env_name, worker_id=self.worker_id,
self.env = UnityEnvironment(file_name=env_path, worker_id=self.worker_id,
self.env_name = (env_name.strip().replace('.app', '').replace('.exe', '').replace('.x86_64', '')
.replace('.x86', ''))
self.env_name = os.path.basename(os.path.normpath(self.env_name))
self.logger.info(str(self.env))
self.env_name = os.path.basename(os.path.normpath(env_path)) # Extract out name of environment
def _get_progress(self):
if self.curriculum_file is not None:

self.logger.info("\t" + n)
return nodes
def _save_model(self, sess, saver, model_path="./", steps=0):
def _save_model(self, sess, saver, steps=0):
:param model_path: Designated model path.
last_checkpoint = model_path + '/model-' + str(steps) + '.cptk'
last_checkpoint = self.model_path + '/model-' + str(steps) + '.cptk'
tf.train.write_graph(sess.graph_def, model_path, 'raw_graph_def.pb', as_text=False)
tf.train.write_graph(sess.graph_def, self.model_path, 'raw_graph_def.pb', as_text=False)
self.logger.info("Saved Model")
def _export_graph(self):

if len(self.env.external_brain_names) > 1:
graph_scope = re.sub('[^0-9a-zA-Z]+', '-', brain_name)
trainer_parameters['graph_scope'] = graph_scope
trainer_parameters['summary_path'] = './summaries/{}'.format(
str(self.run_id)) + '_' + graph_scope
trainer_parameters['summary_path'] = '{basedir}/{name}'.format(
basedir=self.summaries_dir,
name=str(self.run_id) + '_' + graph_scope)
trainer_parameters['summary_path'] = './summaries/{}'.format(self.run_id)
trainer_parameters['summary_path'] = '{basedir}/{name}'.format(
basedir=self.summaries_dir,
name=str(self.run_id))
if brain_name in trainer_config:
_brain_key = brain_name
while not isinstance(trainer_config[_brain_key], dict):

raise UnityEnvironmentException("The trainer config contains an unknown trainer type for brain {}"
.format(brain_name))
@staticmethod
def _load_config(config_filename):
def _load_config(self):
with open(config_filename) as data_file:
with open(self.trainer_config_path) as data_file:
raise UnityEnvironmentException("The file {} could not be found. Will use default Hyperparameters"
.format("trainer_config.yaml"))
raise UnityEnvironmentException("""Parameter file could not be found here {}.
Will use default Hyper parameters"""
.format(self.trainer_config_path))
raise UnityEnvironmentException("There was an error decoding {}".format("trainer_config.yaml"))
raise UnityEnvironmentException("There was an error decoding Trainer Config from this path : {}"
.format(self.trainer_config_path))
@staticmethod
def _create_model_path(model_path):

def start_learning(self):
self.env.curriculum.set_lesson_number(self.lesson)
trainer_config = self._load_config("trainer_config.yaml")
trainer_config = self._load_config()
self._create_model_path(self.model_path)
tf.reset_default_graph()

global_step += 1
if global_step % self.save_freq == 0 and global_step != 0 and self.train_model:
# Save Tensorflow model
self._save_model(sess, model_path=self.model_path, steps=global_step, saver=saver)
self._save_model(sess, steps=global_step, saver=saver)
self._save_model(sess, model_path=self.model_path, steps=global_step, saver=saver)
self._save_model(sess, steps=global_step, saver=saver)
self._save_model(sess, model_path=self.model_path, steps=global_step, saver=saver)
self._save_model(sess, steps=global_step, saver=saver)
pass
self.env.close()
if self.train_model:

12
Dockerfile


# Use an official Python runtime as a parent image
FROM python:3.6-slim
RUN apt-get update && apt-get -y upgrade
ADD python/requirements.txt .
RUN pip install --trusted-host pypi.python.org -r requirements.txt
WORKDIR /execute
COPY python /execute/python
ENTRYPOINT ["python", "python/learn.py"]

44
docs/Using-Docker.md


# Using Docker For ML Agents (Experimental)
We are currently offering an experimental solution for people who'd like to do training or inference using docker. This setup currently forces both python and Unity to rely on _only_ the CPU for computation purposes. So we don't support environments such as [GridWorld](Example-Environments.md#gridworld) which use visual observations for training.
## Setup
- Install [docker](https://www.docker.com/community-edition#/download) if you don't have it setup on your machine.
- Since Docker runs a container in an environment that is isolated from the host machine, we will be using a mounted directory, e.g. `unity-volume` in your host machine in order to share data, e.g. the Unity executable, curriculum files and tensorflow graph.
## Usage
- Docker typically runs a container sharing a (linux) kernel with the host machine, this means that the
Unity environment **has** to be built for the **linux platform**. Please select the architecture to be `x86_64` and choose the build to be `headless` (_this is important because we are running it in a container that does not have graphics drivers installed_).
Save the generated environment in the directory to be mounted (e.g. we have conveniently created an empty directory called at the top level `unity-volume`). Ensure that
`unity-volume/<environment-name>.x86_64` and `unity-volume/environment-name_Data`. So for example, `<environment_name>` might be `3Dball` and you might want to ensure that `unity-volume/3Dball.x86_64` and `unity-volume/3Dball_Data` are both present in the directory `unity-volume`.
- Make sure the docker engine is running on your machine, then build the docker container by running `docker build -t <image_name> .` . in the top level of the source directory. Replace `<image_name>` by the name of the image that you want to use, e.g. `balance.ball.v0.1`.
- Run the container:
```
docker run --mount type=bind,source="$(pwd)"/unity-volume,target=/unity-volume \
<tag-name>:latest <environment-name> \
--docker-target-name=unity-volume
--train --run-id=<run-id>
```
For our balance ball, example this would be:
- Run the container:
```
docker run --mount type=bind,source="$(pwd)"/unity-volume,target=/unity-volume \
balance.ball.v0.1:latest 3Dball \
--docker-target-name=unity-volume
--train --run-id=<run-id>
```
**Note** The docker target volume name, `unity-volume` must be passed to ML-Agents as an argument using the `--docker-target-name` option. The output will be stored in mounted directory.

3
unity-volume/.gitignore


*
# Ignore everything in this directory except for .gitignore. This directory is for illustrative purposes
!.gitignore
正在加载...
取消
保存