ml-agents/notebooks/getting-started-gym.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Unity ML-Agents Toolkit\n",
    "##  Gym Wrapper Basics\n",
    "This notebook contains a walkthrough of the basic functions of the Python Gym Wrapper for the Unity ML-Agents toolkit. For instructions on building a Unity environment, see [here](https://github.com/Unity-Technologies/ml-agents/blob/master/docs/Getting-Started-with-Balance-Ball.md)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Single-Agent Environments\n",
    "\n",
    "The first five steps show how to use the `UnityEnv` wrapper with single-agent environments. See below step five for how to use with multi-agent environments."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Load dependencies\n",
    "\n",
    "The following loads the necessary dependencies and checks the Python version (at runtime). ML-Agents Toolkit (v0.3 onwards) requires Python 3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import sys\n",
    "\n",
    "from gym_unity.envs import UnityEnv\n",
    "\n",
    "%matplotlib inline\n",
    "\n",
    "print(\"Python version:\")\n",
    "print(sys.version)\n",
    "\n",
    "# check Python version\n",
    "if (sys.version_info[0] < 3):\n",
    "    raise Exception(\"ERROR: ML-Agents Toolkit (v0.3 onwards) requires Python 3\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Start the environment\n",
    "`UnityEnv` launches and begins communication with the environment when instantiated. We will be using the `GridWorld` environment. You will need to create an `envs` directory within the  `/python` subfolder of the repository, and build the GridWorld environment to that directory. For more information on building Unity environments, see [here](../docs/Learning-Environment-Executable.md)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "env_name = \"../envs/GridWorld\"  # Name of the Unity environment binary to launch\n",
    "env = UnityEnv(env_name, worker_id=0, use_visual=True)\n",
    "\n",
    "# Examine environment parameters\n",
    "print(str(env))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Examine the observation and state spaces\n",
    "We can reset the environment to be provided with an initial observation of the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Reset the environment\n",
    "initial_observation = env.reset()\n",
    "\n",
    "if len(env.observation_space.shape) == 1:\n",
    "    # Examine the initial vector observation\n",
    "    print(\"Agent state looks like: \\n{}\".format(initial_observation))\n",
    "else:\n",
    "    # Examine the initial visual observation\n",
    "    print(\"Agent observations look like:\")\n",
    "    if env.observation_space.shape[2] == 3:\n",
    "        plt.imshow(initial_observation[:,:,:])\n",
    "    else:\n",
    "        plt.imshow(initial_observation[:,:,0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Take random actions in the environment\n",
    "Once we restart an environment, we can step the environment forward and provide actions to all of the agents within the environment. Here we simply choose random actions using the `env.action_space.sample()` function.\n",
    "\n",
    "Once this cell is executed, 10 messages will be printed that detail how much reward will be accumulated for the next 10 episodes. The Unity environment will then pause, waiting for further signals telling it what to do next. Thus, not seeing any animation is expected when running this cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for episode in range(10):\n",
    "    initial_observation = env.reset()\n",
    "    done = False\n",
    "    episode_rewards = 0\n",
    "    while not done:\n",
    "        observation, reward, done, info = env.step(env.action_space.sample())\n",
    "        episode_rewards += reward\n",
    "    print(\"Total reward this episode: {}\".format(episode_rewards))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. Close the environment when finished\n",
    "When we are finished using an environment, we can close it with the function below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "env.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Multi-Agent Environments\n",
    "\n",
    "It is also possible to use the gym wrapper with multi-agent environments. For these environments, observations, rewards, and done flags will be provided in a list. Likewise, the environment will expect a list of actions when calling `step(action)`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Start the environment\n",
    "\n",
    "We will use the `3DBall` environment for this walkthrough. For more information on building Unity environments, see [here](../docs/Learning-Environment-Executable.md). We will launch it from the `python/envs` sub-directory of the repo. Please create an `envs` folder if one does not already exist."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Name of the Unity environment binary to launch\n",
    "multi_env_name = \"../envs/3DBall\"  \n",
    "multi_env = UnityEnv(multi_env_name, worker_id=1, \n",
    "                     use_visual=False, multiagent=True)\n",
    "\n",
    "# Examine environment parameters\n",
    "print(str(multi_env))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Examine the observation space "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Reset the environment\n",
    "initial_observations = multi_env.reset()\n",
    "\n",
    "if len(multi_env.observation_space.shape) == 1:\n",
    "    # Examine the initial vector observation\n",
    "    print(\"Agent observations look like: \\n{}\".format(initial_observations[0]))\n",
    "else:\n",
    "    # Examine the initial visual observation\n",
    "    print(\"Agent observations look like:\")\n",
    "    if multi_env.observation_space.shape[2] == 3:\n",
    "        plt.imshow(initial_observations[0][:,:,:])\n",
    "    else:\n",
    "        plt.imshow(initial_observations[0][:,:,0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Take random steps in the environment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for episode in range(10):\n",
    "    initial_observation = multi_env.reset()\n",
    "    done = False\n",
    "    episode_rewards = 0\n",
    "    while not done:\n",
    "        actions = [multi_env.action_space.sample() for agent in range(multi_env.number_agents)]\n",
    "        observations, rewards, dones, info = multi_env.step(actions)\n",
    "        episode_rewards += np.mean(rewards)\n",
    "        done = dones[0]\n",
    "    print(\"Total reward this episode: {}\".format(episode_rewards))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Close the environment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "multi_env.close()"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}