45 KiB
Table of Contents
- mlagents_envs.base_env
- mlagents_envs.environment
- mlagents_envs.registry
- mlagents_envs.registry.unity_env_registry
- mlagents_envs.side_channel
- mlagents_envs.side_channel.raw_bytes_channel
- mlagents_envs.side_channel.outgoing_message
- mlagents_envs.side_channel.engine_configuration_channel
- mlagents_envs.side_channel.side_channel_manager
- mlagents_envs.side_channel.stats_side_channel
- mlagents_envs.side_channel.incoming_message
- mlagents_envs.side_channel.float_properties_channel
- mlagents_envs.side_channel.environment_parameters_channel
- mlagents_envs.side_channel.side_channel
mlagents_envs.base_env
Python Environment API for the ML-Agents Toolkit The aim of this API is to expose Agents evolving in a simulation to perform reinforcement learning on. This API supports multi-agent scenarios and groups similar Agents (same observations, actions spaces and behavior) together. These groups of Agents are identified by their BehaviorName. For performance reasons, the data of each group of agents is processed in a batched manner. Agents are identified by a unique AgentId identifier that allows tracking of Agents across simulation steps. Note that there is no guarantee that the number or order of the Agents in the state will be consistent across simulation steps. A simulation steps corresponds to moving the simulation forward until at least one agent in the simulation sends its observations to Python again. Since Agents can request decisions at different frequencies, a simulation step does not necessarily correspond to a fixed simulation time increment.
DecisionStep Objects
class DecisionStep(NamedTuple)
Contains the data a single Agent collected since the last simulation step.
- obs is a list of numpy arrays observations collected by the agent.
- reward is a float. Corresponds to the rewards collected by the agent since the last simulation step.
- agent_id is an int and an unique identifier for the corresponding Agent.
- action_mask is an optional list of one dimensional array of booleans. Only available when using multi-discrete actions. Each array corresponds to an action branch. Each array contains a mask for each action of the branch. If true, the action is not available for the agent during this simulation step.
DecisionSteps Objects
class DecisionSteps(Mapping)
Contains the data a batch of similar Agents collected since the last simulation step. Note that all Agents do not necessarily have new information to send at each simulation step. Therefore, the ordering of agents and the batch size of the DecisionSteps are not fixed across simulation steps.
- obs is a list of numpy arrays observations collected by the batch of agent. Each obs has one extra dimension compared to DecisionStep: the first dimension of the array corresponds to the batch size of the batch.
- reward is a float vector of length batch size. Corresponds to the rewards collected by each agent since the last simulation step.
- agent_id is an int vector of length batch size containing unique identifier for the corresponding Agent. This is used to track Agents across simulation steps.
- action_mask is an optional list of two dimensional array of booleans. Only available when using multi-discrete actions. Each array corresponds to an action branch. The first dimension of each array is the batch size and the second contains a mask for each action of the branch. If true, the action is not available for the agent during this simulation step.
agent_id_to_index
| @property
| agent_id_to_index() -> Dict[AgentId, int]
Returns:
A Dict that maps agent_id to the index of those agents in this DecisionSteps.
__getitem__
| __getitem__(agent_id: AgentId) -> DecisionStep
returns the DecisionStep for a specific agent.
Arguments:
agent_id: The id of the agent
Returns:
The DecisionStep
empty
| @staticmethod
| empty(spec: "BehaviorSpec") -> "DecisionSteps"
Returns an empty DecisionSteps.
Arguments:
spec: The BehaviorSpec for the DecisionSteps
TerminalStep Objects
class TerminalStep(NamedTuple)
Contains the data a single Agent collected when its episode ended.
- obs is a list of numpy arrays observations collected by the agent.
- reward is a float. Corresponds to the rewards collected by the agent since the last simulation step.
- interrupted is a bool. Is true if the Agent was interrupted since the last decision step. For example, if the Agent reached the maximum number of steps for the episode.
- agent_id is an int and an unique identifier for the corresponding Agent.
TerminalSteps Objects
class TerminalSteps(Mapping)
Contains the data a batch of Agents collected when their episode terminated. All Agents present in the TerminalSteps have ended their episode.
- obs is a list of numpy arrays observations collected by the batch of agent. Each obs has one extra dimension compared to DecisionStep: the first dimension of the array corresponds to the batch size of the batch.
- reward is a float vector of length batch size. Corresponds to the rewards collected by each agent since the last simulation step.
- interrupted is an array of booleans of length batch size. Is true if the associated Agent was interrupted since the last decision step. For example, if the Agent reached the maximum number of steps for the episode.
- agent_id is an int vector of length batch size containing unique identifier for the corresponding Agent. This is used to track Agents across simulation steps.
agent_id_to_index
| @property
| agent_id_to_index() -> Dict[AgentId, int]
Returns:
A Dict that maps agent_id to the index of those agents in this TerminalSteps.
__getitem__
| __getitem__(agent_id: AgentId) -> TerminalStep
returns the TerminalStep for a specific agent.
Arguments:
agent_id: The id of the agent
Returns:
obs, reward, done, agent_id and optional action mask for a specific agent
empty
| @staticmethod
| empty(spec: "BehaviorSpec") -> "TerminalSteps"
Returns an empty TerminalSteps.
Arguments:
spec: The BehaviorSpec for the TerminalSteps
ActionTuple Objects
class ActionTuple(_ActionTupleBase)
An object whose fields correspond to actions of different types. Continuous and discrete actions are numpy arrays of type float32 and int32, respectively and are type checked on construction. Dimensions are of (n_agents, continuous_size) and (n_agents, discrete_size), respectively. Note, this also holds when continuous or discrete size is zero.
discrete_dtype
| @property
| discrete_dtype() -> np.dtype
The dtype of a discrete action.
ActionSpec Objects
class ActionSpec(NamedTuple)
A NamedTuple containing utility functions and information about the action spaces for a group of Agents under the same behavior.
- num_continuous_actions is an int corresponding to the number of floats which constitute the action.
- discrete_branch_sizes is a Tuple of int where each int corresponds to the number of discrete actions available to the agent on an independent action branch.
is_discrete
| is_discrete() -> bool
Returns true if this Behavior uses discrete actions
is_continuous
| is_continuous() -> bool
Returns true if this Behavior uses continuous actions
discrete_size
| @property
| discrete_size() -> int
Returns a an int corresponding to the number of discrete branches.
empty_action
| empty_action(n_agents: int) -> ActionTuple
Generates ActionTuple corresponding to an empty action (all zeros) for a number of agents.
Arguments:
n_agents: The number of agents that will have actions generated
random_action
| random_action(n_agents: int) -> ActionTuple
Generates ActionTuple corresponding to a random action (either discrete or continuous) for a number of agents.
Arguments:
n_agents: The number of agents that will have actions generated
create_continuous
| @staticmethod
| create_continuous(continuous_size: int) -> "ActionSpec"
Creates an ActionSpec that is homogenously continuous
create_discrete
| @staticmethod
| create_discrete(discrete_branches: Tuple[int]) -> "ActionSpec"
Creates an ActionSpec that is homogenously discrete
DimensionProperty Objects
class DimensionProperty(IntFlag)
The dimension property of a dimension of an observation.
UNSPECIFIED
No properties specified.
NONE
No Property of the observation in that dimension. Observation can be processed with Fully connected networks.
TRANSLATIONAL_EQUIVARIANCE
Means it is suitable to do a convolution in this dimension.
VARIABLE_SIZE
Means that there can be a variable number of observations in this dimension. The observations are unordered.
ObservationType Objects
class ObservationType(Enum)
An Enum which defines the type of information carried in the observation of the agent.
DEFAULT
Observation information is generic.
GOAL_SIGNAL
Observation contains goal information for current task.
ObservationSpec Objects
class ObservationSpec(NamedTuple)
A NamedTuple containing information about the observation of Agents.
- shape is a Tuple of int : It corresponds to the shape of an observation's dimensions.
- dimension_property is a Tuple of DimensionProperties flag, one flag for each dimension.
- observation_type is an enum of ObservationType.
BehaviorSpec Objects
class BehaviorSpec(NamedTuple)
A NamedTuple containing information about the observation and action spaces for a group of Agents under the same behavior.
- observation_specs is a List of ObservationSpec NamedTuple containing information about the information of the Agent's observations such as their shapes. The order of the ObservationSpec is the same as the order of the observations of an agent.
- action_spec is an ActionSpec NamedTuple.
BaseEnv Objects
class BaseEnv(ABC)
step
| @abstractmethod
| step() -> None
Signals the environment that it must move the simulation forward by one step.
reset
| @abstractmethod
| reset() -> None
Signals the environment that it must reset the simulation.
close
| @abstractmethod
| close() -> None
Signals the environment that it must close.
behavior_specs
| @property
| @abstractmethod
| behavior_specs() -> MappingType[str, BehaviorSpec]
Returns a Mapping from behavior names to behavior specs. Agents grouped under the same behavior name have the same action and observation specs, and are expected to behave similarly in the environment. Note that new keys can be added to this mapping as new policies are instantiated.
set_actions
| @abstractmethod
| set_actions(behavior_name: BehaviorName, action: ActionTuple) -> None
Sets the action for all of the agents in the simulation for the next step. The Actions must be in the same order as the order received in the DecisionSteps.
Arguments:
behavior_name: The name of the behavior the agents are part ofaction: ActionTuple tuple of continuous and/or discrete action. Actions are np.arrays with dimensions (n_agents, continuous_size) and (n_agents, discrete_size), respectively.
set_action_for_agent
| @abstractmethod
| set_action_for_agent(behavior_name: BehaviorName, agent_id: AgentId, action: ActionTuple) -> None
Sets the action for one of the agents in the simulation for the next step.
Arguments:
behavior_name: The name of the behavior the agent is part ofagent_id: The id of the agent the action is set foraction: ActionTuple tuple of continuous and/or discrete action Actions are np.arrays with dimensions (1, continuous_size) and (1, discrete_size), respectively. Note, this initial dimensions of 1 is because this action is meant for a single agent.
get_steps
| @abstractmethod
| get_steps(behavior_name: BehaviorName) -> Tuple[DecisionSteps, TerminalSteps]
Retrieves the steps of the agents that requested a step in the simulation.
Arguments:
behavior_name: The name of the behavior the agents are part of
Returns:
A tuple containing :
- A DecisionSteps NamedTuple containing the observations, the rewards, the agent ids and the action masks for the Agents of the specified behavior. These Agents need an action this step.
- A TerminalSteps NamedTuple containing the observations, rewards, agent ids and interrupted flags of the agents that had their episode terminated last step.
mlagents_envs.environment
UnityEnvironment Objects
class UnityEnvironment(BaseEnv)
__init__
| __init__(file_name: Optional[str] = None, worker_id: int = 0, base_port: Optional[int] = None, seed: int = 0, no_graphics: bool = False, timeout_wait: int = 60, additional_args: Optional[List[str]] = None, side_channels: Optional[List[SideChannel]] = None, log_folder: Optional[str] = None)
Starts a new unity environment and establishes a connection with the environment. Notice: Currently communication between Unity and Python takes place over an open socket without authentication. Ensure that the network where training takes place is secure.
:string file_name: Name of Unity environment binary. :int base_port: Baseline port number to connect to Unity environment over. worker_id increments over this. If no environment is specified (i.e. file_name is None), the DEFAULT_EDITOR_PORT will be used. :int worker_id: Offset from base_port. Used for training multiple environments simultaneously. :bool no_graphics: Whether to run the Unity simulator in no-graphics mode :int timeout_wait: Time (in seconds) to wait for connection from environment. :list args: Addition Unity command line arguments :list side_channels: Additional side channel for no-rl communication with Unity :str log_folder: Optional folder to write the Unity Player log file into. Requires absolute path.
close
| close()
Sends a shutdown signal to the unity environment, and closes the socket connection.
mlagents_envs.registry
mlagents_envs.registry.unity_env_registry
UnityEnvRegistry Objects
class UnityEnvRegistry(Mapping)
UnityEnvRegistry
Provides a library of Unity environments that can be launched without the need of downloading the Unity Editor. The UnityEnvRegistry implements a Map, to access an entry of the Registry, use:
registry = UnityEnvRegistry()
entry = registry[<environment_identifyier>]
An entry has the following properties :
identifier: Uniquely identifies this environmentexpected_reward: Corresponds to the reward an agent must obtained for the task to be considered completed.description: A human readable description of the environment.
To launch a Unity environment from a registry entry, use the make method:
registry = UnityEnvRegistry()
env = registry[<environment_identifyier>].make()
register
| register(new_entry: BaseRegistryEntry) -> None
Registers a new BaseRegistryEntry to the registry. The BaseRegistryEntry.identifier value will be used as indexing key. If two are more environments are registered under the same key, the most recentry added will replace the others.
register_from_yaml
| register_from_yaml(path_to_yaml: str) -> None
Registers the environments listed in a yaml file (either local or remote). Note that the entries are registered lazily: the registration will only happen when an environment is accessed. The yaml file must have the following format :
environments:
- <identifier of the first environment>:
expected_reward: <expected reward of the environment>
description: | <a multi line description of the environment>
<continued multi line description>
linux_url: <The url for the Linux executable zip file>
darwin_url: <The url for the OSX executable zip file>
win_url: <The url for the Windows executable zip file>
- <identifier of the second environment>:
expected_reward: <expected reward of the environment>
description: | <a multi line description of the environment>
<continued multi line description>
linux_url: <The url for the Linux executable zip file>
darwin_url: <The url for the OSX executable zip file>
win_url: <The url for the Windows executable zip file>
- ...
Arguments:
path_to_yaml: A local path or url to the yaml file
clear
| clear() -> None
Deletes all entries in the registry.
__getitem__
| __getitem__(identifier: str) -> BaseRegistryEntry
Returns the BaseRegistryEntry with the provided identifier. BaseRegistryEntry can then be used to make a Unity Environment.
Arguments:
identifier: The identifier of the BaseRegistryEntry
Returns:
The associated BaseRegistryEntry
mlagents_envs.side_channel
mlagents_envs.side_channel.raw_bytes_channel
RawBytesChannel Objects
class RawBytesChannel(SideChannel)
This is an example of what the SideChannel for raw bytes exchange would look like. Is meant to be used for general research purpose.
on_message_received
| on_message_received(msg: IncomingMessage) -> None
Is called by the environment to the side channel. Can be called multiple times per step if multiple messages are meant for that SideChannel.
get_and_clear_received_messages
| get_and_clear_received_messages() -> List[bytes]
returns a list of bytearray received from the environment.
send_raw_data
| send_raw_data(data: bytearray) -> None
Queues a message to be sent by the environment at the next call to step.
mlagents_envs.side_channel.outgoing_message
OutgoingMessage Objects
class OutgoingMessage()
Utility class for forming the message that is written to a SideChannel. All data is written in little-endian format using the struct module.
__init__
| __init__()
Create an OutgoingMessage with an empty buffer.
write_bool
| write_bool(b: bool) -> None
Append a boolean value.
write_int32
| write_int32(i: int) -> None
Append an integer value.
write_float32
| write_float32(f: float) -> None
Append a float value. It will be truncated to 32-bit precision.
write_float32_list
| write_float32_list(float_list: List[float]) -> None
Append a list of float values. They will be truncated to 32-bit precision.
write_string
| write_string(s: str) -> None
Append a string value. Internally, it will be encoded to ascii, and the encoded length will also be written to the message.
set_raw_bytes
| set_raw_bytes(buffer: bytearray) -> None
Set the internal buffer to a new bytearray. This will overwrite any existing data.
Arguments:
buffer:
Returns:
mlagents_envs.side_channel.engine_configuration_channel
EngineConfigurationChannel Objects
class EngineConfigurationChannel(SideChannel)
This is the SideChannel for engine configuration exchange. The data in the engine configuration is as follows :
- int width;
- int height;
- int qualityLevel;
- float timeScale;
- int targetFrameRate;
- int captureFrameRate;
on_message_received
| on_message_received(msg: IncomingMessage) -> None
Is called by the environment to the side channel. Can be called multiple times per step if multiple messages are meant for that SideChannel. Note that Python should never receive an engine configuration from Unity
set_configuration_parameters
| set_configuration_parameters(width: Optional[int] = None, height: Optional[int] = None, quality_level: Optional[int] = None, time_scale: Optional[float] = None, target_frame_rate: Optional[int] = None, capture_frame_rate: Optional[int] = None) -> None
Sets the engine configuration. Takes as input the configurations of the engine.
Arguments:
width: Defines the width of the display. (Must be set alongside height)height: Defines the height of the display. (Must be set alongside width)quality_level: Defines the quality level of the simulation.time_scale: Defines the multiplier for the deltatime in the simulation. If set to a higher value, time will pass faster in the simulation but the physics might break.target_frame_rate: Instructs simulation to try to render at a specified frame rate.capture_frame_rate: Instructs the simulation to consider time between updates to always be constant, regardless of the actual frame rate.
set_configuration
| set_configuration(config: EngineConfig) -> None
Sets the engine configuration. Takes as input an EngineConfig.
mlagents_envs.side_channel.side_channel_manager
SideChannelManager Objects
class SideChannelManager()
process_side_channel_message
| process_side_channel_message(data: bytes) -> None
Separates the data received from Python into individual messages for each registered side channel and calls on_message_received on them.
Arguments:
data: The packed message sent by Unity
generate_side_channel_messages
| generate_side_channel_messages() -> bytearray
Gathers the messages that the registered side channels will send to Unity and combines them into a single message ready to be sent.
mlagents_envs.side_channel.stats_side_channel
StatsSideChannel Objects
class StatsSideChannel(SideChannel)
Side channel that receives (string, float) pairs from the environment, so that they can eventually be passed to a StatsReporter.
on_message_received
| on_message_received(msg: IncomingMessage) -> None
Receive the message from the environment, and save it for later retrieval.
Arguments:
msg:
Returns:
get_and_reset_stats
| get_and_reset_stats() -> EnvironmentStats
Returns the current stats, and resets the internal storage of the stats.
Returns:
mlagents_envs.side_channel.incoming_message
IncomingMessage Objects
class IncomingMessage()
Utility class for reading the message written to a SideChannel. Values must be read in the order they were written.
__init__
| __init__(buffer: bytes, offset: int = 0)
Create a new IncomingMessage from the bytes.
read_bool
| read_bool(default_value: bool = False) -> bool
Read a boolean value from the message buffer.
Arguments:
default_value: Default value to use if the end of the message is reached.
Returns:
The value read from the message, or the default value if the end was reached.
read_int32
| read_int32(default_value: int = 0) -> int
Read an integer value from the message buffer.
Arguments:
default_value: Default value to use if the end of the message is reached.
Returns:
The value read from the message, or the default value if the end was reached.
read_float32
| read_float32(default_value: float = 0.0) -> float
Read a float value from the message buffer.
Arguments:
default_value: Default value to use if the end of the message is reached.
Returns:
The value read from the message, or the default value if the end was reached.
read_float32_list
| read_float32_list(default_value: List[float] = None) -> List[float]
Read a list of float values from the message buffer.
Arguments:
default_value: Default value to use if the end of the message is reached.
Returns:
The value read from the message, or the default value if the end was reached.
read_string
| read_string(default_value: str = "") -> str
Read a string value from the message buffer.
Arguments:
default_value: Default value to use if the end of the message is reached.
Returns:
The value read from the message, or the default value if the end was reached.
get_raw_bytes
| get_raw_bytes() -> bytes
Get a copy of the internal bytes used by the message.
mlagents_envs.side_channel.float_properties_channel
FloatPropertiesChannel Objects
class FloatPropertiesChannel(SideChannel)
This is the SideChannel for float properties shared with Unity. You can modify the float properties of an environment with the commands set_property, get_property and list_properties.
on_message_received
| on_message_received(msg: IncomingMessage) -> None
Is called by the environment to the side channel. Can be called multiple times per step if multiple messages are meant for that SideChannel.
set_property
| set_property(key: str, value: float) -> None
Sets a property in the Unity Environment.
Arguments:
key: The string identifier of the property.value: The float value of the property.
get_property
| get_property(key: str) -> Optional[float]
Gets a property in the Unity Environment. If the property was not found, will return None.
Arguments:
key: The string identifier of the property.
Returns:
The float value of the property or None.
list_properties
| list_properties() -> List[str]
Returns a list of all the string identifiers of the properties currently present in the Unity Environment.
get_property_dict_copy
| get_property_dict_copy() -> Dict[str, float]
Returns a copy of the float properties.
Returns:
mlagents_envs.side_channel.environment_parameters_channel
EnvironmentParametersChannel Objects
class EnvironmentParametersChannel(SideChannel)
This is the SideChannel for sending environment parameters to Unity. You can send parameters to an environment with the command set_float_parameter.
set_float_parameter
| set_float_parameter(key: str, value: float) -> None
Sets a float environment parameter in the Unity Environment.
Arguments:
key: The string identifier of the parameter.value: The float value of the parameter.
set_uniform_sampler_parameters
| set_uniform_sampler_parameters(key: str, min_value: float, max_value: float, seed: int) -> None
Sets a uniform environment parameter sampler.
Arguments:
key: The string identifier of the parameter.min_value: The minimum of the sampling distribution.max_value: The maximum of the sampling distribution.seed: The random seed to initialize the sampler.
set_gaussian_sampler_parameters
| set_gaussian_sampler_parameters(key: str, mean: float, st_dev: float, seed: int) -> None
Sets a gaussian environment parameter sampler.
Arguments:
key: The string identifier of the parameter.mean: The mean of the sampling distribution.st_dev: The standard deviation of the sampling distribution.seed: The random seed to initialize the sampler.
set_multirangeuniform_sampler_parameters
| set_multirangeuniform_sampler_parameters(key: str, intervals: List[Tuple[float, float]], seed: int) -> None
Sets a multirangeuniform environment parameter sampler.
Arguments:
key: The string identifier of the parameter.intervals: The lists of min and max that define each uniform distribution.seed: The random seed to initialize the sampler.
mlagents_envs.side_channel.side_channel
SideChannel Objects
class SideChannel(ABC)
The side channel just get access to a bytes buffer that will be shared between C# and Python. For example, We will create a specific side channel for properties that will be a list of string (fixed size) to float number, that can be modified by both C# and Python. All side channels are passed to the Env object at construction.
queue_message_to_send
| queue_message_to_send(msg: OutgoingMessage) -> None
Queues a message to be sent by the environment at the next call to step.
on_message_received
| @abstractmethod
| on_message_received(msg: IncomingMessage) -> None
Is called by the environment to the side channel. Can be called multiple times per step if multiple messages are meant for that SideChannel.
channel_id
| @property
| channel_id() -> uuid.UUID
Returns:
The type of side channel used. Will influence how the data is processed in the environment.