*NOTE: The Perception package's randomization toolset is currently marked as experimental and is subject to change.*
The randomization toolset simplifies randomizing aspects of generating synthetic data. It facilitates exposing parameters for randomization, offers samplers to pick random values from parameters, and provides scenarios to coordinate a full randomization process. Each of these also allows for custom implementations to fit particular randomization needs.
The randomization toolset simplifies randomizing aspects of generating synthetic data. It facilitates exposing parameters for randomization, offers samplers to pick random values from parameters, and provides Scenarios to coordinate a full randomization process. Each of these also allows for custom implementations to fit particular randomization needs.
#### What is Domain Randomization?
#### How can a Unity project be randomized using the Perception Randomization toolset?
Randomizing a project involves the following steps:
1. Create a scenario
2. Define and add randomizers to the scenario
1. Create a Scenario
2. Define and add Randomizers to the Scenario
Beginning with step 1, add a scenario component to your simulation. This scenario will act as the central hub for all randomization activities that occur when your scene is executed.
Beginning with step 1, add a Scenario component to your simulation. This Scenario will act as the central hub for all randomization activities that occur when your scene is executed.
Next, add a few randomizers to the scenario. The randomizers, in conjunction with the scenario, will perform the actual randomization activities within the simulation.
Next, add a few Randomizers to the Scenario. The Randomizers, in conjunction with the Scenario, will perform the actual randomization activities within the simulation.
Once the project has been randomized and your scene has been configured with the data capture tools available in the Perception package, enter play mode in the editor or execute your scenario through the Unity Simulation Cloud service to generate domain randomized perception data.
Once the project has been randomized and your scene has been configured with the data capture tools available in the Perception package, enter play mode in the editor or execute your Scenario through the Unity Simulation cloud service to generate domain randomized perception data.
Continue reading for more details concerning the primary components driving randomizations in the Perception package, including:
1. Scenarios
## Scenarios
Within a randomized simulation, the scenario component has three responsibilities:
Within a randomized simulation, the Scenario component has three responsibilities:
2. Defining a list of randomizers
2. Defining a list of Randomizers
The fundamental principle of domain randomization is to simulate environments under a variety of randomized conditions. Each **iteration** of a scenario is intended to encapsulate one complete run of a simulated environment under uniquely randomized conditions. Scenarios futher define what conditions determine the end of an iteration and how many iterations to perform.
The fundamental principle of domain randomization is to simulate environments under a variety of randomized conditions. Each Iteration of a Scenario is intended to encapsulate one complete run of a simulated environment under uniquely randomized conditions. Scenarios further define what conditions determine the end of an Iteration and how many Iterations to perform.
To actually randomize a simulation, randomizers can be added to a scenario to vary different simulation properties. At runtime, the scenario will execute each randomizer according to its place within the randomizers list.
To actually randomize a simulation, Randomizers can be added to a Scenario to vary different simulation properties. At runtime, the Scenario will execute each Randomizer according to its place within the Randomizer list.
To read more about scenarios and how to customize them, navigate over to the [scenarios doc](Scenarios.md).
To read more about Scenarios and how to customize them, navigate over to the **[Scenarios documentation](Scenarios.md)**.
Randomizers encapsulate specific randomization activities to perform during the lifecycle of a randomized simulation. For example, randomizers exist for spawning objects, repositioning lights, varying the color of objects, etc. Randomizers expose random Parameters to their inspector interface to further customize these variations.
Randomizers encapsulate specific randomization activities to perform during the lifecycle of a randomized simulation. For example, Randomizers exist for spawning objects, repositioning lights, varying the color of objects, etc. Randomizers expose random Parameters to their inspector interface to further customize these variations.
To read more about how to create custom Parameter types, navigate over to the **[Randomizers doc](Randomizers.md)**.
To read more about how to create custom Parameter types, navigate over to the **[Randomizers documentation](Randomizers.md)**.
RandomizerTags are the primary mechanism by which randomizers query for a certain subset of GameObjects to randoize within a simulation. For example, a rotation randomizer could query for all GameObjects with a RotationRandomizerTag component to obtain an array of all objects the randomizer should vary for the given simulation iteration.
RandomizerTags are the primary mechanism by which Randomizers query for a certain subset of GameObjects to randomize within a simulation. For example, a rotation Randomizer could query for all GameObjects with a RotationRandomizerTag component to obtain an array of all objects the Randomizer should vary for the given simulation Iteration.
To read more about how to use RandomizerTags, navigate over to the **[RandomizerTags doc](RandomizerTags.md)**.
To read more about how to use RandomizerTags, navigate over to the **[RandomizerTags documentation](RandomizerTags.md)**.
Parameters are classes that utilize samplers to deterministically generate random typed values. Parameters are often exposed within the inspector interface of randomizers to allow users to customize said randomizer's behavior. To accomplish this, Parameters combine and transform the float values produced by one or more samplers into various C# types. For example, a Vector3 Parameter can be used to map three samplers to the x, y, and z dimensions of a GameObject. Or a material Parameter can utilize a sampler to randomly select one material from a list of possible options.
Parameters are classes that utilize Samplers to deterministically generate random typed values. Parameters are often exposed within the inspector interface of Randomizers to allow users to customize said Randomizer's behavior. To accomplish this, Parameters combine and transform the float values produced by one or more Samplers into various C# types. For example, a Vector3 Parameter can be used to map three Samplers to the x, y, and z dimensions of a GameObject. Or a material Parameter can utilize a Sampler to randomly select one material from a list of possible options.
To read more about how to create custom Parameter types, navigate over to the **[Parameters doc](Parameters.md)**.
To read more about how to create custom Parameter types, navigate over to the **[Parameters documentation](Parameters.md)**.
RandomizerTags are the primary mechanism by which Randomizers query for a certain subset of GameObjects to randomize within a simulation.
More specifically, RandomizerTags are components that can be added to GameObjects to register them with the active scenario's TagManager. This TagManager is aware of all objects with tags in the scene and can be queried to find all GameObjects that contain a specific tag. Below is a simple example of a ColorRandomizer querying for all GameObjects with a ColorRandomizerTag that it will apply a random material base color to:
More specifically, RandomizerTags are components that can be added to GameObjects to register them with the active Scenario's TagManager. This TagManager is aware of all objects with tags in the scene and can be queried to find all GameObjects that contain a specific tag. Below is a simple example of a ColorRandomizer querying for all GameObjects with a ColorRandomizerTag that it will apply a random material base color to:
1. OnCreate() - called when the Randomizer is added or loaded to a scenario
2. OnIterationStart() - called at the start of a new scenario iteration
3. OnIterationEnd() - called the after a scenario iteration has completed
4. OnScenarioComplete() - called the after the entire scenario has completed
1. OnCreate() - called when the Randomizer is added or loaded to a Scenario
2. OnIterationStart() - called at the start of a new Scenario Iteration
3. OnIterationEnd() - called the after a Scenario Iteration has completed
4. OnScenarioComplete() - called the after the entire Scenario has completed
5. OnStartRunning() - called on the first frame a Randomizer is enabled
6. OnStopRunning() - called on the first frame a disabled Randomizer is updated
7. OnUpdate() - executed every frame for enabled Randomizers
There are a few key things to note from this example:
1. Make sure to add the [Serializable] tag to all Randomizer implementations to ensure that the Randomizer can be customized and saved within the Unity Editor.
2. The [AddRandomizerMenu] attribute customizes the "Add Randomizer" sub menu path in the scenario inspector for a particular randomizer. In this example, the RotationRandomizer can be added to a scenario by opening the _**Add Randomizer**_ menu and clicking `Perception -> Rotation Randomizer`.
2. The [AddRandomizerMenu] attribute customizes the "Add Randomizer" sub menu path in the Scenario inspector for a particular Randomizer. In this example, the RotationRandomizer can be added to a Scenario by opening the _**Add Randomizer**_ menu and clicking `Perception -> Rotation Randomizer`.
3. The line `var taggedObjects = tagManager.Query<RotationRandomizerTag>();` uses RandomizerTags in combination with the current Scenario's tagManager to query for all objects with RotationRandomizerTags and obtain the subset of GameObjects within the simulation that need to have their rotations randomzied. To learn more about how RandomizerTags work, visit the [RandomizerTags documentation page](RandomizerTags.md).
Samplers generate random values that are seeded by the active scenario's current random state. Changing the scenario's random seed will result in Samplers generating different values. Changing the order of Samplers, Parameters, or Randomizers will also result in different values being sampled during a simulation.
Samplers generate random values that are seeded by the active Scenario's current random state. Changing the Scenario's random seed will result in Samplers generating different values. Changing the order of Samplers, Parameters, or Randomizers will also result in different values being sampled during a simulation.
It is recommended that users do not generate random values using the UnityEngine.Random() class or the System.Random() class within a simulation since both of these classes can potentially generate non-deterministic or improperly seeded random values. Using only Perception Samplers to generate random values will help ensure that Perception simulations generate consistent results during local execution and on Unity Simulation in the cloud.
3. Defining constants that can be configured externally from a built Unity player
By default, the perception package includes one ready-made scenario, the `FixedLengthScenario` class. This scenario runs each iteration for a fixed number of frames and is compatible with the Run in Unity Simulation window for cloud simulation execution.
By default, the Perception package includes one ready-made Scenario, the `FixedLengthScenario` class. This Scenario runs each Iteration for a fixed number of frames and is compatible with the Run in Unity Simulation window for cloud simulation execution.
Users can utilize Unity's Unity Simulation service to execute a scenario in the cloud through the perception package's Run in Unity Simulation window. To open this window from the Unity editor using the top menu bar, navigate to `Window -> Run in Unity Simulation`.
Users can utilize Unity's Unity Simulation service to execute a Scenario in the cloud through the perception package's Run in Unity Simulation window. To open this window from the Unity editor using the top menu bar, navigate to `Window -> Run in Unity Simulation`.
2. **Total Iterations** - The number of scenario iterations to complete during the run
2. **Total Iterations** - The number of Scenario Iterations to complete during the run
5. **Scenario** - The scenario to execute
6. **Sys-Param** - The system parameters or the hardware configuration of Unity Simulation worker instances to execute the scenario with. Determines per instance specifications such as the number of CPU cores, amount of memory, and presence of a GPU for accelerated execution.
5. **Scenario** - The Scenario to execute
6. **Sys-Param** - The system parameters or the hardware configuration of Unity Simulation worker instances to execute the Scenario with. Determines per instance specifications such as the number of CPU cores, amount of memory, and presence of a GPU for accelerated execution.
NOTE: To execute a scenario using the Run in Unity Simulation window, the scenario class must implement the UnitySimulationScenario class.
NOTE: To execute a Scenario using the Run in Unity Simulation window, the Scenario class must implement the UnitySimulationScenario class.
For use cases where the scenario should run for an arbitrary number of frames, implementing a custom scenario may be necessary. Below are the two most common scenario properties a user might want to override to implement custom scenario iteration conditions:
1. **isIterationComplete** - determines the conditions that cause the end of a scenario iteration
2. **isScenarioComplete** - determines the conditions that cause the end of a scenario
For use cases where the Scenario should run for an arbitrary number of frames, implementing a custom Scenario may be necessary. Below are the two most common Scenario properties a user might want to override to implement custom Scenario Iteration conditions:
1. **isIterationComplete** - determines the conditions that cause the end of a Scenario Iteration
2. **isScenarioComplete** - determines the conditions that cause the end of a Scenario
Scenarios can be serialized to JSON, modified, and reimported at runtime to configure simulation behavior even after a Unity player has been built. Constants and Randomizer Sampler settings are the two primary sections generated when serializing a scenario. Note that currently, only numerical samplers are serialized. Below is the contents of a JSON configuration file created when serializing the scenario used in Phase 1 of the [Perception Tutorial](../Tutorial/TUTORIAL.md):
Scenarios can be serialized to JSON, modified, and reimported at runtime to configure simulation behavior even after a Unity player has been built. Constants and Randomizer Sampler settings are the two primary sections generated when serializing a Scenario. Note that currently, only numerical samplers are serialized. Below is the contents of a JSON configuration file created when serializing the Scenario used in Phase 1 of the [Perception Tutorial](../Tutorial/TUTORIAL.md):
```
{
"constants": {
### Constants
Constants can include properties such as starting iteration value or total iteration count, and you can always add your own custom constants. Below is an example of the Constants class used in the `FixedLengthScenario` class:
Constants can include properties such as starting Iteration value or total Iteration count, and you can always add your own custom constants. Below is an example of the Constants class used in the `FixedLengthScenario` class:
```
[Serializable]
public class Constants : UnitySimulationScenarioConstants
There are a few key things to note here:
1. The Constants class will need to inherit from `UnitySimulationScenarioConstants` to be compatible with the Run in Unity Simulation window. Deriving from `UnitySimulationScenarioConstants` will add a few key properties to the Constants class that are needed to coordinate a Unity Simulation run.
2. Make sure to include the `[Serializable]` attribute on a constant class. This will ensure that the Constants can be manipulated from the Unity inspector.
3. A scenario class's `SerializeToJson()` and `DeserializeFromJson()` methods can be overridden to implement custom serialization strategies.
3. A Scenario class's `SerializeToJson()` and `DeserializeFromJson()` methods can be overridden to implement custom serialization strategies.
Follow the instructions below to generate a scenario configuration file to modify your scenario Constants and Randomizers in a built player:
1. Click the serialize constants button in the scenario's inspector window. This will generate a `scenario_configuration.json` file and place it in the project's Assets/StreamingAssets folder.
Follow the instructions below to generate a Scenario configuration file to modify your Scenario Constants and Randomizers in a built player:
1. Click the _**Serialize Constants**_ button in the Scenario's inspector window. This will generate a `scenario_configuration.json` file and place it in the project's Assets/StreamingAssets folder.
2. Build your player. The new player will have a [ProjectName]_Data/StreamingAssets folder. A copy of the `scenario_configuration.json` file previously constructed in the editor will be found in this folder.
3. Change the contents of the `scenario_configuration.json` file. Any running player thereafter will utilize the newly authored values.
Each package can come with a set of samples. As seen in the righthand panel, the Perception package includes a sample named _**Tutorial Files**_, which will be required for completing this tutorial. The sample files consist of example foreground and background objects, randomizers, shaders, and other useful elements to work with during this tutorial. **Foreground** objects are those that the eventual machine learning model will try to detect, and **background** objects will be placed in the background as distractors for the model.
Each package can come with a set of samples. As seen in the righthand panel, the Perception package includes a sample named _**Tutorial Files**_, which will be required for completing this tutorial. The sample files consist of example foreground and background objects, Randomizer, shaders, and other useful elements to work with during this tutorial. **Foreground** objects are those that the eventual machine learning model will try to detect, and **background** objects will be placed in the background as distractors for the model.
* **:green_circle: Action**: In the _**Package Manager**_ window, from the list of _**Samples**_ for the Perception package, click on the _**Import into Project**_ button for the sample named _**Tutorial Files**_.
In this folder, you will find a few types of data, depending on your `Perception Camera` settings. These can include:
- Logs
- JSON data
- RGB images (raw camera output) (if the `Save Camera Output to Disk` checkmark is enabled on `Perception Camera`)
- RGB images (raw camera output) (if the `Save Camera Output to Disk` checkmark is enabled on `Perception Camera`)
- Semantic segmentation images (if the `SemanticSegmentationLabeler` is added and active on `Perception Camera`)
The output dataset includes a variety of information about different aspects of the active sensors in the Scene (currently only one), as well as the ground-truth generated by all active labelers. [This page](https://github.com/Unity-Technologies/com.unity.perception/blob/master/com.unity.perception/Documentation%7E/Schema/Synthetic_Dataset_Schema.md) provides a comprehensive explanation on the schema of this dataset. We strongly recommend having a look at the page once you have completed this tutorial.
The purpose of this piece of code is to obtain a random float parameter and assign it to the light's `Intensity` field on the start of every Iteration. Let's go through the code above and understand each part. The `FloatParameter` field makes it possible for us to define a randomized float parameter and modify its properties from the editor UI, similar to how we already modified the properties for the previous Randomizers we used.
The purpose of this piece of code is to obtain a random float Parameter and assign it to the light's `Intensity` field on the start of every Iteration. Let's go through the code above and understand each part. The `FloatParameter` field makes it possible for us to define a randomized float Parameter and modify its properties from the editor UI, similar to how we already modified the properties for the previous Randomizers we used.
> :information_source: If you look at the _**Console**_ tab of the editor now, you will see an error regarding `MyLightRandomizerTag` not being found. This is to be expected, since we have not yet created this class; the error will go away once we create the class later.
}
```
If you now check the UI snippet for `MyLightRandomizer`, you will notice that `Color Parameter` is added. This Parameter includes four separate randomized values for `Red`, `Green`, `Blue` and `Alpha`. Note that the meaningful range for all of these values is 0-1 (and not 0-255). You can see that the sampling range for red, green, and blue is currently also set to 0-1, which means the parameter covers a full range of colors. A color with (0,0,0) RGB components essentially emits no light. So, let's increase the minimum a bit to avoid such a scenario.
If you now check the UI snippet for `MyLightRandomizer`, you will notice that `Color Parameter` is added. This Parameter includes four separate randomized values for `Red`, `Green`, `Blue` and `Alpha`. Note that the meaningful range for all of these values is 0-1 (and not 0-255). You can see that the sampling range for red, green, and blue is currently also set to 0-1, which means the Parameter covers a full range of colors. A color with (0,0,0) RGB components essentially emits no light. So, let's increase the minimum a bit to avoid such a scenario.
* **:green_circle: Action**: Increase the minimum value for red, green, and blue components to 0.4 (this is an arbitrary number that typically produces good-looking results).
</p>
* **:green_circle: Action**: Run the simulation for a few frames to observe the lighting color changing on each iteration.
* **:green_circle: Action**: Run the simulation for a few frames to observe the lighting color changing on each Iteration of the Scenario.
### <aname="step-2">Step 2: Bundle Data and Logic Inside RandomizerTags</a>
* **:green_circle: Action**: Choose `TutorialScene` (which is the Scene we have been working in) as your _**Main Scene**_ and the `SimulationScenario` object as your _**Scenario**_.
Here, you can also specify a name for the run, the number of iterations the Scenario will execute for, and the number of _**Instances**_ (number of nodes the work will be distributed across) for the run.
Here, you can also specify a name for the run, the number of Iterations the Scenario will execute for, and the number of _**Instances**_ (number of nodes the work will be distributed across) for the run.
* **:green_circle: Action**: Name your run `FirstRun`, set the number of iterations to `1000`, and instances to `20`.
* **:green_circle: Action**: Name your run `FirstRun`, set the number of Iterations to `1000`, and Instances to `20`.
* **:green_circle: Action**: Click _**Build and Run**_.
Your project will now be built and then uploaded to Unity Simulation. This may take a few minutes to complete, during which the editor may become frozen; this is normal behaviour.
You may notice that the IDs seen above for the run named `FirstRun` match those we saw earlier in Unity Editor's _**Console**_. You can see here that the single execution for our recently uploaded build is `In_Progress` and that the execution ID is `yegz4WN`.
Unity Simulation utilizes the ability to run simulation instances in parallel. If you enter a number larger than 1 for the number of instances in the _**Run in Unity Simulation**_ window, your run will be parallelized, and multiple simulation instances will simultaneously execute. You can view the status of all simulation instances using the `usim summarize run-execution <execution-id>` command. This command will tell you how many instances have succeeded, failed, have not run yet, or are in progress. Make sure to replace `<execution-id>` with the execution ID seen in your run list. In the above example, this ID would be `yegz4WN`.
Unity Simulation utilizes the ability to run simulation Instances in parallel. If you enter a number larger than 1 for the number of Instances in the _**Run in Unity Simulation**_ window, your run will be parallelized, and multiple simulation Instances will simultaneously execute. You can view the status of all simulation Instances using the `usim summarize run-execution <execution-id>` command. This command will tell you how many Instances have succeeded, failed, have not run yet, or are in progress. Make sure to replace `<execution-id>` with the execution ID seen in your run list. In the above example, this ID would be `yegz4WN`.
* **:green_circle: Action**: Use the `usim summarize run-execution <execution-id>` command to observe the status of your execution nodes:
In order to get the best out of computer vision models, the training data needs to contain a large degree of variation. This is achieved through randomizing various aspects of your simulation between captured frames. While you will use basic randomizations in Phase 1, Phase 2 of the tutorial will help you learn how to randomize your simulations in more complex ways by guiding you through writing your first customized randomizer in C# code. Once you complete this phase, you will know how to:
* Create custom randomizers by extending our provided samples.
* Coordinate the operation of several randomizers by specifying their order of execution and the objects they affect.
In order to get the best out of computer vision models, the training data needs to contain a large degree of variation. This is achieved through randomizing various aspects of your simulation between captured frames. While you will use basic randomizations in Phase 1, Phase 2 of the tutorial will help you learn how to randomize your simulations in more complex ways by guiding you through writing your first customized Randomizer in C# code. Once you complete this phase, you will know how to:
* Create custom Randomizers by extending our provided samples.
* Coordinate the operation of several Randomizers by specifying their order of execution and the objects they affect.
* Have objects specify criteria (e.g. ranges, means, etc.) and logic (e.g. unique behaviors) for their randomizable attributes.
Mohsen Kamalzadeh
3 年前