浏览代码

Renaming SimulationManager to DatasetCapture (#26)

/main
GitHub 4 年前
当前提交
b41538e8
共有 22 个文件被更改,包括 1119 次插入1125 次删除
  1. 6
      TestProjects/PerceptionURP/Assets/ExampleScripts/CustomAnnotationAndMetricReporter.cs
  2. 4
      com.unity.perception/Documentation~/PerceptionCamera.md
  3. 2
      com.unity.perception/Documentation~/index.md
  4. 16
      com.unity.perception/Documentation~/DatasetCapture.md
  5. 2
      com.unity.perception/Runtime/GroundTruth/Ego.cs
  6. 16
      com.unity.perception/Runtime/GroundTruth/PerceptionCamera.cs
  7. 2
      com.unity.perception/Runtime/GroundTruth/SimulationManagementComponentSystem.cs
  8. 12
      com.unity.perception/Runtime/GroundTruth/SimulationState_Json.cs
  9. 44
      com.unity.perception/Runtime/GroundTruth/DatasetCapture.cs
  10. 4
      com.unity.perception/Tests/Editor/PerceptionCameraEditorTests.cs
  11. 8
      com.unity.perception/Tests/Editor/SimulationManagerEditorTests.cs
  12. 6
      com.unity.perception/Tests/Runtime/GroundTruthTests/GroundTruthTestBase.cs
  13. 8
      com.unity.perception/Tests/Runtime/GroundTruthTests/PerceptionCameraIntegrationTests.cs
  14. 176
      com.unity.perception/Tests/Runtime/GroundTruthTests/DatasetCaptureSensorSchedulingTests.cs
  15. 878
      com.unity.perception/Tests/Runtime/GroundTruthTests/DatasetCaptureTests.cs
  16. 878
      com.unity.perception/Tests/Runtime/GroundTruthTests/SimulationManagerTests.cs
  17. 3
      com.unity.perception/Tests/Runtime/GroundTruthTests/SimulationManagerTests.cs.meta
  18. 3
      com.unity.perception/Tests/Runtime/GroundTruthTests/SimulationManagerSensorSchedulingTests.cs.meta
  19. 176
      com.unity.perception/Tests/Runtime/GroundTruthTests/SimulationManagerSensorSchedulingTests.cs
  20. 0
      /com.unity.perception/Documentation~/DatasetCapture.md
  21. 0
      /com.unity.perception/Runtime/GroundTruth/DatasetCapture.cs.meta
  22. 0
      /com.unity.perception/Runtime/GroundTruth/DatasetCapture.cs

6
TestProjects/PerceptionURP/Assets/ExampleScripts/CustomAnnotationAndMetricReporter.cs


public void Start()
{
//Metrics and annotations are registered up-front
lightMetricDefinition = SimulationManager.RegisterMetricDefinition(
lightMetricDefinition = DatasetCapture.RegisterMetricDefinition(
boundingBoxAnnotationDefinition = SimulationManager.RegisterAnnotationDefinition(
boundingBoxAnnotationDefinition = DatasetCapture.RegisterAnnotationDefinition(
"Target bounding box",
"The position of the target in the camera's local space",
id: Guid.Parse("C0B4A22C-0420-4D9F-BAFC-954B8F7B35A7"));

{
//Report the light's position by manually creating the json array string.
var lightPos = light.transform.position;
SimulationManager.ReportMetric(lightMetricDefinition,
DatasetCapture.ReportMetric(lightMetricDefinition,
$@"[{{ ""x"": {lightPos.x}, ""y"": {lightPos.y}, ""z"": {lightPos.z} }}]");
//compute the location of the object in the camera's local space
Vector3 targetPos = transform.worldToLocalMatrix * target.transform.position;

4
com.unity.perception/Documentation~/PerceptionCamera.md


# The Perception Camera component
The Perception Camera component ensures the attached [Camera](https://docs.unity3d.com/Manual/class-Camera.html) runs at deterministic rates and captures RGB and other Camera-related ground truth to the [JSON dataset](Schema/Synthetic_Dataset_Schema.md) using [SimulationManager](SimulationManager.md). It supports HDRP and URP.
The Perception Camera component ensures the attached [Camera](https://docs.unity3d.com/Manual/class-Camera.html) runs at deterministic rates and captures RGB and other Camera-related ground truth to the [JSON dataset](Schema/Synthetic_Dataset_Schema.md) using [DatasetCapture](DatasetCapture.md). It supports HDRP and URP.
<img src="images/PerceptionCamera.PNG" align="middle"/>

| Description | A description of the camera to be registered in the JSON dataset. |
| Period | The amount of simulation time in seconds between frames for this camera. For more on sensor scheduling, see [SimulationManager](SimulationManager.md). |
| Period | The amount of simulation time in seconds between frames for this camera. For more on sensor scheduling, see [DatasetCapture](DatasetCapture.md). |
| Start Time | The simulation time at which to run the first frame. This time will offset the period, useful for allowing multiple cameras to run at the right times relative to each other. |
| Capture Rgb Images | When this is checked, RGB images will be captured as PNG files in the dataset each frame. |
| Produce Segmentation Images| When this is checked at startup, semantic segmentation images will be captured as PNG files in the dataset each frame. Pixel colors are derived from the [Labeling](GroundTruth-Labeling.md) components attached to the GameObjects and the Labeling Configuration specified on the Perception Camera. |

2
com.unity.perception/Documentation~/index.md


|[Labeling](GroundTruth-Labeling.md)|Component which marks a GameObject and its descendants with a set of labels|
|[Labeling Configuration](GroundTruth-Labeling.md#LabelingConfiguration)|Asset which defines a taxonomy of labels for ground truth generation|
|[Perception Camera](PerceptionCamera.md)|Captures RGB images and ground truth from a [Camera](https://docs.unity3d.com/Manual/class-Camera.html)|
|[SimulationManager](SimulationManager.md)|Ensures sensors are triggered at proper rates and accepts data for the JSON dataset|
|[DatasetCapture](DatasetCapture.md)|Ensures sensors are triggered at proper rates and accepts data for the JSON dataset|

16
com.unity.perception/Documentation~/DatasetCapture.md


# SimulationManager
# DatasetCapture
`SimulationManager` tracks egos, sensors, annotations, and metrics, combining them into a unified [JSON-based dataset](Schema/Synthetic_Dataset_Schema.md) on disk. It also controls the simulation time elapsed per frame to accommodate the active sensors.
`DatasetCapture` tracks egos, sensors, annotations, and metrics, combining them into a unified [JSON-based dataset](Schema/Synthetic_Dataset_Schema.md) on disk. It also controls the simulation time elapsed per frame to accommodate the active sensors.
While sensors are registered, `SimulationManager` ensures that frame timing is deterministic and run at the appropriate simulation times to let each sensor run at its own rate.
While sensors are registered, `DatasetCapture` ensures that frame timing is deterministic and run at the appropriate simulation times to let each sensor run at its own rate.
Custom sensors can be registered using `SimulationManager.RegisterSensor()`. The `period` passed in at registration time determines how often in simulation time frames should be scheduled for the sensor to run. The sensor implementation would then check `ShouldCaptureThisFrame` on the returned `SensorHandle` each frame to determine whether it is time for the sensor to perform a capture. `SensorHandle.ReportCapture` should then be called in each of these frames to report the state of the sensor to populate the dataset.
Custom sensors can be registered using `DatasetCapture.RegisterSensor()`. The `period` passed in at registration time determines how often in simulation time frames should be scheduled for the sensor to run. The sensor implementation would then check `ShouldCaptureThisFrame` on the returned `SensorHandle` each frame to determine whether it is time for the sensor to perform a capture. `SensorHandle.ReportCapture` should then be called in each of these frames to report the state of the sensor to populate the dataset.
In addition to the common annotations and metrics produced by [PerceptionCamera](PerceptionCamera.md), scripts can produce their own via `SimulationManager`. Annotation and metric definitions must first be registered using `SimulationManager.RegisterAnnotationDefinition()` or `SimulationManager.RegisterMetricDefinition()`. These return `AnnotationDefinition` and `MetricDefinition` instances which can then be used to report values during runtime.
In addition to the common annotations and metrics produced by [PerceptionCamera](PerceptionCamera.md), scripts can produce their own via `DatasetCapture`. Annotation and metric definitions must first be registered using `DatasetCapture.RegisterAnnotationDefinition()` or `DatasetCapture.RegisterMetricDefinition()`. These return `AnnotationDefinition` and `MetricDefinition` instances which can then be used to report values during runtime.
Annotations and metrics are always associated with the frame they are reported in. They may also be associated with a specific sensor by using the `Report*` methods on `SensorHandle`.

public void Start()
{
//Metrics and annotations are registered up-front
lightMetricDefinition = SimulationManager.RegisterMetricDefinition(
lightMetricDefinition = DatasetCapture.RegisterMetricDefinition(
boundingBoxAnnotationDefinition = SimulationManager.RegisterAnnotationDefinition(
boundingBoxAnnotationDefinition = DatasetCapture.RegisterAnnotationDefinition(
"Target bounding box",
"The position of the target in the camera's local space",
id: Guid.Parse("C0B4A22C-0420-4D9F-BAFC-954B8F7B35A7"));

{
//Report the light's position by manually creating the json array string.
var lightPos = light.transform.position;
SimulationManager.ReportMetric(lightMetricDefinition,
DatasetCapture.ReportMetric(lightMetricDefinition,
$@"[{{ ""x"": {lightPos.x}, ""y"": {lightPos.y}, ""z"": {lightPos.z} }}]");
//compute the location of the object in the camera's local space
Vector3 targetPos = transform.worldToLocalMatrix * target.transform.position;

2
com.unity.perception/Runtime/GroundTruth/Ego.cs


void EnsureEgoInitialized()
{
if (m_EgoHandle == default)
m_EgoHandle = SimulationManager.RegisterEgo(Description);
m_EgoHandle = DatasetCapture.RegisterEgo(Description);
}
}
}

16
com.unity.perception/Runtime/GroundTruth/PerceptionCamera.cs


//CaptureOptions.useAsyncReadbackIfSupported = false;
m_EgoMarker = this.GetComponentInParent<Ego>();
var ego = m_EgoMarker == null ? SimulationManager.RegisterEgo("") : m_EgoMarker.EgoHandle;
SensorHandle = SimulationManager.RegisterSensor(ego, "camera", description, period, startTime);
var ego = m_EgoMarker == null ? DatasetCapture.RegisterEgo("") : m_EgoMarker.EgoHandle;
SensorHandle = DatasetCapture.RegisterSensor(ego, "camera", description, period, startTime);
var myCamera = GetComponent<Camera>();
var width = myCamera.pixelWidth;

pixel_value = l.value
}).ToArray();
m_SegmentationAnnotationDefinition = SimulationManager.RegisterAnnotationDefinition("semantic segmentation", specs, "pixel-wise semantic segmentation label", "PNG");
m_SegmentationAnnotationDefinition = DatasetCapture.RegisterAnnotationDefinition("semantic segmentation", specs, "pixel-wise semantic segmentation label", "PNG");
m_ClassLabelingTextureReader = new RenderTextureReader<short>(labelingTexture, myCamera,
(frameCount, data, tex) => OnSemanticSegmentationImageRead(frameCount, data));

if (produceObjectCountAnnotations)
{
m_ObjectCountMetricDefinition = SimulationManager.RegisterMetricDefinition("object count", labelingMetricSpec, "Counts of objects for each label in the sensor's view", id: new Guid(objectCountId));
m_ObjectCountMetricDefinition = DatasetCapture.RegisterMetricDefinition("object count", labelingMetricSpec, "Counts of objects for each label in the sensor's view", id: new Guid(objectCountId));
m_BoundingBoxAnnotationDefinition = SimulationManager.RegisterAnnotationDefinition("bounding box", labelingMetricSpec, "Bounding box for each labeled object visible to the sensor", id: new Guid(boundingBoxId));
m_BoundingBoxAnnotationDefinition = DatasetCapture.RegisterAnnotationDefinition("bounding box", labelingMetricSpec, "Bounding box for each labeled object visible to the sensor", id: new Guid(boundingBoxId));
m_RenderedObjectInfoMetricDefinition = SimulationManager.RegisterMetricDefinition("rendered object info", labelingMetricSpec, "Information about each labeled object visible to the sensor", id: new Guid(renderedObjectInfoId));
m_RenderedObjectInfoMetricDefinition = DatasetCapture.RegisterMetricDefinition("rendered object info", labelingMetricSpec, "Information about each labeled object visible to the sensor", id: new Guid(renderedObjectInfoId));
m_RenderedObjectInfoGenerator = new RenderedObjectInfoGenerator(LabelingConfiguration);
World.DefaultGameObjectInjectionWorld.GetExistingSystem<GroundTruthLabelSetupSystem>().Activate(m_RenderedObjectInfoGenerator);

}
RenderPipelineManager.beginCameraRendering += OnBeginCameraRendering;
SimulationManager.SimulationEnding += OnSimulationEnding;
DatasetCapture.SimulationEnding += OnSimulationEnding;
}
// ReSharper disable InconsistentNaming

void OnDisable()
{
SimulationManager.SimulationEnding -= OnSimulationEnding;
DatasetCapture.SimulationEnding -= OnSimulationEnding;
OnSimulationEnding();

2
com.unity.perception/Runtime/GroundTruth/SimulationManagementComponentSystem.cs


{
protected override void OnUpdate()
{
SimulationManager.SimulationState?.Update();
DatasetCapture.SimulationState?.Update();
}
}
}

12
com.unity.perception/Runtime/GroundTruth/SimulationState_Json.cs


public void WriteReferences()
{
var egoReference = new JObject();
egoReference["version"] = SimulationManager.SchemaVersion;
egoReference["version"] = DatasetCapture.SchemaVersion;
egoReference["egos"] = new JArray(m_Egos.Select(e =>
{
var egoObj = new JObject();

WriteJObjectToFile(egoReference, "egos.json");
var sensorReferenceDoc = new JObject();
sensorReferenceDoc["version"] = SimulationManager.SchemaVersion;
sensorReferenceDoc["version"] = DatasetCapture.SchemaVersion;
sensorReferenceDoc["sensors"] = new JArray(m_Sensors.Select(kvp =>
{
var sensorReference = new JObject();

if (annotationDefinitionsJArray.Count > 0)
{
var annotationDefinitionsJObject = new JObject();
annotationDefinitionsJObject.Add("version", SimulationManager.SchemaVersion);
annotationDefinitionsJObject.Add("version", DatasetCapture.SchemaVersion);
annotationDefinitionsJObject.Add("annotation_definitions", annotationDefinitionsJArray);
WriteJObjectToFile(annotationDefinitionsJObject, "annotation_definitions.json");
}

var metricDefinitionsJObject = new JObject();
metricDefinitionsJObject.Add("version", SimulationManager.SchemaVersion);
metricDefinitionsJObject.Add("version", DatasetCapture.SchemaVersion);
metricDefinitionsJObject.Add("metric_definitions", metricDefinitionsJArray);
WriteJObjectToFile(metricDefinitionsJObject, "metric_definitions.json");
}

capturesJArray.Add(JObjectFromPendingCapture(pendingCapture));
var capturesJObject = new JObject();
capturesJObject.Add("version", SimulationManager.SchemaVersion);
capturesJObject.Add("version", DatasetCapture.SchemaVersion);
capturesJObject.Add("captures", capturesJArray);
simulationState.WriteJObjectToFile(capturesJObject,

jArray.Add(JObjectFromPendingMetric(pendingMetric));
var metricsJObject = new JObject();
metricsJObject.Add("version", SimulationManager.SchemaVersion);
metricsJObject.Add("version", DatasetCapture.SchemaVersion);
metricsJObject.Add("metrics", jArray);
WriteJObjectToFile(metricsJObject, $"metrics_{metricsFileIndex:000}.json");

44
com.unity.perception/Runtime/GroundTruth/DatasetCapture.cs


/// Global manager for frame scheduling and output capture for simulations.
/// Data capture follows the schema defined in *TODO: Expose schema publicly*
/// </summary>
public static class SimulationManager
public static class DatasetCapture
{
static readonly Guid k_DatasetGuid = Guid.NewGuid();
internal static SimulationState SimulationState { get; private set; } = CreateSimulationData();

/// <summary>
/// The json metadata schema version the SimulationManager's output conforms to.
/// The json metadata schema version the DatasetCapture's output conforms to.
/// </summary>
public static string SchemaVersion => "0.0.1";

}
/// <summary>
/// A handle to a sensor managed by the <see cref="SimulationManager"/>. It can be used to check whether the sensor
/// A handle to a sensor managed by the <see cref="DatasetCapture"/>. It can be used to check whether the sensor
/// is expected to capture this frame and report captures, annotations, and metrics regarding the sensor.
/// </summary>
public struct SensorHandle : IDisposable, IEquatable<SensorHandle>

}
/// <summary>
/// Whether the sensor is currently enabled. When disabled, the SimulationManager will no longer schedule frames for running captures on this sensor.
/// Whether the sensor is currently enabled. When disabled, the DatasetCapture will no longer schedule frames for running captures on this sensor.
get => SimulationManager.SimulationState.IsEnabled(this);
get => DatasetCapture.SimulationState.IsEnabled(this);
SimulationManager.SimulationState.SetEnabled(this, value);
DatasetCapture.SimulationState.SetEnabled(this, value);
}
}

if (!annotationDefinition.IsValid)
throw new ArgumentException("The given annotationDefinition is invalid", nameof(annotationDefinition));
return SimulationManager.SimulationState.ReportAnnotationFile(annotationDefinition, this, filename);
return DatasetCapture.SimulationState.ReportAnnotationFile(annotationDefinition, this, filename);
}
/// <summary>

if (!annotationDefinition.IsValid)
throw new ArgumentException("The given annotationDefinition is invalid", nameof(annotationDefinition));
return SimulationManager.SimulationState.ReportAnnotationValues(annotationDefinition, this, values);
return DatasetCapture.SimulationState.ReportAnnotationValues(annotationDefinition, this, values);
}
/// <summary>

if (!annotationDefinition.IsValid)
throw new ArgumentException("The given annotationDefinition is invalid", nameof(annotationDefinition));
return SimulationManager.SimulationState.ReportAnnotationAsync(annotationDefinition, this);
return DatasetCapture.SimulationState.ReportAnnotationAsync(annotationDefinition, this);
}
/// <summary>

throw new InvalidOperationException("Capture reported in frame when ShouldCaptureThisFrame is false.");
}
SimulationManager.SimulationState.ReportCapture(this, filename, sensorSpatialData, additionalSensorValues);
DatasetCapture.SimulationState.ReportCapture(this, filename, sensorSpatialData, additionalSensorValues);
}
/// <summary>

public bool ShouldCaptureThisFrame => SimulationManager.SimulationState.ShouldCaptureThisFrame(this);
public bool ShouldCaptureThisFrame => DatasetCapture.SimulationState.ShouldCaptureThisFrame(this);
/// <summary>
/// Report a metric regarding this sensor in the current frame.

if (!ShouldCaptureThisFrame)
throw new InvalidOperationException($"Sensor-based metrics may only be reported when SensorHandle.ShouldCaptureThisFrame is true");
SimulationManager.SimulationState.ReportMetric(metricDefinition, values, this, default);
DatasetCapture.SimulationState.ReportMetric(metricDefinition, values, this, default);
}
/// <summary>

if (!ShouldCaptureThisFrame)
throw new InvalidOperationException($"Sensor-based metrics may only be reported when SensorHandle.ShouldCaptureThisFrame is true");
SimulationManager.SimulationState.ReportMetric(metricDefinition, new JRaw(valuesJsonArray), this, default);
DatasetCapture.SimulationState.ReportMetric(metricDefinition, new JRaw(valuesJsonArray), this, default);
}
/// <summary>

if (!ShouldCaptureThisFrame)
throw new InvalidOperationException($"Sensor-based metrics may only be reported when SensorHandle.ShouldCaptureThisFrame is true");
return SimulationManager.SimulationState.CreateAsyncMetric(metricDefinition, this);
return DatasetCapture.SimulationState.CreateAsyncMetric(metricDefinition, this);
}
/// <summary>

/// <summary>
/// Returns whether this SensorHandle is valid in the current simulation. Nil SensorHandles are never valid.
/// </summary>
public bool IsValid => SimulationManager.IsValid(this.Id);
public bool IsValid => DatasetCapture.IsValid(this.Id);
/// <summary>
/// Returns true if this SensorHandle was default-instantiated.
/// </summary>

{
if (!SimulationManager.IsValid(this.Id))
if (!DatasetCapture.IsValid(this.Id))
throw new InvalidOperationException("SensorHandle has been disposed or its simulation has ended");
}

if (!SensorHandle.ShouldCaptureThisFrame)
throw new InvalidOperationException($"Sensor-based metrics may only be reported when SensorHandle.ShouldCaptureThisFrame is true");
SimulationManager.SimulationState.ReportMetric(metricDefinition, values, SensorHandle, this);
DatasetCapture.SimulationState.ReportMetric(metricDefinition, values, SensorHandle, this);
}
/// <summary>

if (!SensorHandle.ShouldCaptureThisFrame)
throw new InvalidOperationException($"Sensor-based metrics may only be reported when SensorHandle.ShouldCaptureThisFrame is true");
SimulationManager.SimulationState.ReportMetric(metricDefinition, new JRaw(valuesJsonArray), SensorHandle, this);
DatasetCapture.SimulationState.ReportMetric(metricDefinition, new JRaw(valuesJsonArray), SensorHandle, this);
}
/// <summary>

/// <returns>A handle to an AsyncMetric, which can be used to report values for this metric in future frames.</returns>
public AsyncMetric ReportMetricAsync(MetricDefinition metricDefinition) => SimulationManager.SimulationState.CreateAsyncMetric(metricDefinition, SensorHandle, this);
public AsyncMetric ReportMetricAsync(MetricDefinition metricDefinition) => DatasetCapture.SimulationState.CreateAsyncMetric(metricDefinition, SensorHandle, this);
/// <inheritdoc/>
public bool Equals(Annotation other)

}
/// <summary>
/// A metric type, used to define a kind of metric. <see cref="SimulationManager.RegisterMetricDefinition"/>.
/// A metric type, used to define a kind of metric. <see cref="DatasetCapture.RegisterMetricDefinition"/>.
/// </summary>
public struct MetricDefinition : IEquatable<MetricDefinition>
{

}
/// <summary>
/// A metric type, used to define a kind of annotation. <see cref="SimulationManager.RegisterAnnotationDefinition"/>.
/// A metric type, used to define a kind of annotation. <see cref="DatasetCapture.RegisterAnnotationDefinition"/>.
/// </summary>
public struct AnnotationDefinition : IEquatable<AnnotationDefinition>
{

/// The ID of the annotation type. Used in the json metadata to associate anntations with the type.
/// </summary>
public readonly Guid Id;
internal bool IsValid => SimulationManager.IsValid(Id);
internal bool IsValid => DatasetCapture.IsValid(Id);
internal AnnotationDefinition(Guid id)
{

4
com.unity.perception/Tests/Editor/PerceptionCameraEditorTests.cs


var expectedLastFrame = Time.frameCount;
yield return null;
SimulationManager.ResetSimulation();
DatasetCapture.ResetSimulation();
var capturesPath = Path.Combine(SimulationManager.OutputDirectory, "captures_000.json");
var capturesPath = Path.Combine(DatasetCapture.OutputDirectory, "captures_000.json");
var capturesJson = File.ReadAllText(capturesPath);
for (int iFrameCount = expectedFirstFrame; iFrameCount <= expectedLastFrame; iFrameCount++)
{

8
com.unity.perception/Tests/Editor/SimulationManagerEditorTests.cs


namespace GroundTruth
{
public class SimulationManagerEditorTests
public class DatasetCaptureEditorTests
Assert.Throws<InvalidOperationException>(() => SimulationManager.RegisterEgo(""));
Assert.Throws<InvalidOperationException>(() => DatasetCapture.RegisterEgo(""));
Assert.Throws<InvalidOperationException>(() => SimulationManager.RegisterAnnotationDefinition(""));
Assert.Throws<InvalidOperationException>(() => DatasetCapture.RegisterAnnotationDefinition(""));
Assert.Throws<InvalidOperationException>(() => SimulationManager.RegisterMetricDefinition(""));
Assert.Throws<InvalidOperationException>(() => DatasetCapture.RegisterMetricDefinition(""));
}
}
}

6
com.unity.perception/Tests/Runtime/GroundTruthTests/GroundTruthTestBase.cs


m_ObjectsToDestroy.Clear();
SimulationManager.ResetSimulation();
DatasetCapture.ResetSimulation();
if (Directory.Exists(SimulationManager.OutputDirectory))
Directory.Delete(SimulationManager.OutputDirectory, true);
if (Directory.Exists(DatasetCapture.OutputDirectory))
Directory.Delete(DatasetCapture.OutputDirectory, true);
}
public void AddTestObjectForCleanup(GameObject @object) => m_ObjectsToDestroy.Add(@object);

8
com.unity.perception/Tests/Runtime/GroundTruthTests/PerceptionCameraIntegrationTests.cs


//a plane is 10x10 by default, so scale it down to be 10x1 to cover the center half of the image
plane.transform.localScale = new Vector3(10f, -1f, .1f);
yield return null;
SimulationManager.ResetSimulation();
DatasetCapture.ResetSimulation();
var capturesPath = Path.Combine(SimulationManager.OutputDirectory, "captures_000.json");
var capturesPath = Path.Combine(DatasetCapture.OutputDirectory, "captures_000.json");
var capturesJson = File.ReadAllText(capturesPath);
StringAssert.Contains(jsonExpected, capturesJson);
}

this.AddTestObjectForCleanup(TestHelper.CreateLabeledPlane());
yield return null;
SimulationManager.ResetSimulation();
DatasetCapture.ResetSimulation();
var capturesPath = Path.Combine(SimulationManager.OutputDirectory, "captures_000.json");
var capturesPath = Path.Combine(DatasetCapture.OutputDirectory, "captures_000.json");
var capturesJson = File.ReadAllText(capturesPath);
var imagePath = Path.Combine("SemanticSegmentation", expectedImageFilename).Replace(@"\", @"\\");
StringAssert.Contains(imagePath, capturesJson);

176
com.unity.perception/Tests/Runtime/GroundTruthTests/DatasetCaptureSensorSchedulingTests.cs


using System.Collections;
using System.Text.RegularExpressions;
using NUnit.Framework;
using UnityEngine;
using UnityEngine.Perception.GroundTruth;
using UnityEngine.TestTools;
namespace GroundTruthTests
{
[TestFixture]
public class DatasetCaptureSensorSchedulingTests
{
[TearDown]
public void TearDown()
{
Time.timeScale = 1;
DatasetCapture.ResetSimulation();
}
[UnityTest]
public IEnumerator FramesScheduledBySensorConfig()
{
var ego = DatasetCapture.RegisterEgo("ego");
var firstCaptureTime = 1.5f;
var period = .4f;
DatasetCapture.RegisterSensor(ego, "cam", "", period, firstCaptureTime);
float[] deltaTimeSamplesExpected =
{
firstCaptureTime,
period,
period,
period
};
float[] deltaTimeSamples = new float[deltaTimeSamplesExpected.Length];
for (int i = 0; i < deltaTimeSamples.Length; i++)
{
yield return null;
Assert.AreEqual(deltaTimeSamplesExpected[i], Time.deltaTime, 0.0001f);
}
}
[UnityTest]
public IEnumerator FramesScheduled_WithTimeScale_ResultsInProperDeltaTime()
{
var ego = DatasetCapture.RegisterEgo("ego");
var firstCaptureTime = 2f;
var period = 1f;
var timeScale = 2;
Time.timeScale = timeScale;
DatasetCapture.RegisterSensor(ego, "cam", "", period, firstCaptureTime);
float[] deltaTimeSamplesExpected =
{
timeScale * firstCaptureTime,
timeScale * period,
timeScale * period,
timeScale * period
};
float[] deltaTimeSamples = new float[deltaTimeSamplesExpected.Length];
for (int i = 0; i < deltaTimeSamples.Length; i++)
{
yield return null;
Assert.AreEqual(deltaTimeSamplesExpected[i], Time.deltaTime, 0.0001f);
}
}
[UnityTest]
public IEnumerator ChangingTimeScale_CausesDebugError()
{
var ego = DatasetCapture.RegisterEgo("ego");
DatasetCapture.RegisterSensor(ego, "cam", "", 1f, 2f);
yield return null;
Time.timeScale = 5;
yield return null;
LogAssert.Expect(LogType.Error, new Regex("Time\\.timeScale may not change mid-sequence.*"));
}
[UnityTest]
public IEnumerator ChangingTimeScale_DuringStartNewSequence_Succeeds()
{
var ego = DatasetCapture.RegisterEgo("ego");
DatasetCapture.RegisterSensor(ego, "cam", "", 1f, 2f);
yield return null;
Time.timeScale = 1;
DatasetCapture.StartNewSequence();
yield return null;
}
[Ignore("Changing timeScale mid-sequence is not supported")]
[UnityTest]
public IEnumerator FramesScheduled_WithChangingTimeScale_ResultsInProperDeltaTime()
{
var ego = DatasetCapture.RegisterEgo("ego");
var firstCaptureTime = 2f;
var period = 1f;
float[] newTimeScalesPerFrame =
{
2f,
10f,
.01f,
1f
};
DatasetCapture.RegisterSensor(ego, "cam", "", period, firstCaptureTime);
float[] deltaTimeSamplesExpected =
{
newTimeScalesPerFrame[0] * firstCaptureTime,
newTimeScalesPerFrame[1] * period,
newTimeScalesPerFrame[2] * period,
newTimeScalesPerFrame[3] * period
};
float[] deltaTimeSamples = new float[deltaTimeSamplesExpected.Length];
for (int i = 0; i < deltaTimeSamples.Length; i++)
{
Time.timeScale = newTimeScalesPerFrame[i];
yield return null;
Assert.AreEqual(deltaTimeSamplesExpected[i], Time.deltaTime, 0.0001f);
}
}
[UnityTest]
public IEnumerator ResetSimulation_ResetsCaptureDeltaTime()
{
var ego = DatasetCapture.RegisterEgo("ego");
DatasetCapture.RegisterSensor(ego, "cam", "", 4, 10);
yield return null;
Assert.AreEqual(10, Time.captureDeltaTime);
DatasetCapture.ResetSimulation();
Assert.AreEqual(0, Time.captureDeltaTime);
}
[UnityTest]
public IEnumerator ShouldCaptureThisFrame_ReturnsTrueOnProperFrames()
{
var ego = DatasetCapture.RegisterEgo("ego");
var firstCaptureTime1 = 10;
var frequencyInMs1 = 4;
var sensor1 = DatasetCapture.RegisterSensor(ego, "cam", "1", frequencyInMs1, firstCaptureTime1);
var firstCaptureTime2 = 10;
var frequencyInMs2 = 6;
var sensor2 = DatasetCapture.RegisterSensor(ego, "cam", "2", frequencyInMs2, firstCaptureTime2);
var sensor3 = DatasetCapture.RegisterSensor(ego, "cam", "3", 1, 1);
sensor3.Enabled = false;
(float deltaTime, bool sensor1ShouldCapture, bool sensor2ShouldCapture)[] samplesExpected =
{
((float)firstCaptureTime1, true, true),
(4, true, false),
(2, false, true),
(2, true, false),
(4, true, true)
};
var samplesActual = new(float deltaTime, bool sensor1ShouldCapture, bool sensor2ShouldCapture)[samplesExpected.Length];
for (int i = 0; i < samplesActual.Length; i++)
{
yield return null;
samplesActual[i] = (Time.deltaTime, sensor1.ShouldCaptureThisFrame, sensor2.ShouldCaptureThisFrame);
}
CollectionAssert.AreEqual(samplesExpected, samplesActual);
}
[Test]
public void Enabled_StartsTrue()
{
var sensor1 = DatasetCapture.RegisterSensor(DatasetCapture.RegisterEgo(""), "cam", "1", 1, 1);
Assert.IsTrue(sensor1.Enabled);
}
}
}

878
com.unity.perception/Tests/Runtime/GroundTruthTests/DatasetCaptureTests.cs


using System;
using System.Collections;
using System.Collections.Generic;
using System.Globalization;
using System.IO;
using System.Linq;
using System.Text.RegularExpressions;
using Newtonsoft.Json;
using Newtonsoft.Json.Linq;
using NUnit.Framework;
using Unity.Mathematics;
using UnityEngine;
using UnityEngine.Perception.GroundTruth;
using UnityEngine.TestTools;
// ReSharper disable InconsistentNaming
// ReSharper disable NotAccessedField.Local
namespace GroundTruthTests
{
[TestFixture]
public class DatasetCaptureTests
{
[Test]
public void RegisterSensor_ReportsProperJson()
{
var egoDescription = @"the main car driving in simulation";
var sensorDescription = "Cam (FL2-14S3M-C)";
var modality = "camera";
var egoJsonExpected =
$@"{{
""version"": ""{DatasetCapture.SchemaVersion}"",
""egos"": [
{{
""id"": <guid>,
""description"": ""{egoDescription}""
}}
]
}}";
var sensorJsonExpected =
$@"{{
""version"": ""{DatasetCapture.SchemaVersion}"",
""sensors"": [
{{
""id"": <guid>,
""ego_id"": <guid>,
""modality"": ""{modality}"",
""description"": ""{sensorDescription}""
}}
]
}}";
var ego = DatasetCapture.RegisterEgo(egoDescription);
var sensorHandle = DatasetCapture.RegisterSensor(ego, modality, sensorDescription, 1, 1);
Assert.IsTrue(sensorHandle.IsValid);
DatasetCapture.ResetSimulation();
Assert.IsFalse(sensorHandle.IsValid);
var sensorsPath = Path.Combine(DatasetCapture.OutputDirectory, "sensors.json");
var egosPath = Path.Combine(DatasetCapture.OutputDirectory, "egos.json");
FileAssert.Exists(egosPath);
FileAssert.Exists(sensorsPath);
AssertJsonFileEquals(egoJsonExpected, egosPath);
AssertJsonFileEquals(sensorJsonExpected, sensorsPath);
}
[Test]
public void ReportCapture_ReportsProperJson()
{
var filename = "my/file.png";
var egoPosition = new float3(.02f, .03f, .04f);
var egoRotation = new quaternion(.1f, .2f, .3f, .4f);
var egoVelocity = new Vector3(.1f, .2f, .3f);
var position = new float3(.2f, 1.1f, .3f);
var rotation = new quaternion(.3f, .2f, .1f, .5f);
var intrinsics = new float3x3(.1f, .2f, .3f, 1f, 2f, 3f, 10f, 20f, 30f);
var capturesJsonExpected =
$@"{{
""version"": ""{DatasetCapture.SchemaVersion}"",
""captures"": [
{{
""id"": <guid>,
""sequence_id"": <guid>,
""step"": 0,
""timestamp"": 0.0,
""sensor"": {{
""sensor_id"": <guid>,
""ego_id"": <guid>,
""modality"": ""camera"",
""translation"": [
{Format(position.x)},
{Format(position.y)},
{Format(position.z)}
],
""rotation"": [
{Format(rotation.value.x)},
{Format(rotation.value.y)},
{Format(rotation.value.z)},
{Format(rotation.value.w)}
],
""camera_intrinsic"": [
[
{Format(intrinsics.c0.x)},
{Format(intrinsics.c0.y)},
{Format(intrinsics.c0.z)}
],
[
{Format(intrinsics.c1.x)},
{Format(intrinsics.c1.y)},
{Format(intrinsics.c1.z)}
],
[
{Format(intrinsics.c2.x)},
{Format(intrinsics.c2.y)},
{Format(intrinsics.c2.z)}
]
]
}},
""ego"": {{
""ego_id"": <guid>,
""translation"": [
{Format(egoPosition.x)},
{Format(egoPosition.y)},
{Format(egoPosition.z)}
],
""rotation"": [
{Format(egoRotation.value.x)},
{Format(egoRotation.value.y)},
{Format(egoRotation.value.z)},
{Format(egoRotation.value.w)}
],
""velocity"": [
{Format(egoVelocity.x)},
{Format(egoVelocity.y)},
{Format(egoVelocity.z)}
],
""acceleration"": null
}},
""filename"": ""{filename}"",
""format"": ""PNG""
}}
]
}}";
var ego = DatasetCapture.RegisterEgo("");
var sensorHandle = DatasetCapture.RegisterSensor(ego, "camera", "", 1, 0);
var sensorSpatialData = new SensorSpatialData(new Pose(egoPosition, egoRotation), new Pose(position, rotation), egoVelocity, null);
sensorHandle.ReportCapture(filename, sensorSpatialData, ("camera_intrinsic", intrinsics));
DatasetCapture.ResetSimulation();
Assert.IsFalse(sensorHandle.IsValid);
var capturesPath = Path.Combine(DatasetCapture.OutputDirectory, "captures_000.json");
FileAssert.Exists(capturesPath);
AssertJsonFileEquals(capturesJsonExpected, capturesPath);
}
[UnityTest]
public IEnumerator StartNewSequence_ProperlyIncrementsSequence()
{
var timingsExpected = new(int step, int timestamp, bool expectNewSequence)[]
{
(0, 0, true),
(1, 2, false),
(0, 0, true),
(1, 2, false)
};
var ego = DatasetCapture.RegisterEgo("");
var sensorHandle = DatasetCapture.RegisterSensor(ego, "", "", 2, 0);
var sensorSpatialData = new SensorSpatialData(default, default, null, null);
Assert.IsTrue(sensorHandle.ShouldCaptureThisFrame);
sensorHandle.ReportCapture("f", sensorSpatialData);
yield return null;
Assert.IsTrue(sensorHandle.ShouldCaptureThisFrame);
sensorHandle.ReportCapture("f", sensorSpatialData);
yield return null;
DatasetCapture.StartNewSequence();
Assert.IsTrue(sensorHandle.ShouldCaptureThisFrame);
sensorHandle.ReportCapture("f", sensorSpatialData);
yield return null;
Assert.IsTrue(sensorHandle.ShouldCaptureThisFrame);
sensorHandle.ReportCapture("f", sensorSpatialData);
DatasetCapture.ResetSimulation();
Assert.IsFalse(sensorHandle.IsValid);
//read all captures from the output directory
List<JObject> captures = new List<JObject>();
foreach (var capturesPath in Directory.EnumerateFiles(DatasetCapture.OutputDirectory, "captures_*.json"))
{
var capturesText = File.ReadAllText(capturesPath);
var jObject = JToken.ReadFrom(new JsonTextReader(new StringReader(capturesText)));
var captureJArray = (JArray)jObject["captures"];
captures.AddRange(captureJArray.Cast<JObject>());
}
Assert.AreEqual(timingsExpected.Length, captures.Count);
var currentSequenceId = "00";
for (int i = 0; i < timingsExpected.Length; i++)
{
var timingExpected = timingsExpected[i];
var text = captures[i];
Assert.AreEqual(timingExpected.step, text["step"].Value<int>());
Assert.AreEqual(timingExpected.timestamp, text["timestamp"].Value<int>());
var newSequenceId = text["sequence_id"].ToString();
if (timingExpected.expectNewSequence)
Assert.AreNotEqual(newSequenceId, currentSequenceId, $"Expected new sequence in frame {i}, but was same");
else
Assert.AreEqual(newSequenceId, currentSequenceId, $"Expected same sequence in frame {i}, but was new");
currentSequenceId = newSequenceId;
}
}
//Format a float to match Newtonsoft.Json formatting
string Format(float value)
{
var result = value.ToString("R", CultureInfo.InvariantCulture);
if (!result.Contains("."))
return result + ".0";
return result;
}
[Test]
public void ReportAnnotation_AddsProperJsonToCapture()
{
var filename = "my/file.png";
var annotationDefinitionGuid = Guid.NewGuid();
var annotationDefinitionsJsonExpected =
$@"{{
""version"": ""{DatasetCapture.SchemaVersion}"",
""annotation_definitions"": [
{{
""id"": <guid>,
""name"": ""semantic segmentation"",
""description"": ""pixel-wise semantic segmentation label"",
""format"": ""PNG""
}}
]
}}";
var annotationsJsonExpected =
$@" ""annotations"": [
{{
""id"": <guid>,
""annotation_definition"": <guid>,
""filename"": ""annotations/semantic_segmentation_000.png""
}}
]";
var ego = DatasetCapture.RegisterEgo("");
var sensorHandle = DatasetCapture.RegisterSensor(ego, "", "", 1, 0);
sensorHandle.ReportCapture(filename, default);
var annotationDefinition = DatasetCapture.RegisterAnnotationDefinition("semantic segmentation", "pixel-wise semantic segmentation label", "PNG", annotationDefinitionGuid);
sensorHandle.ReportAnnotationFile(annotationDefinition, "annotations/semantic_segmentation_000.png");
DatasetCapture.ResetSimulation();
Assert.IsFalse(sensorHandle.IsValid);
var annotationDefinitionsPath = Path.Combine(DatasetCapture.OutputDirectory, "annotation_definitions.json");
var capturesPath = Path.Combine(DatasetCapture.OutputDirectory, "captures_000.json");
AssertJsonFileEquals(annotationDefinitionsJsonExpected, annotationDefinitionsPath);
FileAssert.Exists(capturesPath);
StringAssert.Contains(annotationsJsonExpected, EscapeGuids(File.ReadAllText(capturesPath)));
}
[Test]
public void ReportAnnotationValues_ReportsProperJson()
{
var values = new[]
{
new TestValues()
{
a = "a string",
b = 10
},
new TestValues()
{
a = "a second string",
b = 20
},
};
var expectedAnnotation = $@" ""annotations"": [
{{
""id"": <guid>,
""annotation_definition"": <guid>,
""values"": [
{{
""a"": ""a string"",
""b"": 10
}},
{{
""a"": ""a second string"",
""b"": 20
}}
]
}}
]";
var ego = DatasetCapture.RegisterEgo("");
var annotationDefinition = DatasetCapture.RegisterAnnotationDefinition("");
var sensorHandle = DatasetCapture.RegisterSensor(ego, "", "", 1, 0);
sensorHandle.ReportAnnotationValues(annotationDefinition, values);
DatasetCapture.ResetSimulation();
var capturesPath = Path.Combine(DatasetCapture.OutputDirectory, "captures_000.json");
FileAssert.Exists(capturesPath);
StringAssert.Contains(expectedAnnotation, EscapeGuids(File.ReadAllText(capturesPath)));
}
[Test]
public void ReportAnnotationFile_WhenCaptureNotExpected_Throws()
{
var ego = DatasetCapture.RegisterEgo("");
var annotationDefinition = DatasetCapture.RegisterAnnotationDefinition("");
var sensorHandle = DatasetCapture.RegisterSensor(ego, "", "", 1, 100);
Assert.Throws<InvalidOperationException>(() => sensorHandle.ReportAnnotationFile(annotationDefinition, ""));
}
[Test]
public void ReportAnnotationValues_WhenCaptureNotExpected_Throws()
{
var ego = DatasetCapture.RegisterEgo("");
var annotationDefinition = DatasetCapture.RegisterAnnotationDefinition("");
var sensorHandle = DatasetCapture.RegisterSensor(ego, "", "", 1, 100);
Assert.Throws<InvalidOperationException>(() => sensorHandle.ReportAnnotationValues(annotationDefinition, new int[0]));
}
[Test]
public void ReportAnnotationAsync_WhenCaptureNotExpected_Throws()
{
var ego = DatasetCapture.RegisterEgo("");
var annotationDefinition = DatasetCapture.RegisterAnnotationDefinition("");
var sensorHandle = DatasetCapture.RegisterSensor(ego, "", "", 1, 100);
Assert.Throws<InvalidOperationException>(() => sensorHandle.ReportAnnotationAsync(annotationDefinition));
}
[Test]
public void ResetSimulation_WithUnreportedAnnotationAsync_LogsError()
{
var ego = DatasetCapture.RegisterEgo("");
var annotationDefinition = DatasetCapture.RegisterAnnotationDefinition("");
var sensorHandle = DatasetCapture.RegisterSensor(ego, "", "", 1, 0);
sensorHandle.ReportAnnotationAsync(annotationDefinition);
DatasetCapture.ResetSimulation();
LogAssert.Expect(LogType.Error, new Regex("Simulation ended with pending .*"));
}
[Test]
public void ResetSimulation_CallsSimulationEnding()
{
int timesCalled = 0;
DatasetCapture.SimulationEnding += () => timesCalled++;
DatasetCapture.ResetSimulation();
DatasetCapture.ResetSimulation();
Assert.AreEqual(2, timesCalled);
}
[Test]
public void AnnotationAsyncIsValid_ReturnsProperValue()
{
LogAssert.ignoreFailingMessages = true; //we aren't worried about "Simulation ended with pending..."
var ego = DatasetCapture.RegisterEgo("");
var annotationDefinition = DatasetCapture.RegisterAnnotationDefinition("");
var sensorHandle = DatasetCapture.RegisterSensor(ego, "", "", 1, 0);
var asyncAnnotation = sensorHandle.ReportAnnotationAsync(annotationDefinition);
Assert.IsTrue(asyncAnnotation.IsValid);
DatasetCapture.ResetSimulation();
Assert.IsFalse(asyncAnnotation.IsValid);
}
[Test]
public void AnnotationAsyncReportFile_ReportsProperJson()
{
var expectedAnnotation = $@" ""annotations"": [
{{
""id"": <guid>,
""annotation_definition"": <guid>,
""filename"": ""annotations/output.png""
}}
]";
var ego = DatasetCapture.RegisterEgo("");
var annotationDefinition = DatasetCapture.RegisterAnnotationDefinition("");
var sensorHandle = DatasetCapture.RegisterSensor(ego, "", "", 1, 0);
var asyncAnnotation = sensorHandle.ReportAnnotationAsync(annotationDefinition);
Assert.IsTrue(asyncAnnotation.IsPending);
asyncAnnotation.ReportFile("annotations/output.png");
Assert.IsFalse(asyncAnnotation.IsPending);
DatasetCapture.ResetSimulation();
var capturesPath = Path.Combine(DatasetCapture.OutputDirectory, "captures_000.json");
FileAssert.Exists(capturesPath);
StringAssert.Contains(expectedAnnotation, EscapeGuids(File.ReadAllText(capturesPath)));
}
public struct TestValues
{
public string a;
public int b;
}
[Test]
public void AnnotationAsyncReportValues_ReportsProperJson()
{
var values = new[]
{
new TestValues()
{
a = "a string",
b = 10
},
new TestValues()
{
a = "a second string",
b = 20
},
};
var expectedAnnotation = $@" ""annotations"": [
{{
""id"": <guid>,
""annotation_definition"": <guid>,
""values"": [
{{
""a"": ""a string"",
""b"": 10
}},
{{
""a"": ""a second string"",
""b"": 20
}}
]
}}
]";
var ego = DatasetCapture.RegisterEgo("");
var annotationDefinition = DatasetCapture.RegisterAnnotationDefinition("");
var sensorHandle = DatasetCapture.RegisterSensor(ego, "", "", 1, 0);
var asyncAnnotation = sensorHandle.ReportAnnotationAsync(annotationDefinition);
Assert.IsTrue(asyncAnnotation.IsPending);
asyncAnnotation.ReportValues(values);
Assert.IsFalse(asyncAnnotation.IsPending);
DatasetCapture.ResetSimulation();
var capturesPath = Path.Combine(DatasetCapture.OutputDirectory, "captures_000.json");
FileAssert.Exists(capturesPath);
StringAssert.Contains(expectedAnnotation, EscapeGuids(File.ReadAllText(capturesPath)));
}
[Test]
public void CreateAnnotation_MultipleTimes_WritesProperTypeOnce()
{
var annotationDefinitionGuid = new Guid(10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0);
var annotationDefinitionsJsonExpected =
$@"{{
""version"": ""{DatasetCapture.SchemaVersion}"",
""annotation_definitions"": [
{{
""id"": ""{annotationDefinitionGuid}"",
""name"": ""name"",
""format"": ""json""
}}
]
}}";
var annotationDefinition1 = DatasetCapture.RegisterAnnotationDefinition("name", id: annotationDefinitionGuid);
var annotationDefinition2 = DatasetCapture.RegisterAnnotationDefinition("name", id: annotationDefinitionGuid);
DatasetCapture.ResetSimulation();
var annotationDefinitionsPath = Path.Combine(DatasetCapture.OutputDirectory, "annotation_definitions.json");
Assert.AreEqual(annotationDefinition1, annotationDefinition2);
Assert.AreEqual(annotationDefinitionGuid, annotationDefinition1.Id);
Assert.AreEqual(annotationDefinitionGuid, annotationDefinition2.Id);
AssertJsonFileEquals(annotationDefinitionsJsonExpected, annotationDefinitionsPath, false);
}
[Test]
public void CreateAnnotation_MultipleTimesWithDifferentParameters_WritesProperTypes()
{
var annotationDefinitionGuid = new Guid(10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0);
var annotationDefinitionsJsonExpected =
$@"{{
""version"": ""{DatasetCapture.SchemaVersion}"",
""annotation_definitions"": [
{{
""id"": <guid>,
""name"": ""name"",
""format"": ""json""
}},
{{
""id"": <guid>,
""name"": ""name2"",
""description"": ""description"",
""format"": ""json""
}}
]
}}";
var annotationDefinition1 = DatasetCapture.RegisterAnnotationDefinition("name", id: annotationDefinitionGuid);
var annotationDefinition2 = DatasetCapture.RegisterAnnotationDefinition("name2", description: "description");
DatasetCapture.ResetSimulation();
var annotationDefinitionsPath = Path.Combine(DatasetCapture.OutputDirectory, "annotation_definitions.json");
Assert.AreEqual(annotationDefinitionGuid, annotationDefinition1.Id);
Assert.AreNotEqual(default(Guid), annotationDefinition2.Id);
AssertJsonFileEquals(annotationDefinitionsJsonExpected, annotationDefinitionsPath);
}
[Test]
public void ReportMetricValues_WhenCaptureNotExpected_Throws()
{
var ego = DatasetCapture.RegisterEgo("");
var metricDefinition = DatasetCapture.RegisterMetricDefinition("");
var sensorHandle = DatasetCapture.RegisterSensor(ego, "", "", 1, 100);
Assert.Throws<InvalidOperationException>(() => sensorHandle.ReportMetric(metricDefinition, new int[0]));
}
[Test]
public void ReportMetricAsync_WhenCaptureNotExpected_Throws()
{
var ego = DatasetCapture.RegisterEgo("");
var metricDefinition = DatasetCapture.RegisterMetricDefinition("");
var sensorHandle = DatasetCapture.RegisterSensor(ego, "", "", 1, 100);
Assert.Throws<InvalidOperationException>(() => sensorHandle.ReportMetricAsync(metricDefinition));
}
[Test]
public void ResetSimulation_WithUnreportedMetricAsync_LogsError()
{
var ego = DatasetCapture.RegisterEgo("");
var metricDefinition = DatasetCapture.RegisterMetricDefinition("");
var sensorHandle = DatasetCapture.RegisterSensor(ego, "", "", 1, 0);
sensorHandle.ReportMetricAsync(metricDefinition);
DatasetCapture.ResetSimulation();
LogAssert.Expect(LogType.Error, new Regex("Simulation ended with pending .*"));
}
[Test]
public void MetricAsyncIsValid_ReturnsProperValue()
{
LogAssert.ignoreFailingMessages = true; //we aren't worried about "Simulation ended with pending..."
var ego = DatasetCapture.RegisterEgo("");
var metricDefinition = DatasetCapture.RegisterMetricDefinition("");
var sensorHandle = DatasetCapture.RegisterSensor(ego, "", "", 1, 0);
var asyncMetric = sensorHandle.ReportMetricAsync(metricDefinition);
Assert.IsTrue(asyncMetric.IsValid);
DatasetCapture.ResetSimulation();
Assert.IsFalse(asyncMetric.IsValid);
}
public enum MetricTarget
{
Global,
Capture,
Annotation
}
[UnityTest]
public IEnumerator MetricReportValues_WithNoReportsInFrames_DoesNotIncrementStep()
{
var values = new[] { 1 };
var expectedLine = @"""step"": 0";
var metricDefinition = DatasetCapture.RegisterMetricDefinition("");
DatasetCapture.RegisterSensor(DatasetCapture.RegisterEgo(""), "", "", 1, 0);
yield return null;
yield return null;
yield return null;
DatasetCapture.ReportMetric(metricDefinition, values);
DatasetCapture.ResetSimulation();
var text = File.ReadAllText(Path.Combine(DatasetCapture.OutputDirectory, "metrics_000.json"));
StringAssert.Contains(expectedLine, text);
}
[UnityTest]
public IEnumerator SensorHandleReportMetric_BeforeReportCapture_ReportsProperJson()
{
var values = new[] { 1 };
var expectedLine = @"""step"": 0";
var metricDefinition = DatasetCapture.RegisterMetricDefinition("");
var sensor = DatasetCapture.RegisterSensor(DatasetCapture.RegisterEgo(""), "", "", 1, 0);
yield return null;
sensor.ReportMetric(metricDefinition, values);
sensor.ReportCapture("file", new SensorSpatialData(Pose.identity, Pose.identity, null, null));
DatasetCapture.ResetSimulation();
var metricsTest = File.ReadAllText(Path.Combine(DatasetCapture.OutputDirectory, "metrics_000.json"));
var captures = File.ReadAllText(Path.Combine(DatasetCapture.OutputDirectory, "captures_000.json"));
StringAssert.Contains(expectedLine, metricsTest);
StringAssert.Contains(expectedLine, captures);
}
[Test]
public void MetricAsyncReportValues_ReportsProperJson(
[Values(MetricTarget.Global, MetricTarget.Capture, MetricTarget.Annotation)] MetricTarget metricTarget,
[Values(true, false)] bool async,
[Values(true, false)] bool asStringJsonArray)
{
var values = new[]
{
new TestValues()
{
a = "a string",
b = 10
},
new TestValues()
{
a = "a second string",
b = 20
},
};
var expectedMetric = $@"{{
""version"": ""0.0.1"",
""metrics"": [
{{
""capture_id"": {(metricTarget == MetricTarget.Annotation || metricTarget == MetricTarget.Capture ? "<guid>" : "null")},
""annotation_id"": {(metricTarget == MetricTarget.Annotation ? "<guid>" : "null")},
""sequence_id"": <guid>,
""step"": 0,
""metric_definition"": <guid>,
""values"": [
{{
""a"": ""a string"",
""b"": 10
}},
{{
""a"": ""a second string"",
""b"": 20
}}
]
}}
]
}}";
var metricDefinition = DatasetCapture.RegisterMetricDefinition("");
var sensor = DatasetCapture.RegisterSensor(DatasetCapture.RegisterEgo(""), "", "", 1, 0);
var annotation = sensor.ReportAnnotationFile(DatasetCapture.RegisterAnnotationDefinition(""), "");
var valuesJsonArray = JArray.FromObject(values).ToString(Formatting.Indented);
if (async)
{
AsyncMetric asyncMetric;
switch (metricTarget)
{
case MetricTarget.Global:
asyncMetric = DatasetCapture.ReportMetricAsync(metricDefinition);
break;
case MetricTarget.Capture:
asyncMetric = sensor.ReportMetricAsync(metricDefinition);
break;
case MetricTarget.Annotation:
asyncMetric = annotation.ReportMetricAsync(metricDefinition);
break;
default:
throw new Exception("unsupported");
}
Assert.IsTrue(asyncMetric.IsPending);
if (asStringJsonArray)
asyncMetric.ReportValues(valuesJsonArray);
else
asyncMetric.ReportValues(values);
Assert.IsFalse(asyncMetric.IsPending);
}
else
{
switch (metricTarget)
{
case MetricTarget.Global:
if (asStringJsonArray)
DatasetCapture.ReportMetric(metricDefinition, valuesJsonArray);
else
DatasetCapture.ReportMetric(metricDefinition, values);
break;
case MetricTarget.Capture:
if (asStringJsonArray)
sensor.ReportMetric(metricDefinition, valuesJsonArray);
else
sensor.ReportMetric(metricDefinition, values);
break;
case MetricTarget.Annotation:
if (asStringJsonArray)
annotation.ReportMetric(metricDefinition, valuesJsonArray);
else
annotation.ReportMetric(metricDefinition, values);
break;
default:
throw new Exception("unsupported");
}
}
DatasetCapture.ResetSimulation();
AssertJsonFileEquals(expectedMetric, Path.Combine(DatasetCapture.OutputDirectory, "metrics_000.json"), escapeGuids: true, ignoreFormatting: true);
}
[Test]
public void CreateMetric_MultipleTimesWithDifferentParameters_WritesProperTypes()
{
var metricDefinitionGuid = new Guid(10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0);
var metricDefinitionsJsonExpected =
$@"{{
""version"": ""{DatasetCapture.SchemaVersion}"",
""metric_definitions"": [
{{
""id"": <guid>,
""name"": ""name""
}},
{{
""id"": <guid>,
""name"": ""name2"",
""description"": ""description""
}}
]
}}";
var metricDefinition1 = DatasetCapture.RegisterMetricDefinition("name", id: metricDefinitionGuid);
var metricDefinition2 = DatasetCapture.RegisterMetricDefinition("name2", description: "description");
DatasetCapture.ResetSimulation();
var metricDefinitionsPath = Path.Combine(DatasetCapture.OutputDirectory, "metric_definitions.json");
Assert.AreEqual(metricDefinitionGuid, metricDefinition1.Id);
Assert.AreNotEqual(default(Guid), metricDefinition2.Id);
AssertJsonFileEquals(metricDefinitionsJsonExpected, metricDefinitionsPath);
}
struct TestSpec
{
public int label_id;
public string label_name;
public int[] pixel_value;
}
public enum AdditionalInfoKind
{
Annotation,
Metric
}
[Test]
public void CreateAnnotationOrMetric_WithSpecValues_WritesProperTypes(
[Values(AdditionalInfoKind.Annotation, AdditionalInfoKind.Metric)] AdditionalInfoKind additionalInfoKind)
{
var specValues = new[]
{
new TestSpec
{
label_id = 1,
label_name = "sky",
pixel_value = new[] { 1, 2, 3}
},
new TestSpec
{
label_id = 2,
label_name = "sidewalk",
pixel_value = new[] { 4, 5, 6}
}
};
string filename;
string jsonContainerName;
if (additionalInfoKind == AdditionalInfoKind.Annotation)
{
DatasetCapture.RegisterAnnotationDefinition("name", specValues);
filename = "annotation_definitions.json";
jsonContainerName = "annotation_definitions";
}
else
{
DatasetCapture.RegisterMetricDefinition("name", specValues);
filename = "metric_definitions.json";
jsonContainerName = "metric_definitions";
}
var additionalInfoString = (additionalInfoKind == AdditionalInfoKind.Annotation ? @"
""format"": ""json""," : null);
var annotationDefinitionsJsonExpected =
$@"{{
""version"": ""{DatasetCapture.SchemaVersion}"",
""{jsonContainerName}"": [
{{
""id"": <guid>,
""name"": ""name"",{additionalInfoString}
""spec"": [
{{
""label_id"": 1,
""label_name"": ""sky"",
""pixel_value"": [
1,
2,
3
]
}},
{{
""label_id"": 2,
""label_name"": ""sidewalk"",
""pixel_value"": [
4,
5,
6
]
}}
]
}}
]
}}";
DatasetCapture.ResetSimulation();
var annotationDefinitionsPath = Path.Combine(DatasetCapture.OutputDirectory, filename);
AssertJsonFileEquals(annotationDefinitionsJsonExpected, annotationDefinitionsPath);
}
static void AssertJsonFileEquals(string jsonExpected, string jsonPath, bool escapeGuids = true, bool ignoreFormatting = false)
{
FileAssert.Exists(jsonPath);
var jsonActual = File.ReadAllText(jsonPath);
if (escapeGuids)
jsonActual = EscapeGuids(jsonActual);
if (ignoreFormatting)
{
jsonActual = Regex.Replace(jsonActual, "^\\s*", "", RegexOptions.Multiline);
jsonExpected = Regex.Replace(jsonExpected, "^\\s*", "", RegexOptions.Multiline);
}
Assert.AreEqual(jsonExpected, jsonActual, $"Expected:\n{jsonExpected}\nActual:\n{jsonActual}");
}
static string EscapeGuids(string text)
{
var result = Regex.Replace(text, @"""[a-z0-9]*-[a-z0-9]*-[a-z0-9]*-[a-z0-9]*-[a-z0-9]*""", "<guid>");
return result;
}
}
}

878
com.unity.perception/Tests/Runtime/GroundTruthTests/SimulationManagerTests.cs


using System;
using System.Collections;
using System.Collections.Generic;
using System.Globalization;
using System.IO;
using System.Linq;
using System.Text.RegularExpressions;
using Newtonsoft.Json;
using Newtonsoft.Json.Linq;
using NUnit.Framework;
using Unity.Mathematics;
using UnityEngine;
using UnityEngine.Perception.GroundTruth;
using UnityEngine.TestTools;
// ReSharper disable InconsistentNaming
// ReSharper disable NotAccessedField.Local
namespace GroundTruthTests
{
[TestFixture]
public class SimulationManagerTests
{
[Test]
public void RegisterSensor_ReportsProperJson()
{
var egoDescription = @"the main car driving in simulation";
var sensorDescription = "Cam (FL2-14S3M-C)";
var modality = "camera";
var egoJsonExpected =
$@"{{
""version"": ""{SimulationManager.SchemaVersion}"",
""egos"": [
{{
""id"": <guid>,
""description"": ""{egoDescription}""
}}
]
}}";
var sensorJsonExpected =
$@"{{
""version"": ""{SimulationManager.SchemaVersion}"",
""sensors"": [
{{
""id"": <guid>,
""ego_id"": <guid>,
""modality"": ""{modality}"",
""description"": ""{sensorDescription}""
}}
]
}}";
var ego = SimulationManager.RegisterEgo(egoDescription);
var sensorHandle = SimulationManager.RegisterSensor(ego, modality, sensorDescription, 1, 1);
Assert.IsTrue(sensorHandle.IsValid);
SimulationManager.ResetSimulation();
Assert.IsFalse(sensorHandle.IsValid);
var sensorsPath = Path.Combine(SimulationManager.OutputDirectory, "sensors.json");
var egosPath = Path.Combine(SimulationManager.OutputDirectory, "egos.json");
FileAssert.Exists(egosPath);
FileAssert.Exists(sensorsPath);
AssertJsonFileEquals(egoJsonExpected, egosPath);
AssertJsonFileEquals(sensorJsonExpected, sensorsPath);
}
[Test]
public void ReportCapture_ReportsProperJson()
{
var filename = "my/file.png";
var egoPosition = new float3(.02f, .03f, .04f);
var egoRotation = new quaternion(.1f, .2f, .3f, .4f);
var egoVelocity = new Vector3(.1f, .2f, .3f);
var position = new float3(.2f, 1.1f, .3f);
var rotation = new quaternion(.3f, .2f, .1f, .5f);
var intrinsics = new float3x3(.1f, .2f, .3f, 1f, 2f, 3f, 10f, 20f, 30f);
var capturesJsonExpected =
$@"{{
""version"": ""{SimulationManager.SchemaVersion}"",
""captures"": [
{{
""id"": <guid>,
""sequence_id"": <guid>,
""step"": 0,
""timestamp"": 0.0,
""sensor"": {{
""sensor_id"": <guid>,
""ego_id"": <guid>,
""modality"": ""camera"",
""translation"": [
{Format(position.x)},
{Format(position.y)},
{Format(position.z)}
],
""rotation"": [
{Format(rotation.value.x)},
{Format(rotation.value.y)},
{Format(rotation.value.z)},
{Format(rotation.value.w)}
],
""camera_intrinsic"": [
[
{Format(intrinsics.c0.x)},
{Format(intrinsics.c0.y)},
{Format(intrinsics.c0.z)}
],
[
{Format(intrinsics.c1.x)},
{Format(intrinsics.c1.y)},
{Format(intrinsics.c1.z)}
],
[
{Format(intrinsics.c2.x)},
{Format(intrinsics.c2.y)},
{Format(intrinsics.c2.z)}
]
]
}},
""ego"": {{
""ego_id"": <guid>,
""translation"": [
{Format(egoPosition.x)},
{Format(egoPosition.y)},
{Format(egoPosition.z)}
],
""rotation"": [
{Format(egoRotation.value.x)},
{Format(egoRotation.value.y)},
{Format(egoRotation.value.z)},
{Format(egoRotation.value.w)}
],
""velocity"": [
{Format(egoVelocity.x)},
{Format(egoVelocity.y)},
{Format(egoVelocity.z)}
],
""acceleration"": null
}},
""filename"": ""{filename}"",
""format"": ""PNG""
}}
]
}}";
var ego = SimulationManager.RegisterEgo("");
var sensorHandle = SimulationManager.RegisterSensor(ego, "camera", "", 1, 0);
var sensorSpatialData = new SensorSpatialData(new Pose(egoPosition, egoRotation), new Pose(position, rotation), egoVelocity, null);
sensorHandle.ReportCapture(filename, sensorSpatialData, ("camera_intrinsic", intrinsics));
SimulationManager.ResetSimulation();
Assert.IsFalse(sensorHandle.IsValid);
var capturesPath = Path.Combine(SimulationManager.OutputDirectory, "captures_000.json");
FileAssert.Exists(capturesPath);
AssertJsonFileEquals(capturesJsonExpected, capturesPath);
}
[UnityTest]
public IEnumerator StartNewSequence_ProperlyIncrementsSequence()
{
var timingsExpected = new(int step, int timestamp, bool expectNewSequence)[]
{
(0, 0, true),
(1, 2, false),
(0, 0, true),
(1, 2, false)
};
var ego = SimulationManager.RegisterEgo("");
var sensorHandle = SimulationManager.RegisterSensor(ego, "", "", 2, 0);
var sensorSpatialData = new SensorSpatialData(default, default, null, null);
Assert.IsTrue(sensorHandle.ShouldCaptureThisFrame);
sensorHandle.ReportCapture("f", sensorSpatialData);
yield return null;
Assert.IsTrue(sensorHandle.ShouldCaptureThisFrame);
sensorHandle.ReportCapture("f", sensorSpatialData);
yield return null;
SimulationManager.StartNewSequence();
Assert.IsTrue(sensorHandle.ShouldCaptureThisFrame);
sensorHandle.ReportCapture("f", sensorSpatialData);
yield return null;
Assert.IsTrue(sensorHandle.ShouldCaptureThisFrame);
sensorHandle.ReportCapture("f", sensorSpatialData);
SimulationManager.ResetSimulation();
Assert.IsFalse(sensorHandle.IsValid);
//read all captures from the output directory
List<JObject> captures = new List<JObject>();
foreach (var capturesPath in Directory.EnumerateFiles(SimulationManager.OutputDirectory, "captures_*.json"))
{
var capturesText = File.ReadAllText(capturesPath);
var jObject = JToken.ReadFrom(new JsonTextReader(new StringReader(capturesText)));
var captureJArray = (JArray)jObject["captures"];
captures.AddRange(captureJArray.Cast<JObject>());
}
Assert.AreEqual(timingsExpected.Length, captures.Count);
var currentSequenceId = "00";
for (int i = 0; i < timingsExpected.Length; i++)
{
var timingExpected = timingsExpected[i];
var text = captures[i];
Assert.AreEqual(timingExpected.step, text["step"].Value<int>());
Assert.AreEqual(timingExpected.timestamp, text["timestamp"].Value<int>());
var newSequenceId = text["sequence_id"].ToString();
if (timingExpected.expectNewSequence)
Assert.AreNotEqual(newSequenceId, currentSequenceId, $"Expected new sequence in frame {i}, but was same");
else
Assert.AreEqual(newSequenceId, currentSequenceId, $"Expected same sequence in frame {i}, but was new");
currentSequenceId = newSequenceId;
}
}
//Format a float to match Newtonsoft.Json formatting
string Format(float value)
{
var result = value.ToString("R", CultureInfo.InvariantCulture);
if (!result.Contains("."))
return result + ".0";
return result;
}
[Test]
public void ReportAnnotation_AddsProperJsonToCapture()
{
var filename = "my/file.png";
var annotationDefinitionGuid = Guid.NewGuid();
var annotationDefinitionsJsonExpected =
$@"{{
""version"": ""{SimulationManager.SchemaVersion}"",
""annotation_definitions"": [
{{
""id"": <guid>,
""name"": ""semantic segmentation"",
""description"": ""pixel-wise semantic segmentation label"",
""format"": ""PNG""
}}
]
}}";
var annotationsJsonExpected =
$@" ""annotations"": [
{{
""id"": <guid>,
""annotation_definition"": <guid>,
""filename"": ""annotations/semantic_segmentation_000.png""
}}
]";
var ego = SimulationManager.RegisterEgo("");
var sensorHandle = SimulationManager.RegisterSensor(ego, "", "", 1, 0);
sensorHandle.ReportCapture(filename, default);
var annotationDefinition = SimulationManager.RegisterAnnotationDefinition("semantic segmentation", "pixel-wise semantic segmentation label", "PNG", annotationDefinitionGuid);
sensorHandle.ReportAnnotationFile(annotationDefinition, "annotations/semantic_segmentation_000.png");
SimulationManager.ResetSimulation();
Assert.IsFalse(sensorHandle.IsValid);
var annotationDefinitionsPath = Path.Combine(SimulationManager.OutputDirectory, "annotation_definitions.json");
var capturesPath = Path.Combine(SimulationManager.OutputDirectory, "captures_000.json");
AssertJsonFileEquals(annotationDefinitionsJsonExpected, annotationDefinitionsPath);
FileAssert.Exists(capturesPath);
StringAssert.Contains(annotationsJsonExpected, EscapeGuids(File.ReadAllText(capturesPath)));
}
[Test]
public void ReportAnnotationValues_ReportsProperJson()
{
var values = new[]
{
new TestValues()
{
a = "a string",
b = 10
},
new TestValues()
{
a = "a second string",
b = 20
},
};
var expectedAnnotation = $@" ""annotations"": [
{{
""id"": <guid>,
""annotation_definition"": <guid>,
""values"": [
{{
""a"": ""a string"",
""b"": 10
}},
{{
""a"": ""a second string"",
""b"": 20
}}
]
}}
]";
var ego = SimulationManager.RegisterEgo("");
var annotationDefinition = SimulationManager.RegisterAnnotationDefinition("");
var sensorHandle = SimulationManager.RegisterSensor(ego, "", "", 1, 0);
sensorHandle.ReportAnnotationValues(annotationDefinition, values);
SimulationManager.ResetSimulation();
var capturesPath = Path.Combine(SimulationManager.OutputDirectory, "captures_000.json");
FileAssert.Exists(capturesPath);
StringAssert.Contains(expectedAnnotation, EscapeGuids(File.ReadAllText(capturesPath)));
}
[Test]
public void ReportAnnotationFile_WhenCaptureNotExpected_Throws()
{
var ego = SimulationManager.RegisterEgo("");
var annotationDefinition = SimulationManager.RegisterAnnotationDefinition("");
var sensorHandle = SimulationManager.RegisterSensor(ego, "", "", 1, 100);
Assert.Throws<InvalidOperationException>(() => sensorHandle.ReportAnnotationFile(annotationDefinition, ""));
}
[Test]
public void ReportAnnotationValues_WhenCaptureNotExpected_Throws()
{
var ego = SimulationManager.RegisterEgo("");
var annotationDefinition = SimulationManager.RegisterAnnotationDefinition("");
var sensorHandle = SimulationManager.RegisterSensor(ego, "", "", 1, 100);
Assert.Throws<InvalidOperationException>(() => sensorHandle.ReportAnnotationValues(annotationDefinition, new int[0]));
}
[Test]
public void ReportAnnotationAsync_WhenCaptureNotExpected_Throws()
{
var ego = SimulationManager.RegisterEgo("");
var annotationDefinition = SimulationManager.RegisterAnnotationDefinition("");
var sensorHandle = SimulationManager.RegisterSensor(ego, "", "", 1, 100);
Assert.Throws<InvalidOperationException>(() => sensorHandle.ReportAnnotationAsync(annotationDefinition));
}
[Test]
public void ResetSimulation_WithUnreportedAnnotationAsync_LogsError()
{
var ego = SimulationManager.RegisterEgo("");
var annotationDefinition = SimulationManager.RegisterAnnotationDefinition("");
var sensorHandle = SimulationManager.RegisterSensor(ego, "", "", 1, 0);
sensorHandle.ReportAnnotationAsync(annotationDefinition);
SimulationManager.ResetSimulation();
LogAssert.Expect(LogType.Error, new Regex("Simulation ended with pending .*"));
}
[Test]
public void ResetSimulation_CallsSimulationEnding()
{
int timesCalled = 0;
SimulationManager.SimulationEnding += () => timesCalled++;
SimulationManager.ResetSimulation();
SimulationManager.ResetSimulation();
Assert.AreEqual(2, timesCalled);
}
[Test]
public void AnnotationAsyncIsValid_ReturnsProperValue()
{
LogAssert.ignoreFailingMessages = true; //we aren't worried about "Simulation ended with pending..."
var ego = SimulationManager.RegisterEgo("");
var annotationDefinition = SimulationManager.RegisterAnnotationDefinition("");
var sensorHandle = SimulationManager.RegisterSensor(ego, "", "", 1, 0);
var asyncAnnotation = sensorHandle.ReportAnnotationAsync(annotationDefinition);
Assert.IsTrue(asyncAnnotation.IsValid);
SimulationManager.ResetSimulation();
Assert.IsFalse(asyncAnnotation.IsValid);
}
[Test]
public void AnnotationAsyncReportFile_ReportsProperJson()
{
var expectedAnnotation = $@" ""annotations"": [
{{
""id"": <guid>,
""annotation_definition"": <guid>,
""filename"": ""annotations/output.png""
}}
]";
var ego = SimulationManager.RegisterEgo("");
var annotationDefinition = SimulationManager.RegisterAnnotationDefinition("");
var sensorHandle = SimulationManager.RegisterSensor(ego, "", "", 1, 0);
var asyncAnnotation = sensorHandle.ReportAnnotationAsync(annotationDefinition);
Assert.IsTrue(asyncAnnotation.IsPending);
asyncAnnotation.ReportFile("annotations/output.png");
Assert.IsFalse(asyncAnnotation.IsPending);
SimulationManager.ResetSimulation();
var capturesPath = Path.Combine(SimulationManager.OutputDirectory, "captures_000.json");
FileAssert.Exists(capturesPath);
StringAssert.Contains(expectedAnnotation, EscapeGuids(File.ReadAllText(capturesPath)));
}
public struct TestValues
{
public string a;
public int b;
}
[Test]
public void AnnotationAsyncReportValues_ReportsProperJson()
{
var values = new[]
{
new TestValues()
{
a = "a string",
b = 10
},
new TestValues()
{
a = "a second string",
b = 20
},
};
var expectedAnnotation = $@" ""annotations"": [
{{
""id"": <guid>,
""annotation_definition"": <guid>,
""values"": [
{{
""a"": ""a string"",
""b"": 10
}},
{{
""a"": ""a second string"",
""b"": 20
}}
]
}}
]";
var ego = SimulationManager.RegisterEgo("");
var annotationDefinition = SimulationManager.RegisterAnnotationDefinition("");
var sensorHandle = SimulationManager.RegisterSensor(ego, "", "", 1, 0);
var asyncAnnotation = sensorHandle.ReportAnnotationAsync(annotationDefinition);
Assert.IsTrue(asyncAnnotation.IsPending);
asyncAnnotation.ReportValues(values);
Assert.IsFalse(asyncAnnotation.IsPending);
SimulationManager.ResetSimulation();
var capturesPath = Path.Combine(SimulationManager.OutputDirectory, "captures_000.json");
FileAssert.Exists(capturesPath);
StringAssert.Contains(expectedAnnotation, EscapeGuids(File.ReadAllText(capturesPath)));
}
[Test]
public void CreateAnnotation_MultipleTimes_WritesProperTypeOnce()
{
var annotationDefinitionGuid = new Guid(10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0);
var annotationDefinitionsJsonExpected =
$@"{{
""version"": ""{SimulationManager.SchemaVersion}"",
""annotation_definitions"": [
{{
""id"": ""{annotationDefinitionGuid}"",
""name"": ""name"",
""format"": ""json""
}}
]
}}";
var annotationDefinition1 = SimulationManager.RegisterAnnotationDefinition("name", id: annotationDefinitionGuid);
var annotationDefinition2 = SimulationManager.RegisterAnnotationDefinition("name", id: annotationDefinitionGuid);
SimulationManager.ResetSimulation();
var annotationDefinitionsPath = Path.Combine(SimulationManager.OutputDirectory, "annotation_definitions.json");
Assert.AreEqual(annotationDefinition1, annotationDefinition2);
Assert.AreEqual(annotationDefinitionGuid, annotationDefinition1.Id);
Assert.AreEqual(annotationDefinitionGuid, annotationDefinition2.Id);
AssertJsonFileEquals(annotationDefinitionsJsonExpected, annotationDefinitionsPath, false);
}
[Test]
public void CreateAnnotation_MultipleTimesWithDifferentParameters_WritesProperTypes()
{
var annotationDefinitionGuid = new Guid(10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0);
var annotationDefinitionsJsonExpected =
$@"{{
""version"": ""{SimulationManager.SchemaVersion}"",
""annotation_definitions"": [
{{
""id"": <guid>,
""name"": ""name"",
""format"": ""json""
}},
{{
""id"": <guid>,
""name"": ""name2"",
""description"": ""description"",
""format"": ""json""
}}
]
}}";
var annotationDefinition1 = SimulationManager.RegisterAnnotationDefinition("name", id: annotationDefinitionGuid);
var annotationDefinition2 = SimulationManager.RegisterAnnotationDefinition("name2", description: "description");
SimulationManager.ResetSimulation();
var annotationDefinitionsPath = Path.Combine(SimulationManager.OutputDirectory, "annotation_definitions.json");
Assert.AreEqual(annotationDefinitionGuid, annotationDefinition1.Id);
Assert.AreNotEqual(default(Guid), annotationDefinition2.Id);
AssertJsonFileEquals(annotationDefinitionsJsonExpected, annotationDefinitionsPath);
}
[Test]
public void ReportMetricValues_WhenCaptureNotExpected_Throws()
{
var ego = SimulationManager.RegisterEgo("");
var metricDefinition = SimulationManager.RegisterMetricDefinition("");
var sensorHandle = SimulationManager.RegisterSensor(ego, "", "", 1, 100);
Assert.Throws<InvalidOperationException>(() => sensorHandle.ReportMetric(metricDefinition, new int[0]));
}
[Test]
public void ReportMetricAsync_WhenCaptureNotExpected_Throws()
{
var ego = SimulationManager.RegisterEgo("");
var metricDefinition = SimulationManager.RegisterMetricDefinition("");
var sensorHandle = SimulationManager.RegisterSensor(ego, "", "", 1, 100);
Assert.Throws<InvalidOperationException>(() => sensorHandle.ReportMetricAsync(metricDefinition));
}
[Test]
public void ResetSimulation_WithUnreportedMetricAsync_LogsError()
{
var ego = SimulationManager.RegisterEgo("");
var metricDefinition = SimulationManager.RegisterMetricDefinition("");
var sensorHandle = SimulationManager.RegisterSensor(ego, "", "", 1, 0);
sensorHandle.ReportMetricAsync(metricDefinition);
SimulationManager.ResetSimulation();
LogAssert.Expect(LogType.Error, new Regex("Simulation ended with pending .*"));
}
[Test]
public void MetricAsyncIsValid_ReturnsProperValue()
{
LogAssert.ignoreFailingMessages = true; //we aren't worried about "Simulation ended with pending..."
var ego = SimulationManager.RegisterEgo("");
var metricDefinition = SimulationManager.RegisterMetricDefinition("");
var sensorHandle = SimulationManager.RegisterSensor(ego, "", "", 1, 0);
var asyncMetric = sensorHandle.ReportMetricAsync(metricDefinition);
Assert.IsTrue(asyncMetric.IsValid);
SimulationManager.ResetSimulation();
Assert.IsFalse(asyncMetric.IsValid);
}
public enum MetricTarget
{
Global,
Capture,
Annotation
}
[UnityTest]
public IEnumerator MetricReportValues_WithNoReportsInFrames_DoesNotIncrementStep()
{
var values = new[] { 1 };
var expectedLine = @"""step"": 0";
var metricDefinition = SimulationManager.RegisterMetricDefinition("");
SimulationManager.RegisterSensor(SimulationManager.RegisterEgo(""), "", "", 1, 0);
yield return null;
yield return null;
yield return null;
SimulationManager.ReportMetric(metricDefinition, values);
SimulationManager.ResetSimulation();
var text = File.ReadAllText(Path.Combine(SimulationManager.OutputDirectory, "metrics_000.json"));
StringAssert.Contains(expectedLine, text);
}
[UnityTest]
public IEnumerator SensorHandleReportMetric_BeforeReportCapture_ReportsProperJson()
{
var values = new[] { 1 };
var expectedLine = @"""step"": 0";
var metricDefinition = SimulationManager.RegisterMetricDefinition("");
var sensor = SimulationManager.RegisterSensor(SimulationManager.RegisterEgo(""), "", "", 1, 0);
yield return null;
sensor.ReportMetric(metricDefinition, values);
sensor.ReportCapture("file", new SensorSpatialData(Pose.identity, Pose.identity, null, null));
SimulationManager.ResetSimulation();
var metricsTest = File.ReadAllText(Path.Combine(SimulationManager.OutputDirectory, "metrics_000.json"));
var captures = File.ReadAllText(Path.Combine(SimulationManager.OutputDirectory, "captures_000.json"));
StringAssert.Contains(expectedLine, metricsTest);
StringAssert.Contains(expectedLine, captures);
}
[Test]
public void MetricAsyncReportValues_ReportsProperJson(
[Values(MetricTarget.Global, MetricTarget.Capture, MetricTarget.Annotation)] MetricTarget metricTarget,
[Values(true, false)] bool async,
[Values(true, false)] bool asStringJsonArray)
{
var values = new[]
{
new TestValues()
{
a = "a string",
b = 10
},
new TestValues()
{
a = "a second string",
b = 20
},
};
var expectedMetric = $@"{{
""version"": ""0.0.1"",
""metrics"": [
{{
""capture_id"": {(metricTarget == MetricTarget.Annotation || metricTarget == MetricTarget.Capture ? "<guid>" : "null")},
""annotation_id"": {(metricTarget == MetricTarget.Annotation ? "<guid>" : "null")},
""sequence_id"": <guid>,
""step"": 0,
""metric_definition"": <guid>,
""values"": [
{{
""a"": ""a string"",
""b"": 10
}},
{{
""a"": ""a second string"",
""b"": 20
}}
]
}}
]
}}";
var metricDefinition = SimulationManager.RegisterMetricDefinition("");
var sensor = SimulationManager.RegisterSensor(SimulationManager.RegisterEgo(""), "", "", 1, 0);
var annotation = sensor.ReportAnnotationFile(SimulationManager.RegisterAnnotationDefinition(""), "");
var valuesJsonArray = JArray.FromObject(values).ToString(Formatting.Indented);
if (async)
{
AsyncMetric asyncMetric;
switch (metricTarget)
{
case MetricTarget.Global:
asyncMetric = SimulationManager.ReportMetricAsync(metricDefinition);
break;
case MetricTarget.Capture:
asyncMetric = sensor.ReportMetricAsync(metricDefinition);
break;
case MetricTarget.Annotation:
asyncMetric = annotation.ReportMetricAsync(metricDefinition);
break;
default:
throw new Exception("unsupported");
}
Assert.IsTrue(asyncMetric.IsPending);
if (asStringJsonArray)
asyncMetric.ReportValues(valuesJsonArray);
else
asyncMetric.ReportValues(values);
Assert.IsFalse(asyncMetric.IsPending);
}
else
{
switch (metricTarget)
{
case MetricTarget.Global:
if (asStringJsonArray)
SimulationManager.ReportMetric(metricDefinition, valuesJsonArray);
else
SimulationManager.ReportMetric(metricDefinition, values);
break;
case MetricTarget.Capture:
if (asStringJsonArray)
sensor.ReportMetric(metricDefinition, valuesJsonArray);
else
sensor.ReportMetric(metricDefinition, values);
break;
case MetricTarget.Annotation:
if (asStringJsonArray)
annotation.ReportMetric(metricDefinition, valuesJsonArray);
else
annotation.ReportMetric(metricDefinition, values);
break;
default:
throw new Exception("unsupported");
}
}
SimulationManager.ResetSimulation();
AssertJsonFileEquals(expectedMetric, Path.Combine(SimulationManager.OutputDirectory, "metrics_000.json"), escapeGuids: true, ignoreFormatting: true);
}
[Test]
public void CreateMetric_MultipleTimesWithDifferentParameters_WritesProperTypes()
{
var metricDefinitionGuid = new Guid(10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0);
var metricDefinitionsJsonExpected =
$@"{{
""version"": ""{SimulationManager.SchemaVersion}"",
""metric_definitions"": [
{{
""id"": <guid>,
""name"": ""name""
}},
{{
""id"": <guid>,
""name"": ""name2"",
""description"": ""description""
}}
]
}}";
var metricDefinition1 = SimulationManager.RegisterMetricDefinition("name", id: metricDefinitionGuid);
var metricDefinition2 = SimulationManager.RegisterMetricDefinition("name2", description: "description");
SimulationManager.ResetSimulation();
var metricDefinitionsPath = Path.Combine(SimulationManager.OutputDirectory, "metric_definitions.json");
Assert.AreEqual(metricDefinitionGuid, metricDefinition1.Id);
Assert.AreNotEqual(default(Guid), metricDefinition2.Id);
AssertJsonFileEquals(metricDefinitionsJsonExpected, metricDefinitionsPath);
}
struct TestSpec
{
public int label_id;
public string label_name;
public int[] pixel_value;
}
public enum AdditionalInfoKind
{
Annotation,
Metric
}
[Test]
public void CreateAnnotationOrMetric_WithSpecValues_WritesProperTypes(
[Values(AdditionalInfoKind.Annotation, AdditionalInfoKind.Metric)] AdditionalInfoKind additionalInfoKind)
{
var specValues = new[]
{
new TestSpec
{
label_id = 1,
label_name = "sky",
pixel_value = new[] { 1, 2, 3}
},
new TestSpec
{
label_id = 2,
label_name = "sidewalk",
pixel_value = new[] { 4, 5, 6}
}
};
string filename;
string jsonContainerName;
if (additionalInfoKind == AdditionalInfoKind.Annotation)
{
SimulationManager.RegisterAnnotationDefinition("name", specValues);
filename = "annotation_definitions.json";
jsonContainerName = "annotation_definitions";
}
else
{
SimulationManager.RegisterMetricDefinition("name", specValues);
filename = "metric_definitions.json";
jsonContainerName = "metric_definitions";
}
var additionalInfoString = (additionalInfoKind == AdditionalInfoKind.Annotation ? @"
""format"": ""json""," : null);
var annotationDefinitionsJsonExpected =
$@"{{
""version"": ""{SimulationManager.SchemaVersion}"",
""{jsonContainerName}"": [
{{
""id"": <guid>,
""name"": ""name"",{additionalInfoString}
""spec"": [
{{
""label_id"": 1,
""label_name"": ""sky"",
""pixel_value"": [
1,
2,
3
]
}},
{{
""label_id"": 2,
""label_name"": ""sidewalk"",
""pixel_value"": [
4,
5,
6
]
}}
]
}}
]
}}";
SimulationManager.ResetSimulation();
var annotationDefinitionsPath = Path.Combine(SimulationManager.OutputDirectory, filename);
AssertJsonFileEquals(annotationDefinitionsJsonExpected, annotationDefinitionsPath);
}
static void AssertJsonFileEquals(string jsonExpected, string jsonPath, bool escapeGuids = true, bool ignoreFormatting = false)
{
FileAssert.Exists(jsonPath);
var jsonActual = File.ReadAllText(jsonPath);
if (escapeGuids)
jsonActual = EscapeGuids(jsonActual);
if (ignoreFormatting)
{
jsonActual = Regex.Replace(jsonActual, "^\\s*", "", RegexOptions.Multiline);
jsonExpected = Regex.Replace(jsonExpected, "^\\s*", "", RegexOptions.Multiline);
}
Assert.AreEqual(jsonExpected, jsonActual, $"Expected:\n{jsonExpected}\nActual:\n{jsonActual}");
}
static string EscapeGuids(string text)
{
var result = Regex.Replace(text, @"""[a-z0-9]*-[a-z0-9]*-[a-z0-9]*-[a-z0-9]*-[a-z0-9]*""", "<guid>");
return result;
}
}
}

3
com.unity.perception/Tests/Runtime/GroundTruthTests/SimulationManagerTests.cs.meta


fileFormatVersion: 2
guid: 355a38a8edc948d5ac8649b67a976dd5
timeCreated: 1577122007

3
com.unity.perception/Tests/Runtime/GroundTruthTests/SimulationManagerSensorSchedulingTests.cs.meta


fileFormatVersion: 2
guid: 25a1ad2bc73f47bcbe7138c54bad45e5
timeCreated: 1577125461

176
com.unity.perception/Tests/Runtime/GroundTruthTests/SimulationManagerSensorSchedulingTests.cs


using System.Collections;
using System.Text.RegularExpressions;
using NUnit.Framework;
using UnityEngine;
using UnityEngine.Perception.GroundTruth;
using UnityEngine.TestTools;
namespace GroundTruthTests
{
[TestFixture]
public class SimulationManagerSensorSchedulingTests
{
[TearDown]
public void TearDown()
{
Time.timeScale = 1;
SimulationManager.ResetSimulation();
}
[UnityTest]
public IEnumerator FramesScheduledBySensorConfig()
{
var ego = SimulationManager.RegisterEgo("ego");
var firstCaptureTime = 1.5f;
var period = .4f;
SimulationManager.RegisterSensor(ego, "cam", "", period, firstCaptureTime);
float[] deltaTimeSamplesExpected =
{
firstCaptureTime,
period,
period,
period
};
float[] deltaTimeSamples = new float[deltaTimeSamplesExpected.Length];
for (int i = 0; i < deltaTimeSamples.Length; i++)
{
yield return null;
Assert.AreEqual(deltaTimeSamplesExpected[i], Time.deltaTime, 0.0001f);
}
}
[UnityTest]
public IEnumerator FramesScheduled_WithTimeScale_ResultsInProperDeltaTime()
{
var ego = SimulationManager.RegisterEgo("ego");
var firstCaptureTime = 2f;
var period = 1f;
var timeScale = 2;
Time.timeScale = timeScale;
SimulationManager.RegisterSensor(ego, "cam", "", period, firstCaptureTime);
float[] deltaTimeSamplesExpected =
{
timeScale * firstCaptureTime,
timeScale * period,
timeScale * period,
timeScale * period
};
float[] deltaTimeSamples = new float[deltaTimeSamplesExpected.Length];
for (int i = 0; i < deltaTimeSamples.Length; i++)
{
yield return null;
Assert.AreEqual(deltaTimeSamplesExpected[i], Time.deltaTime, 0.0001f);
}
}
[UnityTest]
public IEnumerator ChangingTimeScale_CausesDebugError()
{
var ego = SimulationManager.RegisterEgo("ego");
SimulationManager.RegisterSensor(ego, "cam", "", 1f, 2f);
yield return null;
Time.timeScale = 5;
yield return null;
LogAssert.Expect(LogType.Error, new Regex("Time\\.timeScale may not change mid-sequence.*"));
}
[UnityTest]
public IEnumerator ChangingTimeScale_DuringStartNewSequence_Succeeds()
{
var ego = SimulationManager.RegisterEgo("ego");
SimulationManager.RegisterSensor(ego, "cam", "", 1f, 2f);
yield return null;
Time.timeScale = 1;
SimulationManager.StartNewSequence();
yield return null;
}
[Ignore("Changing timeScale mid-sequence is not supported")]
[UnityTest]
public IEnumerator FramesScheduled_WithChangingTimeScale_ResultsInProperDeltaTime()
{
var ego = SimulationManager.RegisterEgo("ego");
var firstCaptureTime = 2f;
var period = 1f;
float[] newTimeScalesPerFrame =
{
2f,
10f,
.01f,
1f
};
SimulationManager.RegisterSensor(ego, "cam", "", period, firstCaptureTime);
float[] deltaTimeSamplesExpected =
{
newTimeScalesPerFrame[0] * firstCaptureTime,
newTimeScalesPerFrame[1] * period,
newTimeScalesPerFrame[2] * period,
newTimeScalesPerFrame[3] * period
};
float[] deltaTimeSamples = new float[deltaTimeSamplesExpected.Length];
for (int i = 0; i < deltaTimeSamples.Length; i++)
{
Time.timeScale = newTimeScalesPerFrame[i];
yield return null;
Assert.AreEqual(deltaTimeSamplesExpected[i], Time.deltaTime, 0.0001f);
}
}
[UnityTest]
public IEnumerator ResetSimulation_ResetsCaptureDeltaTime()
{
var ego = SimulationManager.RegisterEgo("ego");
SimulationManager.RegisterSensor(ego, "cam", "", 4, 10);
yield return null;
Assert.AreEqual(10, Time.captureDeltaTime);
SimulationManager.ResetSimulation();
Assert.AreEqual(0, Time.captureDeltaTime);
}
[UnityTest]
public IEnumerator ShouldCaptureThisFrame_ReturnsTrueOnProperFrames()
{
var ego = SimulationManager.RegisterEgo("ego");
var firstCaptureTime1 = 10;
var frequencyInMs1 = 4;
var sensor1 = SimulationManager.RegisterSensor(ego, "cam", "1", frequencyInMs1, firstCaptureTime1);
var firstCaptureTime2 = 10;
var frequencyInMs2 = 6;
var sensor2 = SimulationManager.RegisterSensor(ego, "cam", "2", frequencyInMs2, firstCaptureTime2);
var sensor3 = SimulationManager.RegisterSensor(ego, "cam", "3", 1, 1);
sensor3.Enabled = false;
(float deltaTime, bool sensor1ShouldCapture, bool sensor2ShouldCapture)[] samplesExpected =
{
((float)firstCaptureTime1, true, true),
(4, true, false),
(2, false, true),
(2, true, false),
(4, true, true)
};
var samplesActual = new(float deltaTime, bool sensor1ShouldCapture, bool sensor2ShouldCapture)[samplesExpected.Length];
for (int i = 0; i < samplesActual.Length; i++)
{
yield return null;
samplesActual[i] = (Time.deltaTime, sensor1.ShouldCaptureThisFrame, sensor2.ShouldCaptureThisFrame);
}
CollectionAssert.AreEqual(samplesExpected, samplesActual);
}
[Test]
public void Enabled_StartsTrue()
{
var sensor1 = SimulationManager.RegisterSensor(SimulationManager.RegisterEgo(""), "cam", "1", 1, 1);
Assert.IsTrue(sensor1.Enabled);
}
}
}

/com.unity.perception/Documentation~/SimulationManager.md → /com.unity.perception/Documentation~/DatasetCapture.md

/com.unity.perception/Runtime/GroundTruth/SimulationManager.cs.meta → /com.unity.perception/Runtime/GroundTruth/DatasetCapture.cs.meta

/com.unity.perception/Runtime/GroundTruth/SimulationManager.cs → /com.unity.perception/Runtime/GroundTruth/DatasetCapture.cs

正在加载...
取消
保存