Labeler docs (#30)

* Initial docs for new labelers. Updated changelog.
* Fixing IdLabelConfig import issues
* Adding support for rendering to a user-supplied render texture for semantic segmentation.

TestProjects/PerceptionURP/Assets/IdLabelConfig.asset

   m_GameObject: {fileID: 0}
   m_Enabled: 1
   m_EditorHideFlags: 0
-  m_Script: {fileID: 0}
+  m_Script: {fileID: 11500000, guid: 2f09f279848e42cea259348b13bce4c5, type: 3}
-  m_EditorClassIdentifier: Unity.Perception.Runtime:UnityEngine.Perception.GroundTruth:IdLabelConfig
+  m_EditorClassIdentifier: 
   m_LabelEntries:
   - label: Box
     id: 1

TestProjects/PerceptionURP/Assets/Scenes/SampleScene.unity

   m_BounceIntensity: 1
   m_ColorTemperature: 6570
   m_UseColorTemperature: 0
-  m_BoundingSphereOverride: {x: 1e-45, y: 9e-44, z: 9e-44, w: 0}
+  m_BoundingSphereOverride: {x: 0, y: 1e-45, z: 1.8643e-41, w: 5.9e-44}
   m_UseBoundingSphereOverride: 0
   m_ShadowRadius: 0
   m_ShadowAngle: 0
       data:
         enabled: 1
         objectCountMetricId: 51DA3C27-369D-4929-AEA6-D01614635CE2
-        m_IdLabelConfig: {fileID: 11400000, guid: cedcacfb1d9beb34fbbb231166c472fe,
+        m_LabelConfig: {fileID: 11400000, guid: cedcacfb1d9beb34fbbb231166c472fe,
           type: 2}
     00000001:
       type: {class: BoundingBox2DLabeler, ns: UnityEngine.Perception.GroundTruth,
         enabled: 1
         annotationId: 12F94D8D-5425-4DEB-9B21-5E53AD957D66
         labelConfig: {fileID: 11400000, guid: c140c5aa05dd09e4fadaa26de31b1f39, type: 2}
+        m_TargetTextureOverride: {fileID: 0}
 --- !u!114 &963194231
 MonoBehaviour:
   m_ObjectHideFlags: 0

com.unity.perception/CHANGELOG.md

 
 ## [Unreleased] 
 ### Added
-
+Added CameraLabeler, an extensible base type for all forms of dataset output from a camera.
+Added LabelConfig\<T\>, a base class for mapping labels to data used by a labeler. There are two new derived types - ID label config and semantic segmentation label config.
-
+Moved the various forms of ground truth from PerceptionCamera into various subclasses of CameraLabeler.
+Renamed SimulationManager to DatasetCapture.
+Changed Semantic Segmentation to take a SemanticSegmentationLabelConfig, which maps labels to color pixel values.
 ### Deprecated
 
 ### Removed

com.unity.perception/Documentation~/GroundTruth-Labeling.md

 # Labeling
-Accurately labeling assets with a predefined taxonomy will inform training and testing of algorithms as to which objects in a dataset have importance. Example: assets labeled with “table” and “chair” will provide an algorithm with the information it needs to train on identifying these objects separately within a scene.
+Many labelers require mapping the objects in the view to the values recorded in the dataset. As an example, Semantic Segmentation needs to determine the color to draw each object in the segmentation image.
-You can add a Labeling component to individual GameObjects within a scene although it is a good practice to create a prefab of a GameModel and apply the Labeling component to it.
+This mapping is accomplished for a GameObject by:
+* Finding the nearest Labeling component attached to the object or its ancestors.
+* Find the first label in the Labeling which is present anywhere in the Labeler's Label Config.
+* The resolved Label Entry from the Label Config is used to produce the final output.
-Multiple labels can be assigned to the same `Labeling`. When ground truth which requires unique labels per object is being generated, the first label in the `Labeling` present anywhere in the `LabelingConfiguration` is used.
+## Labeling component
+The `Labeling` component associates a list of string-based labels with a GameObject and its descendants. A `Labeling` component on a descendant overrides its parent's labels.
-## Labeling Configuration
-Many labelers require require a `Labeling Configuration` asset.
-This file specifies a list of all labels to be captured in the dataset for a labeler along with extra information used by the various labelers.
+## Label Config
+Many labelers require require a `Label Config` asset. This asset specifies a list of all labels to be captured in the dataset along with extra information used by the various labelers.
-For example
-An asset representing a box of Rice Krispies cereal could be labeled as `food\cereal\kellogs\ricekrispies`
+For example, an asset representing a box of Rice Krispies cereal could be labeled as `food\cereal\kellogs\ricekrispies`
 
 * “food” - type
 * “cereal” - subtype

com.unity.perception/Documentation~/PerceptionCamera.md

 # The Perception Camera component
 The Perception Camera component ensures the attached [Camera](https://docs.unity3d.com/Manual/class-Camera.html) runs at deterministic rates and captures RGB and other Camera-related ground truth to the [JSON dataset](Schema/Synthetic_Dataset_Schema.md) using [DatasetCapture](DatasetCapture.md). It supports HDRP and URP.
 
-<img src="images/PerceptionCamera.PNG" align="middle"/>
+<img src="images/PerceptionCamera.png" align="middle"/>
 
 ## Properties
 | Property: | Function: |
 | Start Time | The simulation time at which to run the first frame. This time will offset the period, useful for allowing multiple cameras to run at the right times relative to each other. |
 | Capture Rgb Images | When this is checked, RGB images will be captured as PNG files in the dataset each frame. |
-| Produce Segmentation Images| When this is checked at startup, semantic segmentation images will be captured as PNG files in the dataset each frame. Pixel colors are derived from the [Labeling](GroundTruth-Labeling.md) components attached to the GameObjects and the Labeling Configuration specified on the Perception Camera. |
-|Produce Object Count Annotations| When this is checked at startup, a metric reporting object counts for each label in the Labeling Configuration will be added to the dataset each frame. |
-|Object Count Id| The ID to use in the dataset for the Object Count metric. |
-|Produce Bounding Box Annotations| When this is checked at startup, an annotation reporting 2D bounding boxes of each labeled GameObject will be added to the dataset each frame. |
-|Bounding Box Id| The ID to use in the dataset for the Bounding Box annotation.|
-|Produce Rendered Object Info Metric|When this is checked at startup, a metric reporting visible pixels, instance ID and resolved label will be reported for each object each frame. Labels are resolved using the Labeling Configuration. |
-|Rendered Object Info Id| The ID to use in the dataset for the Rendered Object Info annotation.|
-|Bounding Box Origin|Whether the X/Y values reported for bounding boxes should start at the bottom-left or top-left of the object in the frame. |
-|Labeling Configuration| The Labeling Configuration used to resolve each object to a single label, label ID, or semantic segmentation pixel value in the ground truth generated by this Perception Camera.|
+| Camera Labelers | A list of labelers which generate data derived from this camera. |
+
+## Camera Labelers
+Camera Labelers capture data related to the camera into the JSON dataset. This data can be used for model training or for dataset statistics. A number of Camera Labelers are provided with Perception, and additional labelers can be defined by deriving from the CameraLabeler class.
+
+### Semantic Segmentation Labeler
+<img src="images/semantic_segmentation.png" align="middle"/>
+
+_Example semantic segmentation image from a modified [SynthDet](https://github.com/Unity-Technologies/SynthDet) project_
+
+The Semantic Segmentation Labeler generates a 2D RGB image using the attached camera where objects are drawn with the color associated with their label in the provided SemanticSegmentationLabelConfig. If no label is resolved for an object, it is drawn black.
+
+### Bounding Box 2D Labeler
+<img src="images/bounding_boxes.png" align="middle"/>
+
+_example bounding box visualization from [SynthDet](https://github.com/Unity-Technologies/SynthDet) generated by the `SynthDet_Statistics` jupyter notebook_
+
+The Bounding Box 2D Labeler produces 2D bounding boxes for each visible object with a label resolved by the given ID Label Config. Bounding boxes are calculated using the rendered image, so only occluded or out-of-frame portions of the objects are not included.
+
+### Object Count Labeler
+```
+{
+    "label_id": 25,
+    "label_name": "drink_whippingcream_lucerne",
+    "count": 1
+}
+```
+_Example object count for a single label_
+
+The Object Count Labeler records object counts for each label in the provided ID Label Config. Only objects with at least one visible pixel in the camera frame will be recorded.
+
+### Rendered Object Info Labeler
+```
+{
+    "label_id": 24,
+    "instance_id": 320,
+    "visible_pixels": 28957
+}
+```
+_Example rendered object info for a single object_
+
+The Rendered Object Info Labeler records a list of all objects visible in the camera image, including its instance id, resolved label id and visible pixels. Objects not resolved to a label in the given ID Label Config are not recorded.

com.unity.perception/Documentation~/index.md

 |Feature|Description
 |---|---|
 |[Labeling](GroundTruth-Labeling.md)|Component which marks a GameObject and its descendants with a set of labels|
-|[Labeling Configuration](GroundTruth-Labeling.md#LabelingConfiguration)|Asset which defines a taxonomy of labels for ground truth generation|
+|[LabelConfig](GroundTruth-Labeling.md#LabelConfig)|Asset which defines a taxonomy of labels for ground truth generation|
 |[Perception Camera](PerceptionCamera.md)|Captures RGB images and ground truth from a [Camera](https://docs.unity3d.com/Manual/class-Camera.html)|
 |[DatasetCapture](DatasetCapture.md)|Ensures sensors are triggered at proper rates and accepts data for the JSON dataset|

com.unity.perception/Editor/GroundTruth/PerceptionCameraEditor.cs

                 var localOption = option;
                 menu.AddItem(new GUIContent(option.Name),
                     false,
-                    () => AddLabeler(labelers, localOption));
+                    () => AddLabeler(localOption));
-        void AddLabeler(SerializedProperty labelers, Type labelerType)
+        void AddLabeler(Type labelerType)
-            // var insertIndex = labelers.arraySize;
-            // labelers.InsertArrayElementAtIndex(insertIndex);
-            // serializedObject.ApplyModifiedProperties();
-            // var element = labelers.GetArrayElementAtIndex(insertIndex);
-            // element.managedReferenceValue = labeler;
             serializedObject.ApplyModifiedProperties();
         }
 

com.unity.perception/Runtime/GroundTruth/Labelers/SemanticSegmentationLabeler.cs

         /// </summary>
         public event Action<ImageReadbackEventArgs> imageReadback;
 
-        [NonSerialized]
-        internal RenderTexture semanticSegmentationTexture;
+        /// <summary>
+        /// The RenderTexture on which semantic segmentation images are drawn. Will be resized on startup to match
+        /// the camera resolution.
+        /// </summary>
+        public RenderTexture targetTexture => m_TargetTextureOverride;
+        [Tooltip("(Optional) The RenderTexture on which semantic segmentation images will be drawn. Will be reformatted on startup.")]
+        [SerializeField]
+        RenderTexture m_TargetTextureOverride;
 
         AnnotationDefinition m_SemanticSegmentationAnnotationDefinition;
         RenderTextureReader<Color32> m_SemanticSegmentationTextureReader;
         /// Creates a new SemanticSegmentationLabeler with the given <see cref="SemanticSegmentationLabelConfig"/>.
         /// </summary>
         /// <param name="labelConfig">The label config associating labels with colors.</param>
-        public SemanticSegmentationLabeler(SemanticSegmentationLabelConfig labelConfig)
+        /// <param name="targetTextureOverride">Override the target texture of the labeler. Will be reformatted on startup.</param>
+        public SemanticSegmentationLabeler(SemanticSegmentationLabelConfig labelConfig, RenderTexture targetTextureOverride = null)
+            this.m_TargetTextureOverride = targetTextureOverride;
         }
 
         [SuppressMessage("ReSharper", "InconsistentNaming")]
 
             m_AsyncAnnotations = new Dictionary<int, AsyncAnnotation>();
 
-            semanticSegmentationTexture = new RenderTexture(
-                new RenderTextureDescriptor(width, height, GraphicsFormat.R8G8B8A8_UNorm, 8));
-            semanticSegmentationTexture.name = "Labeling";
+            var renderTextureDescriptor = new RenderTextureDescriptor(width, height, GraphicsFormat.R8G8B8A8_UNorm, 8);
+            if (targetTexture != null)
+                targetTexture.descriptor = renderTextureDescriptor;
+            else
+                m_TargetTextureOverride = new RenderTexture(renderTextureDescriptor);
+
+            targetTexture.Create();
+            targetTexture.name = "Labeling";
 
 #if HDRP_PRESENT
             var gameObject = perceptionCamera.gameObject;
-            m_SemanticSegmentationPass = new SemanticSegmentationPass(myCamera, semanticSegmentationTexture, labelConfig)
+            m_SemanticSegmentationPass = new SemanticSegmentationPass(myCamera, targetTexture, labelConfig)
             {
                 name = "Labeling Pass"
             };
-            perceptionCamera.AddScriptableRenderPass(new SemanticSegmentationUrpPass(myCamera, semanticSegmentationTexture, labelConfig));
+            perceptionCamera.AddScriptableRenderPass(new SemanticSegmentationUrpPass(myCamera, targetTexture, labelConfig));
 #endif
 
             var specs = labelConfig.labelEntries.Select((l) => new SemanticSegmentationSpec()
                 "PNG",
                 id: Guid.Parse(annotationId));
 
-            m_SemanticSegmentationTextureReader = new RenderTextureReader<Color32>(semanticSegmentationTexture, myCamera,
+            m_SemanticSegmentationTextureReader = new RenderTextureReader<Color32>(targetTexture, myCamera,
                 (frameCount, data, tex) => OnSemanticSegmentationImageRead(frameCount, data));
         }
 
             {
                 data = data,
                 frameCount = frameCount,
-                sourceTexture = semanticSegmentationTexture
+                sourceTexture = targetTexture
-                width = semanticSegmentationTexture.width,
-                height = semanticSegmentationTexture.height,
+                width = targetTexture.width,
+                height = targetTexture.height,
                 path = localPath
             };
             asyncRequest.Start((r) =>
             m_SemanticSegmentationTextureReader?.Dispose();
             m_SemanticSegmentationTextureReader = null;
 
-            if (semanticSegmentationTexture != null)
-                semanticSegmentationTexture.Release();
+            if (m_TargetTextureOverride != null)
+                m_TargetTextureOverride.Release();
-            semanticSegmentationTexture = null;
+            m_TargetTextureOverride = null;
         }
     }
 }

com.unity.perception/Runtime/GroundTruth/Labeling/IdLabelConfig.cs

 
 namespace UnityEngine.Perception.GroundTruth {
     /// <summary>
-    /// An entry for <see cref="IdLabelConfig"/> mapping a label to an integer id.
-    /// </summary>
-    [Serializable]
-    public struct IdLabelEntry : ILabelEntry
-    {
-        string ILabelEntry.label => this.label;
-        /// <summary>
-        /// The label string to associate with the id.
-        /// </summary>
-        public string label;
-        /// <summary>
-        /// The id to associate with the label.
-        /// </summary>
-        public int id;
-    }
-    /// <summary>
-    public sealed class IdLabelConfig : LabelConfig<IdLabelEntry>
+    public class IdLabelConfig : LabelConfig<IdLabelEntry>
     {
         /// <summary>
         /// Whether the inspector will auto-assign ids based on the id of the first element.

com.unity.perception/Runtime/GroundTruth/Labeling/SemanticSegmentationLabelConfig.cs

 
 namespace UnityEngine.Perception.GroundTruth {
     /// <summary>
-    /// A definition for how a <see cref="Labeling"/> should be resolved to a single label and id for ground truth generation.
+    /// A definition for how a <see cref="Labeling"/> should be resolved to a single label and color for semantic segmentation generation.
-    [CreateAssetMenu(fileName = "SemanticSegmentationLabelgConfig", menuName = "Perception/Semantic Segmentation Label Config", order = 1)]
+    [CreateAssetMenu(fileName = "SemanticSegmentationLabelConfig", menuName = "Perception/Semantic Segmentation Label Config", order = 1)]
     public class SemanticSegmentationLabelConfig : LabelConfig<SemanticSegmentationLabelEntry>
     {
     }

com.unity.perception/Runtime/GroundTruth/Resources/SemanticSegmentation.shader

 {
     Properties
     {
-        [PerObjectData] LabelingId("Labeling Id", Color) = (1,1,1,1)
+        [PerObjectData] LabelingId("Labeling Id", Color) = (0,0,0,1)
     }
 
     HLSLINCLUDE

com.unity.perception/Runtime/GroundTruth/SemanticSegmentationCrossPipelinePass.cs

 
             s_LastFrameExecuted = Time.frameCount;
             var renderList = CreateRendererListDesc(camera, cullingResult, "FirstPass", 0, m_OverrideMaterial, -1);
-            cmd.ClearRenderTarget(true, true, Color.clear);
+            cmd.ClearRenderTarget(true, true, Color.black);
             DrawRendererList(renderContext, cmd, RendererList.Create(renderList));
         }
 
+            bool found = false;
+                    found = true;
-            mpb.SetColor(k_LabelingId, entry.color);
+            if (found)
+                mpb.SetColor(k_LabelingId, entry.color);
         }
     }
 }

com.unity.perception/Tests/Runtime/GroundTruthTests/GroundTruthTestBase.cs

 {
     public class GroundTruthTestBase
     {
-        List<GameObject> m_ObjectsToDestroy = new List<GameObject>();
+        List<Object> m_ObjectsToDestroy = new List<Object>();
         [TearDown]
         public void TearDown()
         {
                 Directory.Delete(DatasetCapture.OutputDirectory, true);
         }
 
-        public void AddTestObjectForCleanup(GameObject @object) => m_ObjectsToDestroy.Add(@object);
+        public void AddTestObjectForCleanup(Object @object) => m_ObjectsToDestroy.Add(@object);
-        public void DestroyTestObject(GameObject @object)
+        public void DestroyTestObject(Object @object)
         {
             Object.DestroyImmediate(@object);
             m_ObjectsToDestroy.Remove(@object);

com.unity.perception/Tests/Runtime/GroundTruthTests/SegmentationGroundTruthTests.cs

 using NUnit.Framework;
 using Unity.Collections;
 using UnityEngine;
+using UnityEngine.Experimental.Rendering;
+using UnityEngine.Rendering;
+using UnityEngine.UIElements;
 using Object = UnityEngine.Object;
 
 namespace GroundTruthTests
             DestroyTestObject(planeObject);
 
             Assert.AreEqual(4, timesSegmentationImageReceived);
+        }
+
+        [UnityTest]
+        public IEnumerator SemanticSegmentationPass_WithLabeledButNotMatchingObject_ProducesBlack()
+        {
+            int timesSegmentationImageReceived = 0;
+            var expectedPixelValue = new Color32(0, 0, 0, 255);
+            void OnSegmentationImageReceived(NativeArray<Color32> data)
+            {
+                timesSegmentationImageReceived++;
+                CollectionAssert.AreEqual(Enumerable.Repeat(expectedPixelValue, data.Length), data);
+            }
+
+            var cameraObject = SetupCameraSemanticSegmentation(a => OnSegmentationImageReceived(a.data));
+
+            AddTestObjectForCleanup(TestHelper.CreateLabeledPlane(label: "non-matching"));
+            yield return null;
+            //destroy the object to force all pending segmented image readbacks to finish and events to be fired.
+            DestroyTestObject(cameraObject);
+            Assert.AreEqual(1, timesSegmentationImageReceived);
+        }
+
+        [UnityTest]
+        public IEnumerator SemanticSegmentationPass_WithEmptyFrame_ProducesBlack()
+        {
+            int timesSegmentationImageReceived = 0;
+            var expectedPixelValue = new Color32(0, 0, 0, 255);
+            void OnSegmentationImageReceived(NativeArray<Color32> data)
+            {
+                timesSegmentationImageReceived++;
+                CollectionAssert.AreEqual(Enumerable.Repeat(expectedPixelValue, data.Length), data);
+            }
+
+            var cameraObject = SetupCameraSemanticSegmentation(a => OnSegmentationImageReceived(a.data));
+
+            yield return null;
+            var segLabeler = (SemanticSegmentationLabeler)cameraObject.GetComponent<PerceptionCamera>().labelers[0];
+            var request = AsyncGPUReadback.Request(segLabeler.targetTexture, callback: r =>
+            {
+                CollectionAssert.AreEqual(Enumerable.Repeat(expectedPixelValue, segLabeler.targetTexture.width * segLabeler.targetTexture.height), r.GetData<Color32>());
+            });
+            AsyncGPUReadback.WaitAllRequests();
+            //request.WaitForCompletion();
+            Assert.IsTrue(request.done);
+            Assert.IsFalse(request.hasError);
+
+            //destroy the object to force all pending segmented image readbacks to finish and events to be fired.
+            DestroyTestObject(cameraObject);
+            Assert.AreEqual(1, timesSegmentationImageReceived);
+        }
+
+        [UnityTest]
+        public IEnumerator SemanticSegmentationPass_WithTextureOverride_RendersToOverride()
+        {
+            int timesSegmentationImageReceived = 0;
+            var expectedPixelValue = new Color32(0, 0, 255, 255);
+            var targetTextureOverride = new RenderTexture(2, 2, 1, RenderTextureFormat.R8);
+
+            var cameraObject = SetupCamera(out var perceptionCamera);
+            var labelConfig = ScriptableObject.CreateInstance<SemanticSegmentationLabelConfig>();
+            labelConfig.Init(new List<SemanticSegmentationLabelEntry>()
+            {
+                new SemanticSegmentationLabelEntry()
+                {
+                    label = "label",
+                    color = expectedPixelValue
+                }
+            });
+            var semanticSegmentationLabeler = new SemanticSegmentationLabeler(labelConfig, targetTextureOverride);
+            perceptionCamera.AddLabeler(semanticSegmentationLabeler);
+            cameraObject.SetActive(true);
+            AddTestObjectForCleanup(cameraObject);
+            AddTestObjectForCleanup(TestHelper.CreateLabeledPlane());
+
+            yield return null;
+            //NativeArray<Color32> readbackArray = new NativeArray<Color32>(targetTextureOverride.width * targetTextureOverride.height, Allocator.Temp);
+            var request = AsyncGPUReadback.Request(targetTextureOverride, callback: r =>
+            {
+                CollectionAssert.AreEqual(Enumerable.Repeat(expectedPixelValue, targetTextureOverride.width * targetTextureOverride.height), r.GetData<Color32>());
+            });
+            AsyncGPUReadback.WaitAllRequests();
+            //request.WaitForCompletion();
+            Assert.IsTrue(request.done);
+            Assert.IsFalse(request.hasError);
         }
 
         [UnityTest]

com.unity.perception/Runtime/GroundTruth/Labeling/IdLabelEntry.cs

+using System;
+
+namespace UnityEngine.Perception.GroundTruth {
+    /// <summary>
+    /// An entry for <see cref="IdLabelConfig"/> mapping a label to an integer id.
+    /// </summary>
+    [Serializable]
+    public struct IdLabelEntry : ILabelEntry
+    {
+        string ILabelEntry.label => this.label;
+        /// <summary>
+        /// The label string to associate with the id.
+        /// </summary>
+        public string label;
+        /// <summary>
+        /// The id to associate with the label.
+        /// </summary>
+        public int id;
+    }
+}

com.unity.perception/Runtime/GroundTruth/Labeling/IdLabelEntry.cs.meta

+fileFormatVersion: 2
+guid: ecabcfb188f0406bb1ac9b222c8bd915
+timeCreated: 1593702144