Fix keypoint issues (#259)

* Fixing two bugs in KeypointLabeler:
AISV-1254 - Some keypoints fall outside of the image bounds
AISV-1257 - Partially visible characters with no visible keypoints are not reported

* Adding JointLabelEditor  with message about Labeling requirement

* Reorganizing keypoint docs and elaborating on setup requirements.

* Updating PerceptionCamera.md

* Fixing issues on 2020.2 and failing test

* Add object filtering modes to keypoint labeler

* Updating docs for object filtering

* Adding link to tutorial

* Moving KeypointLabeler.md to proper directory

* Update TUTORIAL.md

* Updating capitalization

Co-authored-by: Mohsen K <mohsen.kamalzadeh@unity3d.com>

* Apply doc suggestions

Co-authored-by: Mohsen K <mohsen.kamalzadeh@unity3d.com>

* Update PerceptionCamera.md based on feedback

* Update CHANGELOG.md

Co-authored-by: Mohsen K <mohsen.kamalzadeh@unity3d.com>

com.unity.perception/CHANGELOG.md

 ### Added
 
 ### Changed
+Expanded documentation on the Keypoint Labeler
+Updated Keypoint Labeler logic to only report keypoints for visible objects by default
 
 ### Deprecated
 

com.unity.perception/Documentation~/HPTutorial/TUTORIAL.md

 }
 ```
 
-In the above annotation, all of the 18 joints defined in the COCO template we used are listed. For each joint that is present in our character, you can see the X and Y coordinates within the captured frame. However, you may notice three of the joints are listed with (0,0) coordinates. These joints are not present in our character. A fact that is also denoted by the `state` field. A state of **0** means the joint was not present, **1** denotes a joint that is present but not visible (to be implemented in a later version of the package), and **2** means the joint was present and visible.
+In the above annotation, all of the 18 joints defined in the COCO template we used are listed. For each joint that is present in our character, you can see the X and Y coordinates within the captured frame. However, you may notice three of the joints are listed with (0,0) coordinates. These joints are not present in our character. A fact that is also denoted by the `state` field. A state of **0** means the joint either does not exist or is outside of the image's bounds, **1** denotes a joint that is inside of the image but cannot be seen because the part of the object it belongs to is not visible in the image, and **2** means the joint was present and visible.
 
 You may also note that the `pose` field has a value of `unset`. This is because we have not defined poses for our animation clip and `Perception Camera` yet. We will do this next.
 

com.unity.perception/Documentation~/PerceptionCamera.md

 ```
 _Example rendered object info for a single object_
 
-The RenderedObjectInfoLabeler records a list of all objects visible in the Camera image, including its instance ID, resolved label ID and visible pixels. If Unity cannot resolve objects to a label in the IdLabelConfig, it does not record these objects.
-
-### KeypointLabeler
-
-The keypoint labeler captures keypoints of a labeled gameobject. The typical use of this labeler is capturing human pose estimation data. The labeler uses a [keypoint template](#KeypointTemplate) which defines the keypoints to capture for the model and the skeletal connections between those keypoints. The positions of the keypoints are recorded in pixel coordinates. Each keypoint has a state value: 0 - the keypoint either does not exist or is outside of the image's bounds, 1 - the keypoint exists and is inside of the image's bounds but cannot be seen because it is occluded by another object, and 2 - the keypoint exists and is visible.
-
-```
-keypoints {
-  label_id:      <int>   -- Integer identifier of the label
-  instance_id:   <str>   -- UUID of the instance.
-  template_guid: <str>   -- UUID of the keypoint template
-  pose:          <str>   -- Pose ground truth information
-  keypoints [            -- Array of keypoint data, one entry for each keypoint defined in associated template file.
-    {
-      index:     <int>   -- Index of keypoint in template
-      x:         <float> -- X pixel coordinate of keypoint
-      y:         <float> -- Y pixel coordinate of keypoint
-      state:     <int>   -- 0: keypoint does not exist, 1: keypoint exists but is not visible, 2: keypoint exists and is visible
-    }, ...
-  ]
-}
-```
-
-#### Keypoint Template
-
-keypoint templates are used to define the keypoints and skeletal connections captured by the KeypointLabeler. The keypoint
-template takes advantage of Unity's humanoid animation rig, and allows the user to automatically associate template keypoints
-to animation rig joints. Additionally, the user can choose to ignore the rigged points, or add points not defined in the rig.
-A Coco keypoint template is included in the perception package.
-
-##### Editor
-
-The keypoint template editor allows the user to create/modify a keypoint template. The editor consists of the header information,
-the keypoint array, and the skeleton array.
-
-![Header section of the keypoint template](images/keypoint_template_header.png)
-<br/>_Header section of the keypoint template_
-
-In the header section, a user can change the name of the template and supply textures that they would like to use for the keypoint
-visualization.
-
-![The keypoint section of the keypoint template](images/keypoint_template_keypoints.png)
-<br/>_Keypoint section of the keypoint template_
-
-The keypoint section allows the user to create/edit keypoints and associate them with Unity animation rig points. Each keypoint record
-has 4 fields: label (the name of the keypoint), Associate to Rig (a boolean value which, if true, automatically maps the keypoint to
-the gameobject defined by the rig), Rig Label (only needed if Associate To Rig is true, defines which rig component to associate with
-the keypoint), and Color (RGB color value of the keypoint in the visualization).
+The `RenderedObjectInfoLabeler` records a list of all objects visible in the camera image, including their instance IDs, resolved label IDs, and visible pixel counts. If Unity cannot resolve objects to a label in the `IdLabelConfig`, it does not record these objects.
-![Skeleton section of the keypoint template](images/keypoint_template_skeleton.png)
-<br/>_Skeleton section of the keypoint template_
+### Keypoint Labeler
-The skeleton section allows the user to create connections between joints, basically defining the skeleton of a labeled object.
+The keypoint labeler captures the screen locations of specific points on labeled GameObjects. The typical use of this Labeler is capturing human pose estimation data, but it can be used to capture points on any kind of object. The Labeler uses a [Keypoint Template](#KeypointTemplate) which defines the keypoints to capture for the model and the skeletal connections between those keypoints. The positions of the keypoints are recorded in pixel coordinates.
-##### Format
-```
-annotation_definition.spec {
-  template_id:       <str>           -- The UUID of the template
-  template_name:     <str>           -- Human readable name of the template
-  key_points [                       -- Array of joints defined in this template
-    {
-      label:         <str>           -- The label of the joint
-      index:         <int>           -- The index of the joint
-    }, ...
-  ]
-  skeleton [                         -- Array of skeletal connections (which joints have connections between one another) defined in this template
-    {
-      joint1:        <int>           -- The first joint of the connection
-      joint2:        <int>           -- The second joint of the connection
-    }, ...
-  ]
-}
-```
-
-#### Animation Pose Label
-
-This file is used to define timestamps in an animation to a pose label.
+For more information, see [Keypoint Labeler](GroundTruth/KeypointLabeler.md) or the [Human Pose Labeling and Randomization Tutorial](HPTutorial/TUTORIAL.md)
 
 ## Limitations
 

com.unity.perception/Runtime/GroundTruth/Labelers/KeypointLabeler.cs

 using System.Collections.Generic;
 using System.Linq;
 using Unity.Collections;
+using Unity.Mathematics;
 using UnityEngine.Rendering;
 
 namespace UnityEngine.Perception.GroundTruth
         /// The <see cref="IdLabelConfig"/> which associates objects with labels.
         /// </summary>
         public IdLabelConfig idLabelConfig;
+
+        /// <summary>
+        /// Controls which objects will have keypoints recorded in the dataset.
+        /// <see cref="KeypointObjectFilter"/>
+        /// </summary>
+        public KeypointObjectFilter objectFilter;
         // ReSharper restore MemberCanBePrivate.Global
 
         AnnotationDefinition m_AnnotationDefinition;
-        List<KeypointEntry> m_ToReport;
+        List<KeypointEntry> m_KeypointEntriesToReport;
 
         int m_CurrentFrame;
 
         /// </summary>
         public List<AnimationPoseConfig> animationPoseConfigs;
 
+
+
         /// <inheritdoc/>
         protected override void Setup()
         {
             m_KnownStatus = new Dictionary<uint, CachedData>();
 
             m_AsyncAnnotations = new Dictionary<int, (AsyncAnnotation, Dictionary<uint, KeypointEntry>)>();
-            m_ToReport = new List<KeypointEntry>();
+            m_KeypointEntriesToReport = new List<KeypointEntry>();
+            perceptionCamera.RenderedObjectInfosCalculated += OnRenderedObjectInfoReadback;
         }
 
         bool AreEqual(Color32 lhs, Color32 rhs)
 
         bool PixelsMatch(int x, int y, Color32 idColor, (int x, int y) dimensions, NativeArray<Color32> data)
         {
-            var h = dimensions.y - y;
+            var h = dimensions.y - 1 - y;
             var pixelColor = data[h * dimensions.x + x];
             return AreEqual(pixelColor, idColor);
         }
         {
             if (keypoint.state == 0) return 0;
 
-            var centerX = Mathf.RoundToInt(keypoint.x);
-            var centerY = Mathf.RoundToInt(keypoint.y);
+            var centerX = Mathf.FloorToInt(keypoint.x);
+            var centerY = Mathf.FloorToInt(keypoint.y);
-            var pixelOnScreen = false;
+            if (!PixelOnScreen(centerX, centerY, dimensions))
+                return 0;
+
+            var pixelMatched = false;
 
             for (var y = centerY - s_PixelTolerance; y <= centerY + s_PixelTolerance; y++)
             {
 
-                    pixelOnScreen = true;
+                    pixelMatched = true;
                     if (PixelsMatch(x, y, instanceIdColor, dimensions, data))
                     {
                         return 2;
 
-            return pixelOnScreen ? 1 : 0;
+            return pixelMatched ? 1 : 0;
         }
 
         void OnInstanceSegmentationImageReadback(int frameCount, NativeArray<Color32> data, RenderTexture renderTexture)
-
-            m_AsyncAnnotations.Remove(frameCount);
-            m_ToReport.Clear();
-
-                    var shouldReport = false;
-
                     foreach (var keypoint in keypointSet.Value.keypoints)
                     {
                         keypoint.state = DetermineKeypointState(keypoint, idColor, dimensions, data);
                         }
                         else
                         {
-                            shouldReport = true;
+                            keypoint.x = math.clamp(keypoint.x, 0, dimensions.width - .001f);
+                            keypoint.y = math.clamp(keypoint.y, 0, dimensions.height - .001f);
+                }
+            }
-                    if (shouldReport)
-                        m_ToReport.Add(keypointSet.Value);
+        }
+
+        private void OnRenderedObjectInfoReadback(int frameCount, NativeArray<RenderedObjectInfo> objectInfos)
+        {
+            if (!m_AsyncAnnotations.TryGetValue(frameCount, out var asyncAnnotation))
+                return;
+
+            m_AsyncAnnotations.Remove(frameCount);
+
+            m_KeypointEntriesToReport.Clear();
+
+            //filter out objects that are not visible
+            foreach (var keypointSet in asyncAnnotation.keypoints)
+            {
+                var entry = keypointSet.Value;
+
+                var include = false;
+                if (objectFilter == KeypointObjectFilter.All)
+                    include = true;
+                else
+                {
+                    foreach (var objectInfo in objectInfos)
+                    {
+                        if (entry.instance_id == objectInfo.instanceId)
+                        {
+                            include = true;
+                            break;
+                        }
+                    }
+
+                    if (!include && objectFilter == KeypointObjectFilter.VisibleAndOccluded)
+                        include = keypointSet.Value.keypoints.Any(k => k.state == 1);
+                if (include)
+                    m_KeypointEntriesToReport.Add(entry);
-            KeypointsComputed?.Invoke(frameCount, m_ToReport);
-            asyncAnnotation.annotation.ReportValues(m_ToReport);
+            //This code assumes that OnRenderedObjectInfoReadback will be called immediately after OnInstanceSegmentationImageReadback
+            KeypointsComputed?.Invoke(frameCount, m_KeypointEntriesToReport);
+            asyncAnnotation.annotation.ReportValues(m_KeypointEntriesToReport);
         }
 
         /// <param name="scriptableRenderContext"></param>
             return targetTexture != null ?
                 targetTexture.height : Screen.height;
         }
-
-        // Converts a coordinate from world space into pixel space
+            if (Mathf.Approximately(pt.y, perceptionCamera.attachedCamera.pixelHeight))
+                pt.y -= .0001f;
+            if (Mathf.Approximately(pt.x, perceptionCamera.attachedCamera.pixelWidth))
+                pt.x -= .0001f;
+
             return pt;
         }
 
                             var bone = animator.GetBoneTransform(pt.rigLabel);
                             if (bone != null)
                             {
-                                var loc = ConvertToScreenSpace(bone.position);
-                                keypoints[i].index = i;
-                                keypoints[i].x = loc.x;
-                                keypoints[i].y = loc.y;
-                                keypoints[i].state = 2;
+                                InitKeypoint(bone.position, keypoints, i);
                             }
                         }
                     }
                     foreach (var (joint, idx) in cachedData.overrides)
                     {
-                        var loc = ConvertToScreenSpace(joint.transform.position);
-                        keypoints[idx].index = idx;
-                        keypoints[idx].x = loc.x;
-                        keypoints[idx].y = loc.y;
-                        keypoints[idx].state = 2;
+                        InitKeypoint(joint.transform.position, keypoints, idx);
                     }
 
                     cachedData.keypoints.pose = "unset";
             }
         }
 
+        private void InitKeypoint(Vector3 position, Keypoint[] keypoints, int idx)
+        {
+            var loc = ConvertToScreenSpace(position);
+            keypoints[idx].index = idx;
+            if (loc.z < 0)
+            {
+                keypoints[idx].x = 0;
+                keypoints[idx].y = 0;
+                keypoints[idx].state = 0;
+            }
+            else
+            {
+                keypoints[idx].x = loc.x;
+                keypoints[idx].y = loc.y;
+                keypoints[idx].state = 2;
+            }
+        }
+
         string GetPose(Animator animator)
         {
             var info = animator.GetCurrentAnimatorClipInfo(0);
         /// <inheritdoc/>
         protected override void OnVisualize()
         {
-            if (m_ToReport == null) return;
+            if (m_KeypointEntriesToReport == null) return;
 
             var jointTexture = activeTemplate.jointTexture;
             if (jointTexture == null) jointTexture = m_MissingTexture;
 
-            foreach (var entry in m_ToReport)
+            foreach (var entry in m_KeypointEntriesToReport)
             {
                 foreach (var bone in activeTemplate.skeleton)
                 {

com.unity.perception/Tests/Runtime/GroundTruthTests/KeypointGroundTruthTests.cs

     public class KeyPointGroundTruthTests : GroundTruthTestBase, IPrebuildSetup, IPostBuildCleanup
     {
         private const string kAnimatedCubeScenePath = "Packages/com.unity.perception/Tests/Runtime/TestAssets/AnimatedCubeScene.unity";
+        private const double k_Delta = .01;
 
         public void Setup()
         {
 #endif
         }
 
-        static GameObject SetupCamera(IdLabelConfig config, KeypointTemplate template, Action<int, List<KeypointLabeler.KeypointEntry>> computeListener, RenderTexture renderTexture = null)
+        static GameObject SetupCamera(IdLabelConfig config, KeypointTemplate template, Action<int, List<KeypointLabeler.KeypointEntry>> computeListener, RenderTexture renderTexture = null, KeypointObjectFilter keypointObjectFilter = KeypointObjectFilter.Visible)
         {
             var cameraObject = new GameObject();
             cameraObject.SetActive(false);
             var perceptionCamera = cameraObject.AddComponent<PerceptionCamera>();
             perceptionCamera.captureRgbImages = false;
             var keyPointLabeler = new KeypointLabeler(config, template);
+            keyPointLabeler.objectFilter = keypointObjectFilter;
             if (computeListener != null)
                 keyPointLabeler.KeypointsComputed += computeListener;
 
         }
 
         [UnityTest]
-        public IEnumerator Keypoint_TestAllBlockedByOther()
+        public IEnumerator Keypoint_FullyOccluded_DoesNotReport()
         {
             var incoming = new List<List<KeypointLabeler.KeypointEntry>>();
             var template = CreateTestTemplate(Guid.NewGuid(), "TestTemplate");
                 incoming.Add(data);
             }, texture);
 
+            CreateFullyOccludedScene(template, cam);
+
+            yield return null;
+
+            //force all async readbacks to complete
+            DestroyTestObject(cam);
+
+            if (texture != null) texture.Release();
+
+            var testCase = incoming.Last();
+            Assert.AreEqual(0, testCase.Count);
+        }
+        [UnityTest]
+        public IEnumerator Keypoint_FullyOccluded_WithIncludeOccluded_ReportsProperly(
+            [Values(KeypointObjectFilter.VisibleAndOccluded, KeypointObjectFilter.All)] KeypointObjectFilter keypointObjectFilter)
+        {
+            var incoming = new List<List<KeypointLabeler.KeypointEntry>>();
+            var template = CreateTestTemplate(Guid.NewGuid(), "TestTemplate");
+            var texture = new RenderTexture(1024, 768, 16);
+            texture.Create();
+            var cam = SetupCamera(SetUpLabelConfig(), template, (frame, data) =>
+            {
+                incoming.Add(data);
+            }, texture, keypointObjectFilter);
+
+            CreateFullyOccludedScene(template, cam);
+
+            yield return null;
+
+            //force all async readbacks to complete
+            DestroyTestObject(cam);
+
+            if (texture != null) texture.Release();
+
+            var testCase = incoming.Last();
+            Assert.AreEqual(1, testCase.Count);
+
+            var t = testCase.First();
+            Assert.NotNull(t);
+            Assert.AreEqual(1, t.instance_id);
+            Assert.AreEqual(1, t.label_id);
+            Assert.AreEqual(template.templateID.ToString(), t.template_guid);
+            Assert.AreEqual(9, t.keypoints.Length);
+
+            for (var i = 0; i < 8; i++)
+                Assert.AreEqual(1, t.keypoints[i].state);
+
+            for (var i = 0; i < 9; i++) Assert.AreEqual(i, t.keypoints[i].index);
+            Assert.Zero(t.keypoints[8].state);
+            Assert.Zero(t.keypoints[8].x);
+            Assert.Zero(t.keypoints[8].y);
+        }
+
+        private void CreateFullyOccludedScene(KeypointTemplate template, GameObject cam)
+        {
             var cube = TestHelper.CreateLabeledCube(scale: 6, z: 8);
             SetupCubeJoints(cube, template);
 
             AddTestObjectForCleanup(cam);
             AddTestObjectForCleanup(cube);
             AddTestObjectForCleanup(blocker);
+        }
+
+
+        [UnityTest]
+        public IEnumerator Keypoint_Offscreen_DoesNotReport(
+            [Values(KeypointObjectFilter.VisibleAndOccluded, KeypointObjectFilter.Visible)] KeypointObjectFilter keypointObjectFilter)
+        {
+            var incoming = new List<List<KeypointLabeler.KeypointEntry>>();
+            var template = CreateTestTemplate(Guid.NewGuid(), "TestTemplate");
+            var texture = new RenderTexture(1024, 768, 16);
+            texture.Create();
+            var cam = SetupCamera(SetUpLabelConfig(), template, (frame, data) =>
+            {
+                incoming.Add(data);
+            }, texture, keypointObjectFilter);
+
+            var cube = TestHelper.CreateLabeledCube(scale: 6, z: -100);
+            SetupCubeJoints(cube, template);
+
+            cube.SetActive(true);
+            cam.SetActive(true);
+
+            AddTestObjectForCleanup(cam);
+            AddTestObjectForCleanup(cube);
+
+            yield return null;
+
+            //force all async readbacks to complete
+            DestroyTestObject(cam);
+
+            if (texture != null) texture.Release();
+
+            var testCase = incoming.Last();
+            Assert.AreEqual(0, testCase.Count);
+        }
+        [UnityTest]
+        public IEnumerator Keypoint_Offscreen_WithIncludeAll_ReportsProperly()
+        {
+            var incoming = new List<List<KeypointLabeler.KeypointEntry>>();
+            var template = CreateTestTemplate(Guid.NewGuid(), "TestTemplate");
+            var texture = new RenderTexture(1024, 768, 16);
+            texture.Create();
+            var cam = SetupCamera(SetUpLabelConfig(), template, (frame, data) =>
+            {
+                incoming.Add(data);
+            }, texture, KeypointObjectFilter.All);
+
+            var cube = TestHelper.CreateLabeledCube(scale: 6, z: -20);
+            SetupCubeJoints(cube, template);
+
+            cube.SetActive(true);
+            cam.SetActive(true);
+
+            AddTestObjectForCleanup(cam);
+            AddTestObjectForCleanup(cube);
 
             yield return null;
 
 
             var testCase = incoming.Last();
             Assert.AreEqual(1, testCase.Count);
+
             var t = testCase.First();
             Assert.NotNull(t);
             Assert.AreEqual(1, t.instance_id);
 
-            for (var i = 0; i < 8; i++)
-                Assert.AreEqual(1, t.keypoints[i].state);
-
-            for (var i = 0; i < 9; i++) Assert.AreEqual(i, t.keypoints[i].index);
-            Assert.Zero(t.keypoints[8].state);
-            Assert.Zero(t.keypoints[8].x);
-            Assert.Zero(t.keypoints[8].y);
+            for (var i = 0; i < 9; i++)
+            {
+                Assert.Zero(t.keypoints[i].state);
+                Assert.Zero(t.keypoints[i].x);
+                Assert.Zero(t.keypoints[i].y);
+            }
         }
 
         [UnityTest]
             Assert.AreEqual(screenPointCenterExpected.y, t.keypoints[8].y, Screen.height * .1);
             Assert.AreEqual(8, t.keypoints[8].index);
             Assert.AreEqual(2, t.keypoints[8].state);
+        }
+
+        static IEnumerable<(float scale, bool expectObject, int expectedState, KeypointObjectFilter keypointFilter, Vector2 expectedTopLeft, Vector2 expectedBottomRight)> Keypoint_OnBox_ReportsProperCoordinates_TestCases()
+        {
+            yield return (
+                1,
+                true,
+                2,
+                KeypointObjectFilter.Visible,
+                new Vector2(0, 0),
+                new Vector2(1023.99f, 1023.99f));
+            yield return (
+                1.001f,
+                true,
+                0,
+                KeypointObjectFilter.Visible,
+                new Vector2(0, 0),
+                new Vector2(0, 0));
+            yield return (
+                1.2f,
+                true,
+                0,
+                KeypointObjectFilter.Visible,
+                new Vector2(0, 0),
+                new Vector2(0, 0));
+            yield return (
+                0f,
+                false,
+                1,
+                KeypointObjectFilter.Visible,
+                new Vector2(512, 512),
+                new Vector2(512, 512));
+            yield return (
+                0f,
+                true,
+                1,
+                KeypointObjectFilter.VisibleAndOccluded,
+                new Vector2(512, 512),
+                new Vector2(512, 512));
+        }
+        [UnityTest]
+        public IEnumerator Keypoint_OnBox_ReportsProperCoordinates(
+            [ValueSource(nameof(Keypoint_OnBox_ReportsProperCoordinates_TestCases))]
+            (float scale, bool expectObject, int expectedState, KeypointObjectFilter keypointFilter, Vector2 expectedTopLeft, Vector2 expectedBottomRight) args)
+        {
+            var incoming = new List<List<KeypointLabeler.KeypointEntry>>();
+            var template = CreateTestTemplate(Guid.NewGuid(), "TestTemplate");
+            var frameSize = 1024;
+            var texture = new RenderTexture(frameSize, frameSize, 16);
+            var cam = SetupCamera(SetUpLabelConfig(), template, (frame, data) =>
+            {
+                incoming.Add(data);
+            }, texture, args.keypointFilter);
+
+            var camComponent = cam.GetComponent<Camera>();
+            camComponent.orthographic = true;
+            camComponent.orthographicSize = .5f;
+
+            var cube = TestHelper.CreateLabeledCube(scale: args.scale, z: 8);
+            SetupCubeJoints(cube, template);
+
+            cube.SetActive(true);
+            cam.SetActive(true);
+
+            AddTestObjectForCleanup(cam);
+            AddTestObjectForCleanup(cube);
+
+            yield return null;
+
+            //force all async readbacks to complete
+            DestroyTestObject(cam);
+            texture.Release();
+
+            var testCase = incoming.Last();
+            if (!args.expectObject)
+            {
+                Assert.AreEqual(0, testCase.Count);
+                yield break;
+            }
+            Assert.AreEqual(1, testCase.Count);
+            var t = testCase.First();
+            Assert.NotNull(t);
+            Assert.AreEqual(1, t.instance_id);
+            Assert.AreEqual(1, t.label_id);
+            Assert.AreEqual(template.templateID.ToString(), t.template_guid);
+            Assert.AreEqual(9, t.keypoints.Length);
+
+            CollectionAssert.AreEqual(Enumerable.Repeat(args.expectedState, 8), t.keypoints.Take(8).Select(k => k.state), "State mismatch");
+            Assert.AreEqual(args.expectedTopLeft.x, t.keypoints[0].x, k_Delta);
+            Assert.AreEqual(args.expectedBottomRight.y, t.keypoints[0].y, k_Delta);
+
+            Assert.AreEqual(args.expectedTopLeft.x, t.keypoints[1].x, k_Delta);
+            Assert.AreEqual(args.expectedTopLeft.y, t.keypoints[1].y, k_Delta);
+
+            Assert.AreEqual(args.expectedBottomRight.x, t.keypoints[2].x, k_Delta);
+            Assert.AreEqual(args.expectedTopLeft.y, t.keypoints[2].y, k_Delta);
+
+            Assert.AreEqual(args.expectedBottomRight.x, t.keypoints[3].x, k_Delta);
+            Assert.AreEqual(args.expectedBottomRight.y, t.keypoints[3].y, k_Delta);
         }
     }
 }

com.unity.perception/Editor/GroundTruth/JointLabelEditor.cs

+using System.Linq;
+using UnityEngine;
+using UnityEngine.Perception.GroundTruth;
+
+namespace UnityEditor.Perception.GroundTruth
+{
+    [CustomEditor(typeof(JointLabel))]
+    public class JointLabelEditor : Editor
+    {
+        public override void OnInspectorGUI()
+        {
+            base.OnInspectorGUI();
+#if UNITY_2020_1_OR_NEWER
+            //GetComponentInParent<T>(bool includeInactive) only exists on 2020.1 and later
+            if (targets.Any(t => ((Component)t).gameObject.GetComponentInParent<Labeling>(true) == null))
+#else
+            if (targets.Any(t => ((Component)t).GetComponentInParent<Labeling>() == null))
+#endif
+                EditorGUILayout.HelpBox("No Labeling component detected on parents. Keypoint labeling requires a Labeling component on the root of the object.", MessageType.Info);
+        }
+    }
+}

com.unity.perception/Editor/GroundTruth/JointLabelEditor.cs.meta

+fileFormatVersion: 2
+guid: 8b21d46736dd4cbb96bf827457752855
+timeCreated: 1616108507

com.unity.perception/Runtime/GroundTruth/Labelers/KeypointObjectFilter.cs

+namespace UnityEngine.Perception.GroundTruth
+{
+    /// <summary>
+    /// Keypoint filtering modes.
+    /// </summary>
+    public enum KeypointObjectFilter
+    {
+        /// <summary>
+        /// Only include objects which are partially visible in the frame.
+        /// </summary>
+        [InspectorName("Visible objects")]
+        Visible,
+        /// <summary>
+        /// Include visible objects and objects with keypoints in the frame.
+        /// </summary>
+        [InspectorName("Visible and occluded objects")]
+        VisibleAndOccluded,
+        /// <summary>
+        /// Include all labeled objects containing matching skeletons.
+        /// </summary>
+        [InspectorName("All objects")]
+        All
+    }
+}

com.unity.perception/Runtime/GroundTruth/Labelers/KeypointObjectFilter.cs.meta

+fileFormatVersion: 2
+guid: a1ea6aeb2b1c476288b4aa0cbc795280
+timeCreated: 1616179952

com.unity.perception/Documentation~/GroundTruth/KeypointLabeler.md

+# Keypoint Labeler
+
+The Keypoint Labeler captures the screen locations of specific points on labeled GameObjects. The typical use of this Labeler is capturing human pose estimation data, but it can be used to capture points on any kind of object. The Labeler uses a [Keypoint Template](#KeypointTemplate) which defines the keypoints to capture for the model and the skeletal connections between those keypoints. The positions of the keypoints are recorded in pixel coordinates.
+
+## Data Format
+The keypoints captured each frame are in the following format:
+```
+keypoints {
+  label_id:      <int>   -- Integer identifier of the label
+  instance_id:   <str>   -- UUID of the instance.
+  template_guid: <str>   -- UUID of the keypoint template
+  pose:          <str>   -- Current pose
+  keypoints [            -- Array of keypoint data, one entry for each keypoint defined in associated template file.
+    {
+      index:     <int>   -- Index of keypoint in template
+      x:         <float> -- X pixel coordinate of keypoint
+      y:         <float> -- Y pixel coordinate of keypoint
+      state:     <int>   -- Visibility state
+    }, ...
+  ]
+}
+```
+
+The `state` entry has three possible values: 
+* 0 - the keypoint either does not exist or is outside of the image's bounds
+* 1 - the keypoint exists inside of the image bounds but cannot be seen because the object is not visible at its location in the image
+* 2 - the keypoint exists and the object is visible at its location
+
+The annotation definition, captured by the Keypoint Labeler once in each dataset, describes points being captured and their skeletal connections. These are defined by the [Keypoint Template](#KeypointTemplate).
+```
+annotation_definition.spec {
+  template_id:       <str>           -- The UUID of the template
+  template_name:     <str>           -- Human readable name of the template
+  key_points [                       -- Array of joints defined in this template
+    {
+      label:         <str>           -- The label of the joint
+      index:         <int>           -- The index of the joint
+    }, ...
+  ]
+  skeleton [                         -- Array of skeletal connections (which joints have connections between one another) defined in this template
+    {
+      joint1:        <int>           -- The first joint of the connection
+      joint2:        <int>           -- The second joint of the connection
+    }, ...
+  ]
+}
+```
+
+## Setup
+The Keypoint Labeler captures keypoints each frame from each object in the scene that meets the following conditions:
+
+* The object or its children are at least partially visible in the frame 
+  * The _Object Filter_ option on the Keypoint Labeler can be used to also include fully occluded or off-screen objects
+* The root object has a `Labeling` component
+* The object matches at least one entry in the Keypoint Template by either:
+  * Containing an Animator with a [humanoid avatar](https://docs.unity3d.com/Manual/ConfiguringtheAvatar.html) whose rig matches a keypoint OR
+  * Containing children with Joint Label components whose labels match keypoints
+
+For a tutorial on setting up your project for keypoint labeling, see the [Human Pose Labeling and Randomization Tutorial](../HPTutorial/TUTORIAL.md).
+
+## Keypoint Template
+
+Keypoint Templates are used to define the keypoints and skeletal connections captured by the Keypoint Labeler. The Keypoint Template takes advantage of Unity's humanoid animation rig, and allows the user to automatically associate template keypoints to animation rig joints. Additionally, the user can choose to ignore the rigged points, or add points not defined in the rig.
+
+A [COCO](https://cocodataset.org/#home) Keypoint Template is included in the Perception package.
+
+### Editor
+
+The Keypoint Template editor allows the user to create/modify a Keypoint Template. The editor consists of the header information, the keypoint array, and the skeleton array.
+
+![Header section of the keypoint template](../images/keypoint_template_header.png)
+<br/>_Header section of the keypoint template_
+
+In the header section, a user can change the name of the template and supply textures that they would like to use for the keypoint visualization.
+
+![The keypoint section of the keypoint template](../images/keypoint_template_keypoints.png)
+<br/>_Keypoint section of the keypoint template_
+
+The keypoint section allows the user to create/edit keypoints and associate them with Unity animation rig points. Each keypoint record
+has 4 fields: label (the name of the keypoint), Associate to Rig (a boolean value which, if true, automatically maps the keypoint to
+the GameObject defined by the rig), Rig Label (only needed if Associate To Rig is true, defines which rig component to associate with
+the keypoint), and Color (RGB color value of the keypoint in the visualization).
+
+![Skeleton section of the keypoint template](../images/keypoint_template_skeleton.png)
+<br/>_Skeleton section of the keypoint template_
+
+The skeleton section allows the user to create connections between joints, basically defining the skeleton of a labeled object.
+
+#### Animation Pose Label
+
+This file is used to define timestamps in an animation to a pose label.