remove unnecessary allocs in SideChannelMananger (#4886)

com.unity.ml-agents/CHANGELOG.md

 #### com.unity.ml-agents (C#)
 - Fix a compile warning about using an obsolete enum in `GrpcExtensions.cs`. (#4812)
 - CameraSensor now logs an error if the GraphicsDevice is null. (#4880)
+- Removed unnecessary memory allocations in `ActuatorManager.UpdateActionArray()` (#4877)
+- Removed unnecessary memory allocations in `SensorShapeValidator.ValidateSensors()` (#4879)
+- Removed unnecessary memory allocations in `SideChannelManager.GetSideChannelMessage()` (#4886)
 - Removed several memory allocations that happened during inference. On a test scene, this
   reduced the amount of memory allocated by approximately 25%. (#4887)
 

com.unity.ml-agents/Runtime/SideChannels/SideChannelManager.cs

         /// <returns></returns>
         internal static byte[] GetSideChannelMessage(Dictionary<Guid, SideChannel> sideChannels)
         {
+            if (!HasOutgoingMessages(sideChannels))
+            {
+                // Early out so that we don't create the MemoryStream or BinaryWriter.
+                // This is the most common case.
+                return Array.Empty<byte>();
+            }
+
             using (var memStream = new MemoryStream())
             {
                 using (var binaryWriter = new BinaryWriter(memStream))
                     return memStream.ToArray();
                 }
             }
+        }
+
+        /// <summary>
+        /// Check whether any of the sidechannels have queued messages.
+        /// </summary>
+        /// <param name="sideChannels"></param>
+        /// <returns></returns>
+        static bool HasOutgoingMessages(Dictionary<Guid, SideChannel> sideChannels)
+        {
+            foreach (var sideChannel in sideChannels.Values)
+            {
+                var messageList = sideChannel.MessageQueue;
+                if (messageList.Count > 0)
+                {
+                    return true;
+                }
+            }
+
+            return false;
         }
 
         /// <summary>