3D object labeler does not work with synthetic human

[LeonardMendicantBias]

Hi, I am utilizing the synthetic human and perception packages to generate training data for deep learning networks.
I followed the guides on both readme documentation and obtained 2D object detection annotation for humans. However, when I add a 3D object labeler to a perception camera, Unity reports an error as shown

The error persists even on a completely new, freshly initialized Unity project. If I understand correctly, the error says that one of the inputs (vertexBuffer) to the "Compute3DbbAsync" function is null and somehow causes a memory leak.

Another weird thing is that when I generate 100 interactions. The same error only occurs for 94 iterations but not the last 6. Thus, I could only obtain 6 training data instead of 100. Note that when I turn off the 3D object labeler (only enable the 2D object labeler), the simulation works well and I can obtain 100 training samples.

Since I am not an expert on Unity, Perception, or Synthetic human package. Does anyone know where I should start to debug?

[doraemonaaaa]

您好，我正在使用合成人类和感知包为深度学习网络生成训练数据。我遵循了两个 readme 文档中的指南，并为人类获取了 2D 对象检测注释。但是，当我将 3D 对象标记器添加到感知摄像头时，Unity 会报告错误，如下所示

即使在全新的、新初始化的 Unity 项目上，该错误仍然存在。如果我理解正确，该错误指出“Compute3DbbAsync”函数的其中一个输入 （vertexBuffer） 为 null，并以某种方式导致内存泄漏。

另一件奇怪的事情是，当我生成 100 次交互时。相同的错误仅发生在 94 次迭代中，而不会发生在最后 6 次迭代中。因此，我只能获得 6 个训练数据，而不是 100 个。请注意，当我关闭 3D 对象标记器（仅启用 2D 对象标记器）时，模拟运行良好，我可以获得 100 个训练样本。

因为我不是 Unity、Perception 或 Synthetic human package 方面的专家。有谁知道我应该从哪里开始调试？

Hi, I got the same error, too, Not only in Synthetic human, also in other human models, but other type of model work well, have you solved this problem?

[doraemonaaaa]

您好，我正在使用合成人体和感知包为深度学习网络生成训练数据。我按照自述文件中的指南进行作，并获得了人类的 2D 对象检测注释。但是，当我将 3D 对象标记器添加到感知相机时，Unity 报告了一个错误，如下所示

即使在全新的、新初始化的 Unity 项目上，该错误仍然存在。如果我理解正确，该错误表示“Compute3DbbAsync”函数的其中一个输入 （vertexBuffer） 为 null，并以某种方式导致内存泄漏。

另一个奇怪的事情是，当我生成 100 次交互时。相同的错误仅发生在 94 次迭代中，而不是最后 6 次迭代中。因此，我只能获得 6 个训练数据，而不是 100 个。请注意，当我关闭 3D 对象标记器（仅启用 2D 对象标记器）时，模拟运行良好，我可以获得 100 个训练样本。

因为我不是 Unity、Perception 或 Synthetic human package 方面的专家。有谁知道我应该从哪里开始调试？

I solved this problem, this is some bug in source code, you should change code like below
BoundingBox3DLabeler.cs/ProcessMeshFiltersAsync
foreach (var mesh in meshFilters)
{
if (!mesh.GetComponent().enabled)
continue;

            if (mesh.mesh == null)
                continue;

            mesh.mesh.vertexBufferTarget |= GraphicsBuffer.Target.Raw;
            var objectToLabelSpaceTransform =
                labelTransform.worldToLocalMatrix * mesh.transform.localToWorldMatrix;

            var graphicsBuffer = mesh.mesh.GetVertexBuffer(0);
            if (graphicsBuffer == null)
                continue;

            Compute3DbbAsync(context, graphicsBuffer, objectToLabelSpaceTransform, pendingBounds);
            graphicsBuffer.Dispose();
            any = true;
        }
        foreach (var mesh in skinnedMeshRenderers)
        {
            if (!mesh.GetComponent<Renderer>().enabled)
                continue;

            if (mesh.rootBone == null)
                continue;

            var rootBoneLocalToWorldMatrix = mesh.rootBone.localToWorldMatrix;
            var objectToLabelSpaceTransform =
                labelTransform.worldToLocalMatrix *
                rootBoneLocalToWorldMatrix *
                Matrix4x4.Scale(rootBoneLocalToWorldMatrix.lossyScale).inverse;

            var graphicsBuffer = mesh.GetVertexBuffer();
            if (graphicsBuffer == null)
                continue;

            Compute3DbbAsync(context, graphicsBuffer, objectToLabelSpaceTransform, pendingBounds);
            graphicsBuffer.Dispose();
            any = true;
        }