[VR] Single Pass Instanced/Multiview Support for Gaussian Splatting in URP

[arghyasur1991]

Support for Single Pass Instanced/Multiview rendering mode. Tested on Quest3

1. Use correct projection matrices per eye to calculate per eye view data in single shader call
2. Use EyeIndex to index into view data in RenderGaussianSplats shader for drawing correct eye texture for Splat
3. Use single prepatation step but separate render calls for Splat and composite in URP based on texture slices of 2darray texture
4. Use Texture 2dArray in GaussianComposite shader to draw the per eye texture correctly based on pass eye index

Also, ensured that non-VR mode and Multipass continue to work correctly

Not resolved in this PR:

1. HDRP for single pass instanced. We don't often use this pipeline in VR so low priority
2. Debug points and boxes shaders

[ninjamode]

Did not try it, but cool to see someone working on this! Just from looking at it, it seems to only sort for the original camera position, not both eyes? I guess this is fine as long as you don't get too close, but maybe it would be worth it to expose this behavior?
Anyway, I'll try this on a headset when i get then chance and report back

[arvinkx]

@arghyasur1991 @ninjamode I ran into some issues on macOS with latest LTS (6000.0.46f1) in non-VR mode:

Also, I've been working on similar updates but focused on Apple Vision Pro (happy to test on device for AVP); see #17 (comment), maybe you can use some of it: https://github.com/arvinkx/UnityGaussianSplatting/tree/spi. Specifically, handling foveated rendering and I used a separate render pass for the composite that is a normal render pass (as opposed to unsafe) which I believe would let the api optimize the composite stage in the pipeline (if it can find a way to). Hope it can be helpful.

[arghyasur1991]

I ran into some issues on macOS with latest LTS (6000.0.46f1) in non-VR mode:

@arvinkx I upgraded unity editor to latest 46f1 and it works correctly for me in mac M4. Will look at your code for other suggestions. Thanks.

happy to test on device for AVP

@arvinkx That will be very helpful. Pls let me know if it works.

it seems to only sort for the original camera position, not both eyes

@ninjamode Thanks for the input. Will add optional support for sorting per eye.

[arghyasur1991]

My current priority is to improve performance if possible by clubbing draw calls to tex2darray render textures in instanced mode. Have not been able to get it working yet.

[arghyasur1991]

@ninjamode Added parameter for Sorting Per Eye (disabled by default)

[StephenHodgson]

overall looks great thanks for contributing this!

[arvinkx]

@arvinkx I upgraded unity editor to latest 46f1 and it works correctly for me in mac M4. Will look at your code for other suggestions. Thanks.

It might be worth looking into that issue further as that specific splat works in the latest in main but that screenshot is what I get in editor with this PR. That specific splat is in the repo I linked (in my fork), if that's helpful. Trying with other splats I am getting odd results as well such as the preview in scene view not matching game view. I don't know if it makes a difference but I'm on an M1 Max.

@arvinkx That will be very helpful. Pls let me know if it works.

My repo has code in GaussianSplatRenderURPFeature to support foveated rendering as Unity (and visionOS) default to foveated rendering on, so at minimum you'd need to tell the user to disable in Project Settings it if it won't be supported. I unfortunately can't test on device as the project can't be built for visionOS the way it is now. I get build errors about a couple shaders (boxes and composite - probably debug points as well):

Shader error in 'Gaussian Splatting/Debug/Render Boxes': cannot assign to variable 'unity_StereoEyeIndex' with const-qualified type 'const int' at /Applications/Unity/Hub/Editor/6000.0.46f1/Unity.app/Contents/CGIncludes/UnityInstancing.cginc(187) (on metal) 

and

Shader error in 'Hidden/Gaussian Splatting/Composite': cannot assign to variable 'unity_StereoEyeIndex' with const-qualified type 'const int' at /Applications/Unity/Hub/Editor/6000.0.46f1/Unity.app/Contents/CGIncludes/UnityInstancing.cginc(187) (on metal)
Compiling Subshader: 0, Pass: <Unnamed Pass 0>, Vertex program with STEREO_INSTANCING_ON
Platform defines: UNITY_ENABLE_DETAIL_NORMALMAP UNITY_ENABLE_REFLECTION_BUFFERS UNITY_FRAMEBUFFER_FETCH_AVAILABLE UNITY_LIGHTMAP_DLDR_ENCODING UNITY_LIGHT_PROBE_PROXY_VOLUME UNITY_NO_DXT5nm UNITY_PBS_USE_BRDF1 UNITY_SPECCUBE_BLENDING UNITY_SPECCUBE_BOX_PROJECTION UNITY_USE_DITHER_MASK_FOR_ALPHABLENDED_SHADOWS
Disabled keywords: SHADER_API_GLES30 STEREO_MULTIVIEW_ON UNITY_ASTC_NORMALMAP_ENCODING UNITY_COLORSPACE_GAMMA UNITY_HARDWARE_TIER1 UNITY_HARDWARE_TIER2 UNITY_HARDWARE_TIER3 UNITY_LIGHTMAP_FULL_HDR UNITY_LIGHTMAP_RGBM_ENCODING UNITY_METAL_SHADOWS_USE_POINT_FILTERING UNITY_NO_SCREENSPACE_SHADOWS UNITY_PBS_USE_BRDF2 UNITY_PBS_USE_BRDF3 UNITY_PRETRANSFORM_TO_DISPLAY_ORIENTATION UNITY_SINGLE_PASS_STEREO UNITY_UNIFIED_SHADER_PRECISION_MODEL UNITY_VIRTUAL_TEXTURING

Those shaders would have to be updated if you want to be able to support building for Apple Vision Pro. I believe my repo has updated versions of those two debug shaders (but they use unity_StereoEyeIndex so they aren't written the same)

Also, I would recommend using unity_StereoEyeIndex in the shader rather than make two separate draw calls. That built-in variable will tell you which eye it's currently rendering (doesn't work in compute though).

[arghyasur1991]

Editor bug

@arvinkx I have fixed a bug in editor. It should hopefully work now.

Also, I would recommend using unity_StereoEyeIndex in the shader rather than make two separate draw calls.

Unity doesn't set this variable correctly when shader is run using DrawProcedural or Blit. I have tried multiple ways to make it work but this is the only way it will work currently. There is another approach to prevent 2 draw calls which is to use geometry shader for setting rtindex : SV_RenderTargetArrayIndex; But unfortunately that actually degrades the performance. https://github.com/arghyasur1991/UnityGaussianSplatting/tree/multiview8 is a branch where this single draw call thing is pushed if you or someone else wants to take a look.

[arvinkx]

Editor bug

@arvinkx I have fixed a bug in editor. It should hopefully work now.

With the latest changes, it looks correct in scene view but when you run it doesn't display in the game view. Unfortunately, I can't build for device due to the shaders not compiling to test further.

Unity doesn't set this variable correctly when shader is run using DrawProcedural or Blit. I have tried multiple ways to make it work but this is the only way it will work currently. There is another approach to prevent 2 draw calls which is to use geometry shader for setting rtindex : SV_RenderTargetArrayIndex; But unfortunately that actually degrades the performance. https://github.com/arghyasur1991/UnityGaussianSplatting/tree/multiview8 is a branch where this single draw call thing is pushed if you or someone else wants to take a look.

I would recommend looking at my repo - it works correctly (I have not had any issues with unity_StereoIndex being incorrect). The issue I'm having has to do with sort order inconsistencies but the rendering does work correctly using a single draw call and unity_StereoIndex in both draw and composite. There is also a debug shader provided by Unity here that you can use to make sure. It will print a single but different color in each eye. Also, definitely would not recommend using geometry shaders as they are not supported on Metal at all (see here)

[arghyasur1991]

it works correctly (I have not had any issues with unity_StereoIndex being incorrect)

I tried your repo. It didn't work on Quest3 since there are build errors. In editor as well, it behaves very weirdly and stays at same place for both scene view and game view.

but when you run it doesn't display in the game view

Couldn't repro this. it works fine in game view on editor on my end.

[arvinkx]

it works correctly (I have not had any issues with unity_StereoIndex being incorrect)

I tried your repo. It didn't work on Quest3.

I have not tested with Quest 3 but it's mentioned here that it worked so not sure but works on Apple Vision Pro with a single draw call. The point was I don't think there's an issue with unity_StereoEyeIndex at least not on Metal / visionOS. If anything, the debug shader I linked to would be more conclusive of whether there was an issue with unity_StereoIndex.

but when you run it doesn't display in the game view

Couldn't repro this. it works fine in game view on editor on my end.

Ok, glad to hear it works on yours. Hopefully others don't run into the same.

[arghyasur1991]

I have not tested with Quest 3 but it's mentioned here that it worked so not sure

It only worked with multipass until now. This PR adds support for single pass instanced and multiview. Quest3 only supports multiview so I could not directly test "single pass instanced".

"also there is a weird disparity between L and R eye. Tested on Quest 3." The comment mentions the weird disparity between eyes. That is the core issue with unity_stereoIndex. It is always passed as 0 in these shaders so you won't get correct L and R buffers. At least that's the case with Quest. Maybe it works in AVP.

Debug shader

Yes I have tried that and used it as a reference for debugging during my development. That's why I know for sure, in quest3 it didn't work. I spent a lot of time trying getting it to work with unity_stereoIndex

foveated rendering on

In quest3 even if foveated rendering is on through OpenXR settings, it works fine with my change. Although, I am not totally sure if this is actually working since quest3 hardware doesn't support eye tracking.

[arghyasur1991]

@arvinkx I don't have vision Pro but i built a new project with VisionOS target and built with this package installed. It built with my code and the shaders got built as well. Can you directly clone my repo and try to build?

[arvinkx]

@arghyasur1991 I did clone your repo and built with the Apple Vision OS plugin for device and ran into those compile errors which are known issues on AVP with this library, example here.

As I said, not sure about Quest 3 as I have not personally tested it but the repo I've been working on does work with Apple Vision Pro. I'll keep my changes for AVP in a separate repo as the goal there is to support single pass instanced for Apple Vision Pro which doesn't seem compatible with this PR the way it is. It would probably be a good idea to report the unity_StereoIndex bug to Unity if you can reproduce it with the debug shader in a project.

[arghyasur1991]

@arvinkx Thanks for the suggestions. For now, this is the only way I could make it work for Quest3. Added workaround and TODO notes in the file for future revisit.

[cdrintherrieno]

@arghyasur1991 On my side the splats are much smaller when using SPI (either on PC or native Quest 3). It really changes the appearance of the scenes. Is it a known issue?

[arghyasur1991]

On my side the splats are much smaller when using SPI

@cdrintherrieno Strange. Didn't see this issue. I tested in Quest3 with both multipass and SPI (multiview) mode and things look correct and identical in both. In Editor, it is in non-stereo mode right?
Can you pls share more details of your project setup and repro steps if possible?

[cdrintherrieno]

@arghyasur1991 For the repro I simply opened the URP project from the repo (multiview7 branch) under Unity 6000.0.33f (DX12). I added my splat asset to the scene and added the OpenXR plugin to the project.

When set to multi-pass it looks like this:

But when set to SPI it's quite different:

[arghyasur1991]

@cdrintherrieno This is what shows in My macbook M4 (6000.0.46f). Wonder if this is a DX12 specific issue with my code. Will have to check.

[cdrintherrieno]

@arghyasur1991 I tried updating to 6000.0.41f, and also switch to Vulkan, but still the same problem. What is strange is that when not in play mode it works fine in the game window, the splat size is correct but when hitting play and entering VR it becomes scaled down (both in VR and in the game window). Setting a "Splat Scale" of 2 is almost the same as it should be, but then when not in play mode splats are too big. I also noticed that the "Debug Points" render mode is affected by it (points are smaller), but the "Debug Boxes mode" is fine.

[arghyasur1991]

@cdrintherrieno I am not able to repro this on windows machine as well. Graphics card - RTX 4060. DX12 and Vulkan both are giving same results in my machine as mac. Only if I reduce the splat scale from 1, I see similar output as yours.

To diagnose this, can you tell me -

1. What is the URP version, and all the packages installed in your project?
2. Add a debug log in "GaussianSplatRenderer.cs" at line 695 to check the "isStereo" variable. In editor, it should always be false.
3. Do the same thing in SetAssetDataOnMaterial method. Check eyeIndex here. eyeIndex should be -1.

[Aupuma]

@arghyasur1991 I have the same issue as @cdrintherrieno .
I just downloaded this branch, installed the OpenXR plugin and set the graphics API to Vulkan.

1. Unity 6000.0.33f1 with URP 17.0.3
2. It changes constantly between True and False
3. EyeIndex always returns -1

[arghyasur1991]

@Aupuma Thanks for the info. Based on this, it seems in editor mode, somehow isStereo is getting set. Are you using MockHMD package as XR provider by any chance? Anyway, has commited a potential fix just for editor. Pls test and let me know if this fixes editor issue.
CC @cdrintherrieno as well

[cdrintherrieno]

@arghyasur1991 I tested it out but it doesn't work, it actually breaks VR rendering altogether (right eye renders black).

[wells2kb]

I had the same small splats issue in SPI, the splats are exactly half as big as they should be.

I do not know for sure but my suspicion is that _ScreenParams can either be the size of each or both eyes.

For my quest 2 the following change fixed the issue:

        #if defined(UNITY_SINGLE_PASS_STEREO) || defined(STEREO_INSTANCING_ON) || defined(STEREO_MULTIVIEW_ON)
          float2 deltaScreenPos = (quadPos.x * view.axis1 + quadPos.y * view.axis2) * 4 / _ScreenParams.xy;
        #else
          float2 deltaScreenPos = (quadPos.x * view.axis1 + quadPos.y * view.axis2) * 2 / _ScreenParams.xy;
        #endif