3D Glass Portal Card Impact with React Three Fiber and Gaussian Splatting

There’s a good video tutorial by Max on how to try this with Spline. However being a developer myself I wish to have extra management over the scene, so I made a decision to copy it utilizing React Three Fiber (R3F) and Three.js.

An enormous shoutout to the parents at Poimandres, whose work makes all this attainable!

Let’s dive into this impact immediately!

Fundamental scene

Due to the wonderful R3F group there’s a MeshTransmissionMaterial within the Drei repository now. I additionally took a mannequin to place behind that tumbler object. Needless to say for glass and such supplies it’s tremendous vital so as to add surroundings lighting (often that’s a .hdr or .exr file), in order that we’ll have the ability to see all these good reflections.

View the reside demo/code.

Now, there may be our first challenge with this. We shouldn’t be seeing the article behind the glass. Solely by the glass. The way in which this materials works, it renders the sculpture twice: first as a texture for the glass, and a second time as an precise object.

Fortunately, there may be a simple approach to repair this, simply by hiding the article for the ultimate render.

useFrame((state) => {
	sculpture.present.seen = true;
	glass.present.seen = false;
	// this buffer might be utilized by MeshTransmissionMaterial
	state.gl.setRenderTarget(buffer);
	state.gl.render(state.scene, state.digital camera);
	// we'd like this to modify rendering again to "on display"
	state.gl.setRenderTarget(null);
	sculpture.present.seen = false; // hiding sculpture
	glass.present.seen = true;
  });

And now utilizing the Transmission materials props we are able to set the “behind-the-glass scene” for it manually, like this:

<MeshTransmissionMaterial
      buffer={buffer.texture}
  >

Not solely can we obtain the portal impact now, we additionally optimize the entire scene rendering considerably! That’s as a result of we solely render the article as soon as.

That is what we’ve got now:

View the reside demo/code.

Splats

3D portal is fascinating, however let’s go additional and use Gaussian Splatting!

When you haven’t heard about Gaussian Splatting earlier than, I actually suggest watching this video, explaining it intimately. TLDW: It’s a brand new superior approach to render 3D scenes.

In our case it truly opens up an entire actual world to be built-in into our 3D scenes. In Max’s authentic Spline scene, a human scan was used. I noticed this as an important alternative to characteristic one of many symbols of my beloved Kyiv—the Ukraine Motherland Monument.

The issue is, the sculpture is over 100 meters tall. And you may’t fly civil drones in Ukraine, we’ve got common air assaults in right here, even this very second I’m writing these phrases.

So, I went to YouTube and located 4 12 months outdated footage of the monument. I uploaded the video on to the Luma app (however you would additionally use Polycam). And right here is the outcome:

Truthfully, I didn’t count on it to be that good! It amazes me that some outdated video can be utilized to recreate objects fairly precisely. However our quest just isn’t completed right here. There’s plenty of unneeded metropolis panorama within the scan, too. And if we obtain this “splat” file, it’s 260MB! Not very usable in an online surroundings.

Enhancing the scene

That’s the place the editor from Playcanvas builders involves the rescue. You’ll be able to simply take away the splats you don’t want.

So after cleansing it up the splat file is barely 800kb, which is fairly regular for such a 3D mannequin.

Now we are able to lastly export this as .splat format, which might then be used with Three.js!

Again to R3F

Lately, there was vital work in adapting this rendering method to be used in Three.js. Excitingly, simply final week, Paul Henschel efficiently ported it to React Three Fiber! When you’re not acquainted with Paul, I extremely suggest following him to remain on the forefront of the most recent developments in 3D internet know-how!

So, I received an opportunity and used his latest instance to simply add my edited .splat file to my scene. Coding it couldn’t be less complicated:

  <Splat
    	scale={2}
    	rotation={[0, 0, 0]}
    	place={[0, 0, 0]}
    	src="my.splat"
  />

Keep in mind, this code is an early model, so there is perhaps modifications sooner or later. It’s more likely to develop into part of the incredible Drei assortment as a module.

Right here’s what it seems like:

Try the code and reside demo in this CodeSandbox.

And we’re achieved! I additionally added a further field with stripes, so as to add some depth. It has facet="bottom" enabled, so we solely see the insides of it, that provides to the phantasm.

Right here’s one other object, a sculpture by Oleksii Zolotariov. I snapped a few pictures at an exhibition and effortlessly integrated it into my scene:

View the reside demo/code.

I’d like to see what you give you and the way you broaden on these demos. Be happy to let me know on Twitter!

In closing, it’s actually fascinating to see the involvement of so many wonderful people within the fast unfold of this new know-how. Whereas I can’t point out everybody, I encourage you to proceed supporting all of the contributors and open supply on the whole, because it’s the driving power behind these developments. I additionally encourage you to sponsor wonderful authors of Drei and React Three Fiber.

It’s additionally mind-blowing that I can document a random video on my cellphone and have these real-life objects in my Three.js scene inside an hour, trying incredible! Isn’t that unbelievable?