3D Gaussian Splatting

Radiance Field methods have recently revolutionized the visualization synthesis of scenes captured with multiple photos or videos. However, achieving high-quality visualization still requires neural networks that are expensive and slow to train and render.

This is where 3D Gaussian Splatting comes in, a method that promises to render a 3D scene in real-time – here are all the details!

What is 3D Gaussian Splatting?

Gaussian Splatting is a rendering technique created in the 90s in the scientific field. Still, its recent application to the visualization of scenes in real-time, presented last August at SIGGRAPH 2023, has brought it back into fashion.

Gaussian Splatting is a rasterization technique for real-time 3D reconstruction and rendering of images taken from multiple points of view.

The 3D space is defined as a set of Gaussians, and each Gaussian’s parameters are calculated by machine learning. This means:

• Have data describing the scene.
• Draw the data on the screen.

It is, therefore, analogous to triangle rasterization in computer graphics, which is used to draw many triangles on the screen. However, instead of drawing triangles, they are Gaussian. Therefore, it is described by the following parameters:

• Position: where it is located (XYZ)
• Covariance: how it stretches/scales (3×3 matrix)
• Color: RGB
• Alpha: its transparency (α)

Here’s an example

*Raster triangle – Gaussian

How does Gaussian Splatting work?

3D Gaussian Splatting represents a 3D scene as millions of particles (Gaussians), and each 3D Gaussian comes with a position/orientation/scale, as well as an opacity and a color.

To render these particles, they are converted into 2D space and then organized for efficient rendering. The steps to follow are:

1. Structure from Motion: We start using the Structure from Motion (SfM) method to create a point cloud from images. This can be done using COLMAP.
2. Convert to Gaussian: each point is converted to Gaussian, making rasterization possible. However, only position and color can be inferred from the SfM data.
3. Training: For a representation to produce high-quality results, it must be trained. For this purpose, Stochastic Gradient Descent, similar to a neural network, is used.
   1. Rasterise Gaussians in an image using differentiable Gaussian rasterization.
   2. Calculate the loss as a function of the difference between the raster and real terrain images.
   3. Adjust the Gaussian parameters according to the loss.
   4. Apply automated densification.
4. Differentiable Gaussian rasterization: here, each 2D Gaussian must be projected from the camera perspective, sorted by the depth, and, for each pixel, each Gaussian must be repeated backward and forward combined.

Photometry vs. NERF

Photometry is a technique for generating 3D polygons from images from multiple viewpoints. Although it is very useful for scanning objects, it cannot represent scenes without contours, such as the sky, or fine details in distant scenes. In addition, it is not possible to generate 3D polygons accurately in some cases, such as reflections or transparent elements.

To solve these limitations, NeRF or Neural Radiance Field has emerged. It is also a method for rendering a 3D scene from images taken from different viewpoints but uses AI to accurately render open scenes and surfaces with reflections or translucent surfaces. The only disadvantage is that very powerful hardware is required to render a scene in real-time.

3D Gaussian Splatting is another approach to solve the same problem but does not require as much computing power to render a scene in real-time. This is because, although it requires machine learning to train the Gaussian parameters, the rendering does not require any heavy processing and can be done in real-time.

How we are doing it at Plain Concepts

At Plain Concepts, we are working on integrating 3D Gaussian splatting technology into our Evergine 3D engine. Although it is a new technology in which continuous advances are being made, from Plain Concepts, we believe that it is ready to be used in real projects, and we are convinced that we will start to see it implemented in our next projects.

The Future of Graphics

There are still some unknowns around Gaussian Splatting, especially about the possibility of being able to animate them or to avoid the appearance of some visual artifacts or the management of collisions solved by having an associated geometric model.

But as we mentioned before, we will likely see 3D Gaussian Splatting fully implemented in a production environment.

Also, embedded AI is going to be a key factor in 3D space research. But what is clear is that this new technique will create high-quality photorealistic scenes, fast and real-time rasterization, and relatively quick to train.