4D reconstruction

In computer vision and computer graphics, 4D reconstruction is the process of capturing the shape and appearance of real objects along a temporal dimension. It adds the concept of time to a 3D representation to show how a scene changes over time. This process can be accomplished by methods such as depth camera imaging, photometric stereo, or structure from motion, and is also referred to as spatio-temporal reconstruction. The applications include free-viewpoint video and telepresence, and it is also used in other fields, such as medical imaging.

Methods

4D reconstruction methods can mainly be differentiated in three ways: how to capture data, how to handle changes over time, and how to represent the scene.

For capturing data, the input can come from a single moving camera, several synchronized cameras, or a sensor that measures depth directly. Many methods use structure from motion to estimate the camera's location first. The next step is to reconstruct the scene.

There are two main approaches to handling change over time. One is keeping a single canonical model and changing it each frame. The other is reconstructing the scene at each time step.

For scene representation, early methods used templates or used depth cameras to build it in real time. Later, neural network methods gave higher quality but were slower. More recent methods use 3D Gaussians and can render dynamic scenes in real time.

4D Gaussian splatting

Extending 3D Gaussian splatting to dynamic scenes, 3D temporal Gaussian splatting incorporates a time component, allowing for real-time rendering of dynamic scenes with high resolutions. It represents and renders dynamic scenes by modeling complex motions while maintaining efficiency. The method uses a HexPlane to connect adjacent Gaussians, providing an accurate representation of position and shape deformations. By utilizing only a single set of canonical 3D Gaussians and predictive analytics, it models how they move over different timestamps.

It is sometimes referred to as "4D Gaussian splatting"; however, this naming convention implies the use of 4D Gaussian primitives (parameterized by a 4×4 mean and a 4×4 covariance matrix). Most work in this area still employs 3D Gaussian primitives, applying temporal constraints as an extra parameter of optimization.[citation needed]

Achievements of this technique include real-time rendering on dynamic scenes with high resolutions, while maintaining quality. It showcases potential applications for future developments in film and other media, although there are current limitations regarding the length of motion captured.

Applications

In immersive media, 4D reconstruction is used for free-viewpoint video. A performance is recorded by many cameras at the same time to generate a 3D video that can be compressed and played online, so viewers can choose their own viewing angle.

It can also be used for telepresence. A person is reconstructed as a 3D model in real time and sent to a distant location, where someone wearing a mixed reality headset can see and interact with them as if they were standing in front of them.

In virtual reality, a musician's performance can be recorded and compressed into a very small file, and then viewers wearing a VR headset can watch it from any angle. In 2025, researchers developed a framework towards real-time 3D reconstruction about live sport events.

In medical imaging

Besides computer graphics, the field of medical imaging also has a similar need for 4D reconstruction. Work in computer graphics focuses on the shape of object surfaces, as it is mainly used in film and extended reality. In medical imaging, it focuses on how organs change with human breathing and heartbeat because it is used to evaluate how organs function.

See also