Learning 3D environments with handheld RGB-D cameras
To this point, now we have only seen offline approaches for modeling 3D scenes (e.g., NeRF, SRNs, DeepSDF [2, 3, 4]). Despite their impressive performance, these approaches require days, and even weeks, of computation time for the underlying neural networks to be trained. For instance, NeRFs are trained for nearly two days for just representing a single scene. Plus, using the the neural net to guage a brand new scene viewpoint will be quite expensive too! With this in mind, we’d ponder whether it’s possible to learn a scene representation a bit faster than this.
This query was explored in [1] with the proposal of iMAP, a real-time system for representing scenes and localizing (i.e., tracking the pose of) devices within the scene. To know what this implies, consider a camera that’s moving through a scene and capturing the encompassing environment. The duty of iMAP is to (i) absorb this data, (ii) construct a 3D representation of the scene being observed, and (iii) infer the placement and orientation of the camera (i.e., the device) because it captures the scene!
iMAP adopts an approach that is kind of just like NeRF [2] with just a few differences:
- It is predicated upon RGB-D data.
- A streaming setup is assumed.
As such, the model receives each depth and color information as input. Moreover, the training process begins from a totally random initialization, and iMAP must learn from recent, incoming RGB-D images in real-time. Given this setup, iMAP is predicted to (i) model the scene and (ii) predict the pose of the RGB-D camera for every incoming image (i.e., prior methods assume pose information as an input!). Despite this difficult training setup, iMAP can learn 3D representations of entire rooms in real-time!
why is that this paper necessary? This post is a component of my series on deep learning for 3D shapes and scenes. This area was recently revolutionized by the proposal of NeRF [2]. With a NeRF representation, we will produce an arbitrary variety of synthetic viewpoints of a scene and even generate 3D representations of relevant objects; see below.
iMAP was proposed barely after NeRFs, and it’s capable of manufacturing high-quality scene representations without requiring several days of coaching time. In comparison with NeRF, iMAP learns in an affordable, on-the-fly manner and still performs relatively well.
Now we have seen some necessary background concepts in prior overviews on this series that shall be relevant here:
- Feed-forward neural networks https://towardsdatascience.com/imap-modeling-3d-scenes-in-real-time-6202365d80ee?source=rss—-7f60cf5620c9—4
- Representing 3D shapes https://towardsdatascience.com/imap-modeling-3d-scenes-in-real-time-6202365d80ee?source=rss—-7f60cf5620c9—4
- Camera poses https://towardsdatascience.com/imap-modeling-3d-scenes-in-real-time-6202365d80ee?source=rss—-7f60cf5620c9—4 (scroll to “camera viewpoints” sub-header)
- What are “frames” in a video? https://towardsdatascience.com/imap-modeling-3d-scenes-in-real-time-6202365d80ee?source=rss—-7f60cf5620c9—4
To have all of the context essential for understanding iMAP, we’d like to quickly cover the concepts of SLAM systems and online learning.
Prior scene representation methods now we have seen use deep neural networks to form an implicit representation of an underlying shape or scene by learning from available observations (e.g., point cloud data, images, etc.) of 3D space. iMAP is a bit different, since it is a Simultaneous Localization and Mapping (SLAM) system. Because of this iMAP performs two tasks:
- Localization: tracking the placement and pose of the camera that’s capturing the underlying scene.
- Mapping: forming a representation of the underlying scene.
Most prior techniques now we have seen only perform mapping, but iMAP goes beyond these techniques by also performing localization. Namely, viewpoints of the underlying scene are passed in real-time to the iMAP system, which each maps the underlying scene and predicts the camera’s trajectory because it traverses the scene.
The output of a SLAM system, as shown above, has two components. A 3D representation of the scene is generated. Inside this representation, nonetheless, we will see the camera’s trajectory because it captures the scene, shown by the yellow line (camera position) and associated 3D bounding boxes (camera pose).
how is that this different? Beyond predicting camera poses on incoming data, SLAM systems are different from what now we have seen to this point due to manner by which they receive data. Namely, with prior methods we (i) get a bunch of images of a scene and (ii) train a neural network over these images to model the scene. As a substitute, SLAM systems receive data in a streaming fashion. Because the system receives recent images, it must take them in, predict a pose, and update its underlying scene representation in real-time.
Typically, deep neural networks are trained in an offline fashion. Now we have a big, static training dataset available, and we allow our neural network to perform several training passes (or epochs) over this dataset; see above. But, what happens when our dataset will not be static? For instance, we could also be receiving recent data in real time or correcting labels applied to existing data.
We’ll consult with this setting generally as “online learning”, which refers back to the indisputable fact that we’re sequentially receiving recent data with which to coach a neural network.
Many differing types of online learning exist; see above. For instance, incremental learning assumes that the neural network sequentially receives recent batches of knowledge, while streaming learning mandates that the network receives data one example at a time. All types of online learning assume that data is learned in a single pass — we will’t “look back” at old data after receiving recent data.
catastrophic forgetting. Online learning is taken into account harder than offline learning because we never have access to the total dataset during training. Slightly, we must learn over small subsets of knowledge which might be made available sequentially. If the incoming data is non-i.i.d., our neural network may suffer from catastrophic forgetting. As discussed in [6], catastrophic forgetting refers back to the neural network completely forgetting about older data because it learns from recent, incoming data.
For instance, if we’re learning to categorise cats and dogs, possibly we sequentially receive loads of data that only has pictures of cats (e.g., this might occur on an IoT doorbell camera at someone’s house!). On this case, the neural network would learn from only pictures of cats for a very long time, causing it to catastrophically forget the concept of a dog. This could not be an issue if the incoming data was distributed equally between cats and dogs. Forgetting occurs because incoming data is non-i.i.d., and online learning techniques aim to avoid such forgetting.
replay buffers. One popular approach to avoiding catastrophic forgetting is via a replay buffer. At a high level, replay buffers just store a cache of knowledge that has been observed up to now. Then, when the network is updated over recent data, we will sample some data from the replay buffer as well. This manner, we get an equal sampling of knowledge that the neural network has seen; see below.
All online learning techniques share the common goal of maximizing a neural network’s performance by minimizing the impact of catastrophic forgetting. Although replay buffers are widely used, easy, and quite effective, many other techniques exist as well — online learning is an lively area of research throughout the deep learning community. For an outline of some existing techniques, I like to recommend reading my summaries below!
- Overview of Online Learning Techniques https://towardsdatascience.com/imap-modeling-3d-scenes-in-real-time-6202365d80ee?source=rss—-7f60cf5620c9—4
- Overview of Streaming Learning Techniques https://towardsdatascience.com/imap-modeling-3d-scenes-in-real-time-6202365d80ee?source=rss—-7f60cf5620c9—4
why should we care about online learning? iMAP is a SLAM system. Fundamentally, SLAM systems are quite related to online learning, as they’re expected to receive a stream of scene images and supply tracking and mapping leads to return. Notably, iMAP is a web based learning-based technique that relies upon a replay buffer to represent and track an underlying scene in real-time!
After first learning about iMAP, it’d appear to be the strategy is simply too good to be true. Initially, iMAP is solving a harder problem than prior work — camera pose information is predicted from RGB-D data fairly than given. Then, we must also learn all the scene representation in real-time, as a substitute of coaching for several days? There’s no way that this is feasible…
But, it’s! iMAP does all of this via a two-part processing pipeline that features:
- Tracking: predicts the placement and pose of the camera because it moves through the scene.
- Mapping: learns the 3D scene representation.
These two components run in parallel as an RGB-D camera captures the underlying scene from various viewpoints. Notably, tracking has to run quite fast, as we try to actively localize the camera as recent data is coming in. The mapping component is completed in parallel, nevertheless it only operates on keyframes which might be really necessary to representing the underlying scene, which allows iMAP to learn in real-time. Let’s get into a few of the details!
the network. Step one in understanding iMAP is learning concerning the neural network upon which it is predicated. Like most prior work, iMAP uses a feed-forward network architecture. The network takes a 3D coordinate as input and produces an RGB color and a volume density (i.e., captures opaqueness) as output; see above.
Notably, iMAP’s network doesn’t take a viewing angle as input, as in NeRF [2], since it doesn’t try and model view-dependent effects (e.g., reflections). Much like NeRF [2], iMAP converts coordinates into higher-dimensional positional embeddings before passing them as input, following the approach of [5]; see below.
quick note on rendering. Assuming access to camera poses (we are going to learn the way iMAP predicts these later), we will evaluate this network over many various spatial locations, then aggregate color and depth information right into a rendering of the underlying scene using an approach like Ray Marching. The rendering approach of [1] is analogous to that of NeRF, but we would like to incorporate depth information in our rendering (i.e., render an RGB-D image). This makes the rendering process barely different, but the fundamental idea is identical, and all the process continues to be differentiable.
how will we optimize this? iMAP is attempting to learn:
- The parameters of the feed-forward scene network.
- Camera poses for incoming frames from the RGB-D camera.
To coach the iMAP system to predict this information accurately, there are two sorts of loss metric upon which we rely:
- Photometric: error in RGB pixel values.
- Geometric: error in depth information.
These errors are computed by utilizing iMAP to render an RGB-D view of the underlying scene, then comparing this rendering to the bottom truth samples being captured from the camera. To make this comparison more efficient, we only consider a subset of pixels throughout the RGB-D image; see below.
To optimize photometric and geometric error jointly, we just mix them via a weighted sum; see below.
To render a picture with iMAP, we must:
- Use the scene network to acquire color and geometry information.
- Infer the proper camera pose.
- Render and RGB-D viewpoint based on this combined information.
All of those steps are differentiable, so we will optimize the photometric and geometric loss with techniques like stochastic gradient descent, thus training the system to provide renderings that closely match the bottom truth RGB-D images.
tracking and mapping. The goal of tracking in iMAP is to learn the pose of the camera that’s actively capturing the underlying scene. Provided that now we have to do that for each incoming RGB-D frame from the camera, tracking should be efficient. iMAP handles tracking by (i) freezing the scene network and (ii) solving for the present RGB-D frame’s optimal pose given the fixed network. That is just an initial estimate of the camera pose that we generate as efficiently as possible — we may refine this pose afterward.
The goal of mapping is to jointly optimize the scene network and camera poses, implicitly creating an accurate representation of the scene. Attempting to do that over all incoming, RGB-D camera frames can be way too expensive. As a substitute, we maintain a set of keyframes based on importance (i.e., whether or not they capture a “recent” a part of the scene). Inside this set of keyframes, we refine estimated camera poses and train the scene network to provide renderings that closely match chosen keyframes.
putting all of it together. The total iMAP framework is depicted above. Tracking is performed on each incoming frame to predict camera pose information. The feed-forward network is frozen during this process to make tracking more efficient. If a frame is chosen as a keyframe, it’s added to the set of keyframes which have their poses refined and are used to coach the scene network.
To make it more efficient, mapping runs in parallel to the tracking process and only samples training data from the set of keyframes, which is way smaller than the whole variety of incoming images. Plus, the loss is simply computed over a small subset of pixels, sampled using a hierarchical strategy (i.e., lively sampling) that identifies regions of the image with the best loss values (i.e., dense or detailed regions within the scene) and prioritizes sampling in these areas; see above.
online learning. Recall that iMAP is randomly-initialized at first. Because the RGB-D camera moves through a scene, iMAP (i) tracks the camera via the tracking module and (ii) begins choosing keyframes to update the mapping module. This set of keyframes acts as a replay buffer for iMAP. Namely, the feed-forward scene network (and associated camera poses) are trained in a web based fashion by aggregating relevant keyframes and performing updates over this data in parallel to the tracking process. Each update within the mapping process considers five frames: three random keyframes, the newest keyframe, and the present, incoming RGB-D frame.
The feed-forward scene network doesn’t suffer from catastrophic forgetting since it samples training data from a various set of keyframes as a substitute of coaching directly over the incoming data stream only. Unlike vanilla replay mechanisms, nonetheless, iMAP follows a selected strategy for sampling training data from the replay buffer. Specifically, keyframes with higher loss values are prioritized (i.e., keyframe lively sampling); see above.
iMAP is evaluated and in comparison with several traditional SLAM baselines across each real-world (i.e., from a handheld RGB-D camera) and artificial scene datasets (e.g., the Replica dataset). Notably, iMAP processes every incoming frame at a frequency of 10 Hz. For evaluation, mesh reconstructions will be recovered (and in comparison with ground truth if available) by querying the neural network over a voxel grid and running the marching cubes algorithm.
On synthetic scenes, we see that iMAP is incredibly capable at jointly creating coherent scene reconstructions (see below) and accurate camera tracks (see above). The truth is, iMAP is even found to be more capable than baselines at accurately filling in unobserved portions of a scene. Such a profit is on account of the flexibility of deep learning to leverage prior data and inductive biases to handle ambiguity in an affordable/predictable manner.
iMAP accurately performs tracking and mapping concurrently on account of its joint optimization of 3D scene representations and camera poses in every frame. Despite starting the training process from a totally random initialization, iMAP quickly learns accurate scene representations and tracking information. Most notably, the iMAP framework will be used at any scale, from small objects to a complete room that accommodates various, detailed objects; see below.
Despite crafting a comparably-accurate scene representation, the memory usage of iMAP is kind of low in comparison with SLAM baselines. iMAP just needs enough memory to store keyframes (including associated data) and the parameters of the neural network; see below.
On real-world data from a handheld RGB-D camera, iMAP continues to outperform SLAM baselines. Specifically, iMAP is surprisingly effective at accurately rendering regions of a scene where the depth camera has inaccurate readings (i.e., this tends to occur for black, reflective, or transparent surfaces); see below.
In comparison with most techniques now we have seen to this point, iMAP is kind of different. Specifically, it’s a SLAM system, which implies that it performs localization along with constructing a scene representation. Moreover, it relies upon online learning techniques to learn all the pieces a couple of scene in real-time. iMAP begins from a random initialization and learns to represent the underlying scene from scratch! Some major takeaways are as follows.
super fast! Most techniques now we have scene for representing 3D scenes are quite expensive. For instance, NeRFs take 2 days to coach on a single GPU, while LLFF performs ~10 minutes of preprocessing before generating novel scene views. iMAP learns all the pieces in real time as recent images are made available to it from an RGB-D camera. This can be a massive change within the computational cost of crafting scene representations. Some timing data of iMAP tracking and mapping pipelines is shown above. iMAP is implemented in PyTorch and may run on a Desktop CPU/GPU system.
why deep learning? Once we compare the iMAP system to other SLAM baselines, we see that it’s in a position to “fill in” blank spaces which might be left by the opposite systems. Put simply, iMAP can reasonably infer the contents of regions which have not been explicitly observed in a scene. Such a capability is on account of using a deep, feed-forward neural network, which leverages priors throughout the data/architecture to estimate geometry from limited data.
learning on the fly. When starting to find out about a scene, iMAP has no information. The truth is, it begins from a totally random initialization, then uses online learning techniques to learn a representation of the scene on the fly from incoming data. It’s very surprising that an approach like this works well, provided that prior approaches (e.g., NeRF) require several days of coaching to represent a scene. iMAP shows us that there could also be shortcuts or more lightweight techniques that allow us to acquire high-quality scene representations more easily.
positional embeddings. Although a smaller point, we see with iMAP that positional encodings, originally seen in NeRFs, have gotten standard. Recall that converting input coordinates into higher-dimensional positional embeddings before passing as input to a feed-forward network enables learning of high frequency features more easily. Here, we again see positional embeddings being adopted, revealing that such an approach is becoming standard.
- Understanding NeRFs https://towardsdatascience.com/imap-modeling-3d-scenes-in-real-time-6202365d80ee?source=rss—-7f60cf5620c9—4
- Local Light Field Fusion https://towardsdatascience.com/imap-modeling-3d-scenes-in-real-time-6202365d80ee?source=rss—-7f60cf5620c9—4
- Scene Representation Networks https://towardsdatascience.com/imap-modeling-3d-scenes-in-real-time-6202365d80ee?source=rss—-7f60cf5620c9—4
- Shape Reconstruction with ONets https://towardsdatascience.com/imap-modeling-3d-scenes-in-real-time-6202365d80ee?source=rss—-7f60cf5620c9—4
- 3D Generative Modeling with DeepSDF https://towardsdatascience.com/imap-modeling-3d-scenes-in-real-time-6202365d80ee?source=rss—-7f60cf5620c9—4
Thanks a lot for reading this text. I’m Cameron R. Wolfe, Director of AI at Rebuy and PhD student at Rice University. I study the empirical and theoretical foundations of deep learning. You may also try my other writings on medium! When you liked it, please follow me on twitter or subscribe to my Deep (Learning) Focus newsletter, where I help readers construct a deeper understanding of topics in deep learning research via comprehensible overviews of popular papers on that topic.
[1] Sucar, Edgar, et al. “iMAP: Implicit mapping and positioning in real-time.” Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.
[2] Mildenhall, Ben, et al. “Nerf: Representing scenes as neural radiance fields for view synthesis.” Communications of the ACM 65.1 (2021): 99–106.
[3] Sitzmann, Vincent, Michael Zollhöfer, and Gordon Wetzstein. “Scene representation networks: Continuous 3d-structure-aware neural scene representations.” Advances in Neural Information Processing Systems 32 (2019).
[4] Park, Jeong Joon, et al. “Deepsdf: Learning continuous signed distance functions for shape representation.” Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019.
[5] Tancik, Matthew, et al. “Fourier features let networks learn high frequency functions in low dimensional domains.” Advances in Neural Information Processing Systems 33 (2020): 7537–7547.
[6] Kemker, Ronald, et al. “Measuring catastrophic forgetting in neural networks.” Proceedings of the AAAI conference on artificial intelligence. Vol. 32. №1. 2018.