As a automotive travels along a narrow city street, reflections off the glossy paint or side mirrors of parked vehicles might help the driving force glimpse things that might otherwise be hidden from view, like a baby playing on the sidewalk behind the parked cars.

Drawing on this concept, researchers from MIT and Rice University have created a pc vision technique that leverages reflections to image the world. Their method uses reflections to show glossy objects into “cameras,” enabling a user to see the world as in the event that they were searching through the “lenses” of on a regular basis objects like a ceramic coffee mug or a metallic paper weight.   

Using images of an object taken from different angles, the technique converts the surface of that object right into a virtual sensor which captures reflections. The AI system maps these reflections in a way that allows it to estimate depth within the scene and capture novel views that might only be visible from the item’s perspective. One could use this system to see around corners or beyond objects that block the observer’s view.

This method could possibly be especially useful in autonomous vehicles. As an example, it could enable a self-driving automotive to make use of reflections from objects it passes, like lamp posts or buildings, to see around a parked truck.

“We’ve got shown that any surface could be converted right into a sensor with this formulation that converts objects into virtual pixels and virtual sensors. This could be applied in many various areas,” says Kushagra Tiwary, a graduate student within the Camera Culture Group on the Media Lab and co-lead writer of a paper on this research.

Tiwary is joined on the paper by co-lead writer Akshat Dave, a graduate student at Rice University; Nikhil Behari, an MIT research support associate; Tzofi Klinghoffer, an MIT graduate student; Ashok Veeraraghavan, professor of electrical and computer engineering at Rice University; and senior writer Ramesh Raskar, associate professor of media arts and sciences and leader of the Camera Culture Group at MIT. The research shall be presented on the Conference on Computer Vision and Pattern Recognition.

Reflecting on reflections

The heroes in crime television shows often “zoom and enhance” surveillance footage to capture reflections — perhaps those caught in a suspect’s sunglasses — that help them solve a criminal offense. 

“In real life, exploiting these reflections is just not as easy as just pushing an enhance button. Getting useful information out of those reflections is pretty hard because reflections give us a distorted view of the world,” says Dave.

This distortion is determined by the form of the item and the world that object is reflecting, each of which researchers could have incomplete details about. As well as, the glossy object could have its own color and texture that mixes with reflections. Plus, reflections are two-dimensional projections of a three-dimensional world, which makes it hard to guage depth in reflected scenes.

The researchers found a approach to overcome these challenges. Their technique, often called ORCa (which stands for Objects as Radiance-Field Cameras), works in three steps. First, they take pictures of an object from many vantage points, capturing multiple reflections on the glossy object.

Then, for every image from the true camera, ORCa uses machine learning to convert the surface of the item right into a virtual sensor that captures light and reflections that strike each virtual pixel on the item’s surface. Finally, the system uses virtual pixels on the item’s surface to model the 3D environment from the standpoint of the item.

Catching rays

Imaging the item from many angles enables ORCa to capture multiview reflections, which the system uses to estimate depth between the glossy object and other objects within the scene, along with estimating the form of the glossy object. ORCa models the scene as a 5D radiance field, which captures additional information concerning the intensity and direction of sunshine rays that emanate from and strike each point within the scene.

The extra information contained on this 5D radiance field also helps ORCa accurately estimate depth. And since the scene is represented as a 5D radiance field, relatively than a 2D image, the user can see hidden features that might otherwise be blocked by corners or obstructions.

In reality, once ORCa has captured this 5D radiance field, the user can put a virtual camera anywhere within the scene and synthesize what that camera would see, Dave explains. The user could also insert virtual objects into the environment or change the looks of an object, akin to from ceramic to metallic.

“It was especially difficult to go from a 2D image to a 5D environment. You might have to make sure that that mapping works and is physically accurate, so it relies on how light travels in space and the way light interacts with the environment. We spent lots of time fascinated about how we will model a surface,” Tiwary says.

Accurate estimations

The researchers evaluated their technique by comparing it with other methods that model reflections, which is a rather different task than ORCa performs. Their method performed well at separating out the true color of an object from the reflections, and it outperformed the baselines by extracting more accurate object geometry and textures.

They compared the system’s depth estimations with simulated ground truth data on the actual distance between objects within the scene and located ORCa’s predictions to be reliable.   

“Consistently, with ORCa, it not only estimates the environment accurately as a 5D image, but to attain that, within the intermediate steps, it also does a great job estimating the form of the item and separating the reflections from the item texture,” Dave says.

Constructing off of this proof-of-concept, the researchers need to apply this system to drone imaging. ORCa could use faint reflections from objects a drone flies over to reconstruct a scene from the bottom. Additionally they want to reinforce ORCa so it will possibly utilize other cues, akin to shadows, to reconstruct hidden information, or mix reflections from two objects to image latest parts of a scene.

“Estimating specular reflections is de facto essential for seeing around corners, and that is the following natural step to see around corners using faint reflections within the scene,” says Raskar.

“Ordinarily, shiny objects are difficult for vision systems to handle. This paper may be very creative because it turns the longstanding weakness of object shininess into a bonus. By exploiting environment reflections off a shiny object, the paper is just not only capable of see hidden parts of the scene, but in addition understand how the scene is lit. This allows applications in 3D perception that include, but usually are not limited to, a capability to composite virtual objects into real scenes in ways in which appear seamless, even in difficult lighting conditions,” says Achuta Kadambi, assistant professor of electrical engineering and computer science on the University of California at Los Angeles, who was not involved with this work. “One reason that others haven’t been capable of use shiny objects on this fashion is that almost all prior works require surfaces with known geometry or texture. The authors have derived an intriguing, latest formulation that doesn’t require such knowledge.”

The research was supported, partly, by the Intelligence Advanced Research Projects Activity and the National Science Foundation.