The entire pipeline for producing digital tours (known as DIGITOUR and shown in Figure 3) is as follows.
2.1 Tag Placement and Image Capturing
While making a digital tour for any real-estate property, it is important to click 360β¦ images from different property locations comparable to bedroom, lounge, kitchen, etc., then routinely stitching them together to have a βwalkthroughβ experience without being physically present at the situation. Subsequently, to attach multiple equirectangular images, we propose placing paper tags on the ground covering each location of the property, and placing the camera (in our case, we used Ricoh-Theta) in the midst of the scene to capture the entire site (front, back, left, right and bottom).
Furthermore, we be sure that the scene is evident of all noisy elements comparable to dim lighting and βunwantedβ artifacts for higher model training and inference. As shown in Figure 4, we’ve got standardized the tags with dimensions of 6β Γ 6β with two properties:
- they’re numbered which is able to help the photographer place tags in sequence and
- they’re bi-colored to formulate the digit recognition problem as classification task and facilitate higher learning of downstream computer vision tasks (i.e. tag detection and digit recognition).
Please note that different colours are assigned to every digit (from 0 to 9) using the HSV color scheme and leading digit of a tag has a black circle to differentiate it from the trailing digit as shown in Figure 4. The intuition behind standardizing the paper tags is that it allows to coach tag detection and digit recognition models, that are invariant to distortions, tag placement angle, reflection from lighting sources, blur conditions, and camera quality.
2.2 Mapping Equirectangular Image to Cubemap Projection
An equirectangular image consists of a single image whose width and height correlate as 2 : 1 (as shown in Figure 1). In our case, images are clicked using a Ricoh-Theta camera having dimensions 4096 Γ 2048 Γ 3. Typically, each point in an equirectangular image corresponds to a degree in a sphere, and the pictures are stretched within the βlatitudeβ direction. Because the contents of an equirectangular image are distorted, it becomes difficult to detect tags and recognize digits directly from it. For instance, in Figure 1, the tag is stretched on the middle-bottom of the image. Subsequently, it’s obligatory to map the image to a less-distorted projection and switch back to the unique equirectangular image to construct the digital tour.
On this work, we propose to make use of cube map projection, which is a set of six images representing six faces of a cube. Here, every point within the spherical coordinate space corresponds to a degree within the face of the cube. As shown in Figure 5, we map the equirectangular image to 6 faces (left, right, front, back, top and bottom) of a cube having dimensions 1024 Γ 1024 Γ 3 using python library vrProjector.
2.3 Tag Detection
Once we get the six images corresponding to the faces of a cube, we detect the situation of tags placed in each image. For tag detection, we’ve got used the state-of-the-art YOLOv5 model. We initialized the network with COCO weights followed by training on our dataset. As shown in Figure 6, the model takes a picture as input and returns the detected tag together with coordinates of the bounding box and confidence of the prediction. The model is trained on our dataset for 100 epochs with a batch size of 32.
2.4 Digit Recognition
For the detected tags, we want to acknowledge the digits from the tag. In a real-world environment, the detected tags might need incorrect orientation, poor luminosity, reflection from the bulbs within the room, etc. On account of these reasons, it’s difficult to make use of Optical Character Recognition (OCR) engines to have good digit recognition performance. Subsequently, we’ve got used a custom MobileNet model initialized on Imagenet weights, which uses color information in tags for digit recognition. Within the proposed architecture, we’ve got replaced the ultimate classification block of the unique MobileNet with the dropout layer and dense layer with 20 nodes representing our tags from 1 to twenty. Figure 7 illustrates the proposed architecture. For training the model, we’ve got used Adam as an optimizer with a learning rate of 0.001 and a discounting factor (π) to be 0.1. We’ve got used categorical cross-entropy as a loss function and set the batch size to 64 and the variety of epochs to 50.
2.5 Mapping tag coordinates to the unique 360β¦ Image and Virutal Tour Creation
Once we’ve got detected the tags and recognized the digits we use the python library vrProjector to map the cube map coordinates back to the unique equirectangular image. An example output is shown in Figure 8. For every equirectangular image, the detected tags form the nodes of a graph with an edge between them. In the next equirectangular images of a property, the graph gets populated with more nodes, as more tags are detected. Finally, we connect multiple equirectangular images in sequence based on recognized digits written on them and the resulting graph is the
virtual tour as shown in Figure 2(b).
We’ve got collected data by placing tags and clicking equirectangular images using Ricoh-Theta camera for several residential properties in Gurugram, India (Tier 1 city). While collecting images we made sure that certain conditions were met comparable to all doors were opened, lights were turned on, βunwantedβ objects were removed and the tags were placed covering each area of the property. Following these instructions, average variety of equirectangular images clicked per residential property was 7 or 8. Finally, we’ve got validated our approach on the next generated datasets (based on background color of the tags).
- Green Coloured Tags: We’ve got kept the background color of those tags (numbered 1 to twenty) to be green. We’ve got collected 1572 equirectangular images from 212 properties. Once we convert these equirectangular images to cubemap projection, we get 9432 images (corresponding to cube faces). Since not all the cube faces have tags (for e.g. top face), we get 1503 images with atleast one tag.
- Proposed Bi-colored Tags (see Figure 4): For these tags, we’ve got collected 2654 equirectangular images from 350 properties. Finally, we got 2896 images (corresponding to cube faces) with atleast one tag.
Finally, we label the tags present in cube map projection images using LabelImg which is an open-source tool for labeling images in several formats comparable to Pascal VOC and YOLO. For all of the experiments, we reserved 20% of knowledge for testing and the remaining for training.
For any input image, we first detect the tags and at last recognize the digits written on the tags. From this we were capable of discover the true positives (tags detected and browse appropriately), false positives (tags detected but read incorrectly) and false negatives (tags not detected). The obtained mAP, Precision, Recall and f1-score at 0.5 IoU threshold are 88.12, 93.83, 97.89 and 95.81 respectively. Please note that every one metrics are averaged (weighted) over all of the 20 classes. If all tags across all equirectangular images of a property are detected and browse appropriately, we receive a 100% accurate virtual tour since all nodes of the graph are detected and connected with their appropriate edges. In our experiments, we were capable of accurately generate 100% accurate virtual tour for 94.55% of the properties. The inaccuracies were on account of the presence of colourful artifacts that were falsely detected as tags; and bad lightning conditions.
Figure 9 demonstrates the performance of Yolov5 model for tag detection based on green coloured and bi-colored tags. Further, experiments and comparison of models on digit recognition is shown in Figure 10.
We propose an end-to-end pipeline (DIGITOUR) for routinely generating digital tours for real-estate properties. For any such property, we first place the proposed bi-colored paper tags covering each area of the property. Then, we click equirectangular images, followed by mapping these images to less distorted cubemap images. Once we get the six images corresponding to cube faces, we detect the situation of tags using the YOLOv5 model, followed by digit recognition using the MobileNet model. The subsequent step is to map the detected coordinates together with recognized digits to the unique equirectangular images. Finally, we stitch together all of the equirectangular images to construct a virtual tour. We’ve got validated our pipeline on a real-world dataset and shown that the end-to-end pipeline performance is 88.12 and 95.81 by way of mAP and f1-score at 0.5 IoU threshold averaged (weighted) over all classes.
In case you find our work helpful and put it to use in your projects, we kindly request that you just cite it. π
@inproceedings{chhikara2023digitour,
title={Digitour: Automatic digital tours for real-estate properties},
writer={Chhikara, Prateek and Kuhar, Harshul and Goyal, Anil and Sharma, Chirag},
booktitle={Proceedings of the sixth Joint International Conference on Data Science & Management of Data (tenth ACM IKDD CODS and twenty eighth COMAD)},
pages={223--227},
yr={2023}
}
[1] Dragomir Anguelov, Carole Dulong, Daniel Filip, Christian Frueh, StΓ©phane Lafon, Richard Lyon, Abhijit Ogale, Luc Vincent, and Josh Weaver. 2010. Google street view: Capturing the world at street level. Computer 43, 6 (2010), 32β38.
[2] Mohamad Zaidi Sulaiman, Mohd Nasiruddin Abdul Aziz, Mohd Haidar Abu Bakar, Nur Akma Halili, and Muhammad Asri Azuddin. 2020. Matterport: virtual tour as a brand new marketing approach in real estate business during pandemic COVID-19. In International Conference of Innovation in Media and Visual Design (IMDES 2020). Atlantis Press, 221β226.
[3] Chinu Subudhi. 2021. Cutting-Edge 360-Degree Virtual Tours. https://www.mindtree.com/insights/resources/cutting-edge-360-degree-virtual-tours
[4] Glenn Jocher, Ayush Chaurasia, Alex Stoken, Jirka Borovec, NanoCode012, Yonghye Kwon, TaoXie, Jiacong Fang, imyhxy, Kalen Michael, Lorna, Abhiram V, Diego Montes, Jebastin Nadar, Laughing, tkianai, yxNONG, Piotr Skalski, Zhiqiang Wang, Adam Hogan, Cristi Fati, Lorenzo Mammana, AlexWang1900, Deep Patel, Ding Yiwei, Felix You, Jan Hajek, Laurentiu Diaconu, and Mai Thanh Minh. 2022. ultralytics/yolov5: v6.1 β TensorRT, TensorFlow Edge TPU and OpenVINO Export and Inference.
[5] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and LiangChieh Chen. 2018. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition. 4510β4520.