Geosemantic Segmentation
| Paper: | Geo-semantic Segmentation |
| Contact: | Shervin Ardeshir, Kofi Malcolm Collins-Sibley, Mubarak Shah |
We propose a method for performing geo-referenced semantic segmentation for an image by using the information acquired from GIS databases.
The main steps of our framework are illustrated in the following block diagram.
Given a database of object locations from GIS and a query image with metadata, we project the buildings and streets that are in the fiwld of view of the camera. An example is shown in the following figure.
We also divide the image into a set of initial super-pixels.
Then, we fuse the GIS projections with the super-pixel segmentation results and evaluate the probability of each super-pixel for belonging to different projected entities. For that purpose we perform multiple random walks on the graph constructed over the super-pixels. The matlab code for parallel random walks can be found here.
In order to cope with various inaccuracies and practical complications, such as noisy metadata, occlusion, inaccuracies in GIS, we update the projections iteratively, by evaluating the quality of the projections, and aligning them with their corresponding semantic segments.
The following figure illustrates the improvement in the semantic segmentation accuracy before and after the alignment.
PDF file of the paper can be downloaded here .
GIS dataset for building outlines and objects can be downloaded here .
Dataset of images can be downloaded here.
Code can be downloaded here .
Power point presentation can be downloaded here .
Shervin Ardeshir, Kofi Malcolm Collins-Sibley and Mubarak Shah, “Geo-semantic Segmentation”, in Proceedings of IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), June 2015 [PDF | Project Page]
Take a look at a few of our other papers and projects in the area of geo-spatial analysis of images and videos:
Shervin Ardeshir, Amir Roshan Zamir, Alejandro Torroella, and Mubarak Shah, “GIS-Assisted Object Detection and Geospatial Localization”, in Proceedings of IEEE European Conference on Computer Vision (ECCV), September 2014 [PDF | Project Page]
Amir Roshan Zamir, Shervin Ardeshir and Mubarak Shah, “Robust Refinement of GPS-Tags using Random Walks with an Adaptive Damping Factor”, in Proceedings of IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), June 2014 [PDF | Project Page]
Amir Roshan Zamir, Afshin Dehghan and Mubarak Shah, “Visual Business Recognition – A Multimodal Approach”, In Proceeding of ACM International Conf. on Multimedia (ACM MM), 2013 [PDF | BibTeX | Supplemental Mat.]
Take a look at a few of our other papers and projects in the area of geo-spatial analysis of images and videos:
Amir Roshan Zamir, Shervin Ardeshir and Mubarak Shah, “Robust Refinement of GPS-Tags using Random Walks with an Adaptive Damping Factor”, in Proceedings of IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), June 2014 [PDF | Project Page]
Amir Roshan Zamir and Mubarak Shah, “Image Geo-localization Based on Multiple Nearest Neighbor Feature Matching using Generalized Graphs”, In IEEE Transactions on Pattern Analysis and Machine Intelligence (T-PAMI), 2014 [PDF | Project Page]
Gonzalo Vaca, Amir Roshan Zamir and Mubarak Shah, “City Scale Geo-spatial Trajectory Estimation of a Moving Camera”, in Proceedings of IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), June 2012 [PDF | BibTeX| Project Page]
Amir Roshan Zamir and Mubarak Shah, “Accurate Image Localization Based on Google Maps Street View”, In Proceedings of European Conference on Computer Vision (ECCV), 2010 [PDF | BibTeX | Project Page]