Adeep cnn-based Framework for enhanced aerial imagery registration with applications to uav geo-localization /
Ahmed Nassar
Adeep cnn-based Framework for enhanced aerial imagery registration with applications to uav geo-localization / Ahmed Nassar - 2017 - 102 p. ill. 21 cm.
Supervisor: Mohamed A. El-Helw
Thesis (M.A.)—Nile University, Egypt, 2017 .
"Includes bibliographical references"
Contents:
1 Introduction 1
1.1 Overview and Motivation . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Objectives and Contributions . . . . . . . . . . . . . . . . . . . . . . 4
1.3 Organization of Thesis . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2 Review of Image Registration and Semantic Segmentation 7
2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.2 Earth Observation Image Registration . . . . . . . . . . . . . . . . . 8
2.2.1 About Earth Observation Images . . . . . . . . . . . . . . . . 8
2.2.2 Overview of Local Feature Detectors and Handcrafted Features 10
2.2.3 Matching Local Features . . . . . . . . . . . . . . . . . . . . . 17
2.2.4 Motion Estimation . . . . . . . . . . . . . . . . . . . . . . . . 19
2.3 Deep Learning and Semantic Segmentation . . . . . . . . . . . . . . . 21
2.3.1 Neural Networks . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.3.2 Convolutional Neural Networks . . . . . . . . . . . . . . . . . 25
2.3.3 Non-Linearity Functions . . . . . . . . . . . . . . . . . . . . . 26
2.3.4 Semantic Segmentation . . . . . . . . . . . . . . . . . . . . . . 30
2.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
3 Integrated Framework for Enhanced UAV Localization 35
3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
3.2 Framework Architecture . . . . . . . . . . . . . . . . . . . . . . . . . 36
3.3 Calibration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
3.4 Registration and Detection Using Semantic Segmentation . . . . . . . 41
3.4.1 Setting Up the Segmentation Network . . . . . . . . . . . . . 41
3.4.2 Registration by Shape Matching . . . . . . . . . . . . . . . . . 46
3.4.3 Detection by Semantic Segmentation . . . . . . . . . . . . . . 48
3.5 Registration of Sequence Frames . . . . . . . . . . . . . . . . . . . . . 50
3.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
4 Results and Analysis 53
4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
4.2 Platform . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
4.3 Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
iii
4.3.1 ISPRS WG II/4 Potsdam . . . . . . . . . . . . . . . . . . . . 55
4.3.2 Potsdam Extended . . . . . . . . . . . . . . . . . . . . . . . . 56
4.3.3 UAV Videos . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
4.4 Earth Observation Image Registration . . . . . . . . . . . . . . . . . 59
4.4.1 The Calibration Component Experiment . . . . . . . . . . . . 60
4.4.2 The UAV Sequence Experiment . . . . . . . . . . . . . . . . . 61
4.4.3 Local Feature Registration Results . . . . . . . . . . . . . . . 62
4.5 Semantic Segmentation . . . . . . . . . . . . . . . . . . . . . . . . . . 65
4.5.1 The Semantic Segmentation Experiment . . . . . . . . . . . . 65
4.5.2 Semantic Segmentation Results . . . . . . . . . . . . . . . . . 66
4.5.3 Semantic Segmentation Registration using Shape Matching Re-
sults . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
4.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
5 Conclusions and Future Work 75
5.1 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
5.2 Summary of Contributions . . . . . . . . . . . . . . . . . . . . . . . . 76
5.3 Future Work Directions . . . . . . . . . . . . . . . . . . . . . . . . . . 77
A Libraries and Frameworks 79
A.1 OpenCV . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
A.2 Keras . . . . .
Abstract:
Using the momentous advancement of deep learning in traditional computer vision
problems such as perception has been greatly benecial. Along with the availability of
high resolution Earth Observation imagery, extraction of information such as roads,
buildings, vegetation, etc., using deep learning is a major topic for remote sensing
applications. We transfer these methods to UAVs or drones equipped with an on-
board camera, and GPU.
In this thesis, we present a method for geo-localizing UAVs using only it's onboard
camera. A pipeline has been implemented that makes use of the availability of satellite
imagery, and traditional computer vision (SIFT), along with renowned deep learning
methods (semantic segmentation) to be able to locate the aerial image captured from
the UAV inside of the satellite imagery. A novel solution of matching segmented
shapes is used to register the segmented images, and increase the accuracy of the
system overall. Additionally, this method enables the UAV to be self-aware of its
surrounding and the context its in. Consequently, this allows UAVs to be able to
navigate without the use of GPS, which could be benecial in environments where
GPS is denied or weak, or be an additional enhancement module to GPS.
Our framework is evaluated on two dierent datasets generated from dierent
regions, to show the viability of the method in dierent situations. These datasets,
and the process of generating them is explained in the literature. Furthermore, we
believe our results show that the framework oers a promising approach for outdoor
navigation for UAVs using vision only.
Text in English, abstracts in English.
Informatics-IFM
Dissertation, Academic
610
Adeep cnn-based Framework for enhanced aerial imagery registration with applications to uav geo-localization / Ahmed Nassar - 2017 - 102 p. ill. 21 cm.
Supervisor: Mohamed A. El-Helw
Thesis (M.A.)—Nile University, Egypt, 2017 .
"Includes bibliographical references"
Contents:
1 Introduction 1
1.1 Overview and Motivation . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Objectives and Contributions . . . . . . . . . . . . . . . . . . . . . . 4
1.3 Organization of Thesis . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2 Review of Image Registration and Semantic Segmentation 7
2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.2 Earth Observation Image Registration . . . . . . . . . . . . . . . . . 8
2.2.1 About Earth Observation Images . . . . . . . . . . . . . . . . 8
2.2.2 Overview of Local Feature Detectors and Handcrafted Features 10
2.2.3 Matching Local Features . . . . . . . . . . . . . . . . . . . . . 17
2.2.4 Motion Estimation . . . . . . . . . . . . . . . . . . . . . . . . 19
2.3 Deep Learning and Semantic Segmentation . . . . . . . . . . . . . . . 21
2.3.1 Neural Networks . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.3.2 Convolutional Neural Networks . . . . . . . . . . . . . . . . . 25
2.3.3 Non-Linearity Functions . . . . . . . . . . . . . . . . . . . . . 26
2.3.4 Semantic Segmentation . . . . . . . . . . . . . . . . . . . . . . 30
2.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
3 Integrated Framework for Enhanced UAV Localization 35
3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
3.2 Framework Architecture . . . . . . . . . . . . . . . . . . . . . . . . . 36
3.3 Calibration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
3.4 Registration and Detection Using Semantic Segmentation . . . . . . . 41
3.4.1 Setting Up the Segmentation Network . . . . . . . . . . . . . 41
3.4.2 Registration by Shape Matching . . . . . . . . . . . . . . . . . 46
3.4.3 Detection by Semantic Segmentation . . . . . . . . . . . . . . 48
3.5 Registration of Sequence Frames . . . . . . . . . . . . . . . . . . . . . 50
3.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
4 Results and Analysis 53
4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
4.2 Platform . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
4.3 Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
iii
4.3.1 ISPRS WG II/4 Potsdam . . . . . . . . . . . . . . . . . . . . 55
4.3.2 Potsdam Extended . . . . . . . . . . . . . . . . . . . . . . . . 56
4.3.3 UAV Videos . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
4.4 Earth Observation Image Registration . . . . . . . . . . . . . . . . . 59
4.4.1 The Calibration Component Experiment . . . . . . . . . . . . 60
4.4.2 The UAV Sequence Experiment . . . . . . . . . . . . . . . . . 61
4.4.3 Local Feature Registration Results . . . . . . . . . . . . . . . 62
4.5 Semantic Segmentation . . . . . . . . . . . . . . . . . . . . . . . . . . 65
4.5.1 The Semantic Segmentation Experiment . . . . . . . . . . . . 65
4.5.2 Semantic Segmentation Results . . . . . . . . . . . . . . . . . 66
4.5.3 Semantic Segmentation Registration using Shape Matching Re-
sults . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
4.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
5 Conclusions and Future Work 75
5.1 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
5.2 Summary of Contributions . . . . . . . . . . . . . . . . . . . . . . . . 76
5.3 Future Work Directions . . . . . . . . . . . . . . . . . . . . . . . . . . 77
A Libraries and Frameworks 79
A.1 OpenCV . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
A.2 Keras . . . . .
Abstract:
Using the momentous advancement of deep learning in traditional computer vision
problems such as perception has been greatly benecial. Along with the availability of
high resolution Earth Observation imagery, extraction of information such as roads,
buildings, vegetation, etc., using deep learning is a major topic for remote sensing
applications. We transfer these methods to UAVs or drones equipped with an on-
board camera, and GPU.
In this thesis, we present a method for geo-localizing UAVs using only it's onboard
camera. A pipeline has been implemented that makes use of the availability of satellite
imagery, and traditional computer vision (SIFT), along with renowned deep learning
methods (semantic segmentation) to be able to locate the aerial image captured from
the UAV inside of the satellite imagery. A novel solution of matching segmented
shapes is used to register the segmented images, and increase the accuracy of the
system overall. Additionally, this method enables the UAV to be self-aware of its
surrounding and the context its in. Consequently, this allows UAVs to be able to
navigate without the use of GPS, which could be benecial in environments where
GPS is denied or weak, or be an additional enhancement module to GPS.
Our framework is evaluated on two dierent datasets generated from dierent
regions, to show the viability of the method in dierent situations. These datasets,
and the process of generating them is explained in the literature. Furthermore, we
believe our results show that the framework oers a promising approach for outdoor
navigation for UAVs using vision only.
Text in English, abstracts in English.
Informatics-IFM
Dissertation, Academic
610