Verification of Machine Learning/Deep Learning Based Pedestrian Detectors /Ahmed Hosny Abdel-Gawad Ahmed

Supervisor:
Ahmed G. Radwan

Thesis (M.A.)—Nile University, Egypt, 2024 .

"Includes bibliographical references"

Contents:
Chapters:
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.2 Importance of Robustness . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.3 Current Challenges and Limitations . . . . . . . . . . . . . . . . . . . . . 9
1.4 Problem Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.5 Research Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
1.6 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
1.7 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
1.8 Thesis Outline and Summary of Contributions . . . . . . . . . . . . . . . . 15
2. Literature Survey . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.2 Sensors for Environment Perception . . . . . . . . . . . . . . . . . . . . . 18
2.2.1 Visible-light Imaging . . . . . . . . . . . . . . . . . . . . . . . . . . 24
2.2.2 Thermal Imaging . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
2.3 Benchmark Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
2.4 Pedestrian Detection Techniques: A Brief History . . . . . . . . . . . . . . 32
2.5 Typical Detection Pipeline . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
2.5.1 Regions of Interest . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
2.5.2 Feature Extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
2.5.3 Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
2.6 Deep Learning for Pedestrians Detection . . . . . . . . . . . . . . . . . . . 37
2.7 Deep Learning Architectures for pedestrians Detection . . . . . . . . . . . 41
2.7.1 Two-Stage Detectors . . . . . . . . . . . . . . . . . . . . . . . . . . 42
2.7.2 Single Stage Detectors . . . . . . . . . . . . . . . . . . . . . . . . . 45
2.8 Pedestrians Detection Challenges . . . . . . . . . . . . . . . . . . . . . . . 46
2.8.1 Image Corruption . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
2.8.2 Pedestrian Occlusion . . . . . . . . . . . . . . . . . . . . . . . . . . 51
3. Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
3.1 Baseline Development . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
ii

3.1.1 Dataset Selection and Preparation . . . . . . . . . . . . . . . . . . 56
3.1.2 Model Selection and Training . . . . . . . . . . . . . . . . . . . . . 59
3.2 Baseline Corruption Robustness . . . . . . . . . . . . . . . . . . . . . . . . 63
3.2.1 Selection of Image Corruptions . . . . . . . . . . . . . . . . . . . . 64
3.2.2 Image Quality Assessment . . . . . . . . . . . . . . . . . . . . . . . 65
3.2.3 Artificial Generation of Corrupted Data . . . . . . . . . . . . . . . 68
3.2.4 Noise corruptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
3.2.5 Blur corruptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
3.2.6 Digital corruptions . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
3.2.7 Weather corruptions . . . . . . . . . . . . . . . . . . . . . . . . . . 75
3.2.8 Baseline Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . 78
3.3 Enhancing Robustness Against Image Corruptions . . . . . . . . . . . . . 79
3.3.1 Robust Training . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
3.3.2 Stylized Training . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
3.4 Baseline Occlusion Robustness . . . . . . . . . . . . . . . . . . . . . . . . 84
3.4.1 Naturally Occluded JAAD Dataset . . . . . . . . . . . . . . . . . . 84
3.4.2 Baseline Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . 84
3.5 Enhancing Robustness Against Pedestrian Occlusion . . . . . . . . . . . . 85
3.5.1 Artificial Generation of Occluded Datasets . . . . . . . . . . . . . . 86
3.5.2 Occluded Training . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
4. Results and Discussions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
4.1 Evaluation Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
4.1.1 Detection Performance Metrics . . . . . . . . . . . . . . . . . . . . 89
4.1.2 Corruption Robustness Evaluation Metrics . . . . . . . . . . . . . . 91
4.1.3 Occlusion Robustness Evaluation Metrics . . . . . . . . . . . . . . 92
4.1.4 Computational Efficiency: Inference Time . . . . . . . . . . . . . . 94
4.2 Baseline Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
4.2.1 Performance on Corrupted Dataset . . . . . . . . . . . . . . . . . . 96
4.2.2 Performance on Occluded Dataset . . . . . . . . . . . . . . . . . . 106
4.3 Enhancing Corruption Robustness through Robust Training . . . . . . . . 110
4.3.1 Training with All Corruptions as Augmentations . . . . . . . . . . 111
4.3.2 Training on Each Category Alone . . . . . . . . . . . . . . . . . . . 120
4.4 Enhancing Corruption Robustness through Stylized Training . . . . . . . . 131
4.4.1 Training on Stylized data only . . . . . . . . . . . . . . . . . . . . 131
4.4.2 Training on both Clean and Stylized data combined . . . . . . . . 139
4.5 Enhancing Occlusion Robustness through Occluded Training . . . . . . . . 148
5. Conclusion and Future work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152
5.1 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152
5.2 Future directions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154
Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

Abstract:
In the context of assisted and automated driving vehicles and advanced driver-assistance
systems underscores the critical need for accurate pedestrian detection to ensure road safety.
However, deploying pedestrian detection models in real-world scenarios poses a significant
challenge due to image corruptions and partial occlusions. This thesis addresses these challenges through the development and evaluation of robust, stylized, and occluded training
techniques.
Robust Training incorporates intentionally generated corrupted examples during model
training to simulate real-world challenges. Results demonstrate significant improvements
in model robustness to various corruption types. Further refinement through categorized
training data emphasizes the potential benefits of tailored augmentation to specific corruption
profiles.
Stylized Training leverages the Adaptive Instance Normalization (AdaIN) method to introduce variations in texture and style, enriching the training dataset. Empirical results showcase a substantial improvement in pedestrian detection accuracy, particularly in mean Average Precision (mAP50) on clean data. Stylized training significantly enhances the model’s
robustness in adverse conditions, demonstrating proficiency in handling noise, blur, digital,
and weather-related distortions.
Occluded Training introduces a novel methodology for generating occluded datasets, simulating scenarios with varying levels of occlusion. The augmented training approach leads to
a substantial enhancement in performance when faced with occluded scenarios. Specifically,
the model exhibits a ∼2% increase in mean Average Precision (mAP50) on both partially and
fully occluded samples compared to standard training. This underscores the effectiveness of
occlusion training in fortifying the model’s performance in occluded real-world scenarios.
The utilization of robust, stylized, and occluded training techniques significantly advances the
adaptability and performance of pedestrian detection models in diverse real-world conditions
and corruption scenarios, with an improvement of ∼2-4%. These findings establish a robust
foundation for deploying pedestrian detection models in complex environments, ensuring
safer and more reliable assisted and automated driving systems. This thesis contributes
to the forefront of pedestrian detection, underscoring the efficacy of tailored augmentation
strategies for model robustness in dynamic real-world environments.
Keywords— Assisted and Automated Vehicles, Pedestrian Detection, Image Corruptions, Robust
Training, Stylized Training, Occluded Training, Model Robustness

Text in English, abstracts in English and Arabic