TY  - BOOK
AU  - Karim Magdy Amer
TI  - Self-Supervised Learning Framework for Sequential Data Applications / 
U1  - 610 
PY  - 2021///
KW  - Informatics-IFM
KW  - NULIB
KW  - Dissertation, Academic
N1  - Thesis (M.A.)—Nile University, Egypt, 2021; "Includes bibliographical references"; Contents:
Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iii
Dedication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iv
Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . v
List of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viii
List of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix
Chapters:
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1 Overview and Motivation . . . . . . . . . . . . . . . . . . . . 1
1.2 Problem Statement . . . . . . . . . . . . . . . . . . . . . . . . 3
1.3 Objectives and Contributions . . . . . . . . . . . . . . . . . . 4
1.4 Organization of Thesis . . . . . . . . . . . . . . . . . . . . . . 5
2. Literature Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.2 Deep Neural Network Architectures . . . . . . . . . . . . . . . 7
2.2.1 Convolutional Neural Network . . . . . . . . . . . . . 7
2.2.2 Sequence Models . . . . . . . . . . . . . . . . . . . . . 9
2.3 Self Supervised Learning . . . . . . . . . . . . . . . . . . . . . 11
2.3.1 Computer Vision . . . . . . . . . . . . . . . . . . . . . 11
2.3.2 Natural Language Processing . . . . . . . . . . . . . . 21
2.3.3 Automatic Speech Recognition . . . . . . . . . . . . . 23
2.4 Sequential Data Applications using Machine Learning and
Deep Learning . . . . . . . . . . . . . . . . . . . . . . . . . . 26
2.4.1 Bioinformatics . . . . . . . . . . . . . . . . . . . . . . 26
2.4.2 Agriculture . . . . . . . . . . . . . . . . . . . . . . . . 26
2.5 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
vi
3. Self-Supervised Learning Framework for Sequential Data . . . . . . 29
3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
3.2 Framework Architecture . . . . . . . . . . . . . . . . . . . . . 29
3.3 Self Supervised Learning Modules . . . . . . . . . . . . . . . . 31
3.3.1 Denoising Autoencoder Module . . . . . . . . . . . . . 31
3.3.2 Contrastive Learning Module . . . . . . . . . . . . . . 33
3.4 Supervised Learning Module . . . . . . . . . . . . . . . . . . . 35
3.5 Deep Neural Network Module . . . . . . . . . . . . . . . . . . 36
3.6 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
4. Experiments and Results . . . . . . . . . . . . . . . . . . . . . . . . 37
4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
4.2 Case Study: Crop Identification . . . . . . . . . . . . . . . . . 37
4.2.1 Problem Description . . . . . . . . . . . . . . . . . . . 37
4.2.2 Dataset and Evaluation Metrics . . . . . . . . . . . . . 39
4.2.3 Experimental setup . . . . . . . . . . . . . . . . . . . . 44
4.2.4 Results . . . . . . . . . . . . . . . . . . . . . . . . . . 48
4.3 Case Study: mRNA Vaccine Degradation Prediction . . . . . 51
4.3.1 Problem Description . . . . . . . . . . . . . . . . . . . 51
4.3.2 Dataset and Evaluation Metrics . . . . . . . . . . . . . 52
4.3.3 Experimental Setup . . . . . . . . . . . . . . . . . . . 56
4.3.4 Results . . . . . . . . . . . . . . . . . . . . . . . . . . 59
4.4 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
5. Conclusions and Future Work . . . . . . . . . . . . . . . . . . . . . 63
5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
5.2 Summary and Contributions . . . . . . . . . . . . . . . . . . . 63
5.3 Future Work Directions . . . . . . . . . . . . . . . . . . . . . 64
Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
N2  - Abstract:
Recent advances in Deep Learning (DL) and Artificial Intelligence (AI) algorithms have proven to be greatly beneficial in many applications. The
success of such algorithms in the last decade was mainly built on supervised training of very complex models using large labeled datasets. However, the data labeling process is expensive and time consuming especially for applications with sequential data.
Self-Supervised Learning (SSL) appeared as a solution to train complex
models with unlabeled datasets. The main idea behind SSL is to create auxiliary
task based on some properties in the dataset to learn a latent space
that captures the data semantics. Models trained using SSL can be fine-tuned
later on a much smaller labeled datasets. This learning scheme becomes very
successful in Computer Vision (CV) and Natural Language Processing (NLP)
applications but hasn’t yet generalized to other sequential data applications.
In this work, we propose a novel DL framework for sequential data processing
while adopting SSL techniques to make use of unlabeled data. Our framework
is evaluated on two different applications from two different domains,
Crop Identification and mRNA Vaccine Degradation Prediction. Details of
the applications’ datasets, pre-processing steps and learning objectives of both
applications are explained. Experimental results demonstrate that the proposed
framework offers a promising approach for utilizing SSL in the training
pipeline of commonly-used DL models
ER  -