000			06042nam a22002537a 4500
008			220110b2021 |||a|||f mb|| 00| 0 eng d
040			_aEG-CaNU _cEG-CaNU
041	0		_aeng _beng
082			_a610
100	0		_aKarim Magdy Amer _91058
245	1		_aSelf-Supervised Learning Framework for Sequential Data Applications / _cKarim Magdy Amer
260			_c2021
300			_a95 p. _bill. _c21 cm.
500			_3Supervisor: Mohamed Elhelw
502			_aThesis (M.A.)—Nile University, Egypt, 2021 .
504			_a"Includes bibliographical references"
505	0		_aContents: Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iii Dedication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iv Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . v List of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viii List of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix Chapters: 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.1 Overview and Motivation . . . . . . . . . . . . . . . . . . . . 1 1.2 Problem Statement . . . . . . . . . . . . . . . . . . . . . . . . 3 1.3 Objectives and Contributions . . . . . . . . . . . . . . . . . . 4 1.4 Organization of Thesis . . . . . . . . . . . . . . . . . . . . . . 5 2. Literature Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 2.2 Deep Neural Network Architectures . . . . . . . . . . . . . . . 7 2.2.1 Convolutional Neural Network . . . . . . . . . . . . . 7 2.2.2 Sequence Models . . . . . . . . . . . . . . . . . . . . . 9 2.3 Self Supervised Learning . . . . . . . . . . . . . . . . . . . . . 11 2.3.1 Computer Vision . . . . . . . . . . . . . . . . . . . . . 11 2.3.2 Natural Language Processing . . . . . . . . . . . . . . 21 2.3.3 Automatic Speech Recognition . . . . . . . . . . . . . 23 2.4 Sequential Data Applications using Machine Learning and Deep Learning . . . . . . . . . . . . . . . . . . . . . . . . . . 26 2.4.1 Bioinformatics . . . . . . . . . . . . . . . . . . . . . . 26 2.4.2 Agriculture . . . . . . . . . . . . . . . . . . . . . . . . 26 2.5 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 vi 3. Self-Supervised Learning Framework for Sequential Data . . . . . . 29 3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 3.2 Framework Architecture . . . . . . . . . . . . . . . . . . . . . 29 3.3 Self Supervised Learning Modules . . . . . . . . . . . . . . . . 31 3.3.1 Denoising Autoencoder Module . . . . . . . . . . . . . 31 3.3.2 Contrastive Learning Module . . . . . . . . . . . . . . 33 3.4 Supervised Learning Module . . . . . . . . . . . . . . . . . . . 35 3.5 Deep Neural Network Module . . . . . . . . . . . . . . . . . . 36 3.6 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 4. Experiments and Results . . . . . . . . . . . . . . . . . . . . . . . . 37 4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 4.2 Case Study: Crop Identification . . . . . . . . . . . . . . . . . 37 4.2.1 Problem Description . . . . . . . . . . . . . . . . . . . 37 4.2.2 Dataset and Evaluation Metrics . . . . . . . . . . . . . 39 4.2.3 Experimental setup . . . . . . . . . . . . . . . . . . . . 44 4.2.4 Results . . . . . . . . . . . . . . . . . . . . . . . . . . 48 4.3 Case Study: mRNA Vaccine Degradation Prediction . . . . . 51 4.3.1 Problem Description . . . . . . . . . . . . . . . . . . . 51 4.3.2 Dataset and Evaluation Metrics . . . . . . . . . . . . . 52 4.3.3 Experimental Setup . . . . . . . . . . . . . . . . . . . 56 4.3.4 Results . . . . . . . . . . . . . . . . . . . . . . . . . . 59 4.4 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 5. Conclusions and Future Work . . . . . . . . . . . . . . . . . . . . . 63 5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 5.2 Summary and Contributions . . . . . . . . . . . . . . . . . . . 63 5.3 Future Work Directions . . . . . . . . . . . . . . . . . . . . . 64 Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
520	3		_aAbstract: Recent advances in Deep Learning (DL) and Artificial Intelligence (AI) algorithms have proven to be greatly beneficial in many applications. The success of such algorithms in the last decade was mainly built on supervised training of very complex models using large labeled datasets. However, the data labeling process is expensive and time consuming especially for applications with sequential data. Self-Supervised Learning (SSL) appeared as a solution to train complex models with unlabeled datasets. The main idea behind SSL is to create auxiliary task based on some properties in the dataset to learn a latent space that captures the data semantics. Models trained using SSL can be fine-tuned later on a much smaller labeled datasets. This learning scheme becomes very successful in Computer Vision (CV) and Natural Language Processing (NLP) applications but hasn’t yet generalized to other sequential data applications. In this work, we propose a novel DL framework for sequential data processing while adopting SSL techniques to make use of unlabeled data. Our framework is evaluated on two different applications from two different domains, Crop Identification and mRNA Vaccine Degradation Prediction. Details of the applications’ datasets, pre-processing steps and learning objectives of both applications are explained. Experimental results demonstrate that the proposed framework offers a promising approach for utilizing SSL in the training pipeline of commonly-used DL models.
546			_aText in English, abstracts in English.
650		4	_aInformatics-IFM _9266
655		7	_2NULIB _aDissertation, Academic _9187
690			_aInformatics-IFM _9266
942			_2ddc _cTH
999			_c9253 _d9253