A Multi-Embeddings Approach Coupled with Deep Learning for Arabic Named Entity Recognition / Abeer Youssef Mohamed

Supervisor: Samhaa El-Beltagy

Thesis (M.A.)—Nile University, Egypt, 2021 .

"Includes bibliographical references"

Contents:
ABSTRACT ................................................................................................................................ I
DEDICATION ........................................................................................................................... II
ACKNOWLEDGEMENTS..................................................................................................... III
PUBLICATION RELATED TO THIS WORK .................................................................... IV
LIST OF TABLES ................................................................................................................. VII
LIST OF FIGURES .............................................................................................................. VIII
LIST OF ABBREVIATIONS .................................................................................................. X
CHAPTER 1 INTRODUCTION .............................................................................................. 1
1.1 PROBLEM DEFINITION ......................................................................................................... 1
1.2 OBJECTIVES ......................................................................................................................... 2
1.3 MOTIVATION ....................................................................................................................... 3
1.4 CONTRIBUTION .................................................................................................................... 3
1.5 THESIS ORGANIZATION ....................................................................................................... 3
CHAPTER 2 BACKGROUND ................................................................................................. 5
2.1 ARABIC DATASETS .............................................................................................................. 5
2.2 ARTIFICIAL NEURAL NETWORKS (ANN) ............................................................................ 6
2.2.1 Recurrent Neural Networks (RNN) ............................................................................. 7
2.3 CONDITIONAL RANDOM FIELD (CRF) ................................................................................. 9
2.4 WORD EMBEDDINGS ......................................................................................................... 10
2.4.1 Static Embeddings ..................................................................................................... 11
2.4.2 Contextual Embeddings ............................................................................................ 12
CHAPTER 3 LITERATURE REVIEW ON NAMED ENTITY RECOGNITION .......... 14
3.1 PREAMBLE ......................................................................................................................... 14
3.2 RULE-BASED APPROACHES FOR NER USING GAZEETTER AND POS .................................. 14
3.3 MACHINE LEARNING APPROACHES FOR NER ................................................................... 18
3.3.1 A Machine Learning Approach using Brown Clustering Technique ........................ 18
3.3.2 Machine Learning Approach using Bayesian Classifier Combination .................... 19
3.4 HYBRID APPROACHES FOR NER ....................................................................................... 20
3.4.1 A Hybrid Approach using Naïve Bayes Classifier and Dictionary ........................... 20
3.4.2 A Hybrid Approach using Decision Tree and POS ................................................... 21
3.5 DEEP LEARNING APPROACHES FOR NER .......................................................................... 22
3.5.1 A Deep Learning Approaches using Conditional Random Field.............................. 23
3.5.2 A Deep Learning Approaches using Attention Mechanism ...................................... 28
3.5.3 A Deep Learning Approach using Pooled Contextualized Embeddings ................... 32
vi
3.5.4 A Deep Learning Approach using Semi-Supervised Co-Training (SVM and BiLSTM-CRF) .................................................................................................................................. 33
3.6 CHAPTER SUMMARY ......................................................................................................... 36
CHAPTER 4 RESEARCH METHODOLOGY .................................................................... 37
4.1 PREAMBLE ......................................................................................................................... 37
4.2 DATASET ........................................................................................................................... 37
4.3 PROPOSED MODEL ............................................................................................................ 39
4.3.1 Multi-Embeddings ..................................................................................................... 41
4.3.2 Encoding ................................................................................................................... 44
4.3.3 Decoding ................................................................................................................... 44
4.4 TESTING OF OTHER MULTI-EMBEDDINGS MODELS ........................................................... 45
4.4.1 Pooled Contextual Embedding with Word2Vec ........................................................ 45
4.4.2 Pooled Contextual Embedding with fastText ............................................................ 47
4.4.3 Pooled Contextual Embedding with Multilingual BERT .......................................... 49
4.4.4 Pooled Contextual Embedding with Arabic BERT ................................................... 50
4.4.5 Pooled Contextual Embedding with fastText and Word2Vec ................................... 51
4.4.6 Pooled Contextual Embedding with Arabic BERT, and Word2Vec ......................... 53
4.4.7 Pooled Contextual Embedding with XLNet, and fastText ......................................... 54
4.4.8 Pooled Contextual Embedding with XLM-RoBERTa, and fastText .......................... 55
4.4.9 Pooled Contextual Embedding with Arabic BERT, and fastText .............................. 56
CHAPTER 5 EVALUATION ................................................................................................. 58
5.1 PREAMBLE ...................................................................................................................... 58
5.2 EXPERIMENTATION SETUP AND IMPLEMENTATION............................................................ 58
5.3 QUALITY METRICS ............................................................................................................ 58
5.4 RESULTS AND DISCUSSION ................................................................................................ 59
5.5 ERROR ANALYSIS .............................................................................................................. 64
5.6 CHAPTER SUMMARY ......................................................................................................... 65
CHAPTER 6 CONCLUSION AND FUTURE WORK ........................................................ 67
6.1 CONCLUSION ..................................................................................................................... 67
6.2 FUTURE WORK .................................................................................................................. 67
REFERENCES .........................................................................................................................

Abstract:
Named Entity Recognition (NER) is an essential task in many natural language processing applications. Extracting crucial information using NER is a primary phase in most NLP downstream tasks, it is used to identify entities into predefined classes. There are several studies that have focused on NER for the English language. However, there are some limitations when applying the current methodologies directly to the Arabic language text. Recent studies have shown the effectiveness of contextual embedding representations and their significant improvements that were shown in English NER tasks.
The aim of this thesis is to examine the hypotheses, that a multi-embeddings approach coupled with a deep learning network is capable of extracting named entities from Arabic sentences and thereby reduces the manual costs. Also, to examine the hypothesis, that the end-to-end deep learning NER approach is capable to outperform the traditional models based on rule-based or machine learning approaches.
To evaluate these hypotheses, an end-to-end deep learning model has been presented which was able to extract Arabic Named Entities and eventually reduce human efforts such as preprocessing work and feature engineering. To reach the best model, different types of classical and contextual embeddings have been combined and evaluated. Thus, the presented model utilized a combination of several types of embeddings (multi-embeddings) such as traditional word embeddings as well as contextual embeddings to gain the advantage of both. For encoding and decoding, we chose to use the models that are heavily used in sequential data modeling. The results of the proposed model achieved a competitive advantage, outperforming all previously published results of deep learning, and non-deep learning models working on the same dataset. The presented results also surpassed those of the winning systems for the same task on the same data in the Topcoder.com competition. Since this approach does not rely on any external resources or handcrafted features, the model can be easily extended to other Arabic NER domains.

Text in English, abstracts in English.