PPPP: PRECOLLEGE PREDICTION OF PROGRAMMING PERFORMANCE; A TALENT IDENTIFICATION FRAMEWORK / Ayman Aboulmagd Ahmed Abdelraheem Farghaly

Supervisor:
Dr. Passent M. El-Kafrawy

Thesis (M.A.)—Nile University, Egypt, 2022 .

"Includes bibliographical references"

Contents:

Dedication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii
Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix
List of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi
List of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii
Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xix
Chapters:
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2 Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.3 Research questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2. Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.1 Introduction to Cognitive Abilities . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.2 Relevant broad abilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.3 Definition of programming performance . . . . . . . . . . . . . . . . . . . . . . . . 7
2.4 Spatial skills and STEM achievement . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.4.1 Components of spatial skills in relation to programming . . . . . . . . . . . 8
2.4.2 Males vs females differences . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.5 Theoretical frameworks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.5.1 Object-visual-verbal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.5.2 Spatial encoding strategy ”SpES” theory . . . . . . . . . . . . . . . . . . . 9
2.6 Neuroimaging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.6.1 Relevant neuroimaging studies . . . . . . . . . . . . . . . . . . . . . . . . . 11
3. Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
3.1 Systematic literature review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
3.1.1 Research questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
3.1.2 Conducting searches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
3.1.3 Exclusion Criteria . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
3.1.4 Grey Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
3.1.5 Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
3.1.6 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
xv
4. Experimentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
4.1 Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
4.1.1 Participants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
4.1.2 Predictive modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
4.1.3 Model evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
4.1.4 Shortlisted cognitive Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
4.1.5 Data preparation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
4.1.6 Descriptive statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
5. Results and Discussions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
5.1 PSTV-R Spatial rotation test results . . . . . . . . . . . . . . . . . . . . . . . . . . 31
5.2 Non-Verbal Reasoning test results . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
5.3 Gender Differences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
5.4 Predictive Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
5.4.1 Method 1: Aggregate score of cognitive tests . . . . . . . . . . . . . . . . . . 35
5.4.2 Method 2: Individual answers of questions as features . . . . . . . . . . . . 38
5.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
5.5.1 Threats to validity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
6. Conclusions and Future work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
6.1 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
6.2 Future directions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

Abstract:

Universities and students face the challenge of finding the career that best suits the students.
Programming is currently one of the highly sought-after careers and is one of the highest paying
jobs. However not all students succeed in this career. In fact failure rates in programming field
is relatively high globally. Thus, the need for an assessment of student skills is required before
admission to a CS program. Cognitive tests are widely used in this regard in addition to other
means (i.e. school grades, personal information...etc). The aim of this study is to develop a model to
predict programming performance with the minimum set of tests. We conducted a literature review
on previously used cognitive tests to identify programming performance. Then we used a spatial
rotation test and Non-Verbal reasoning test, as well as gender, at the beginning of the semester
to observe the relationship between those tests and students’ relative programming grades at the
end of the semester. We conducted this research on 3 groups. Group 1 are CS students studying
programming course, group 2 are non-CS students studying programming course and group 3 CS
students studying non-programming course. For each group, we created a predictive model using
2 methods. Method 1 uses the aggregate score of cognitive tests and method 2 uses answers to
selected questions in each test. In method 2, we applied a data-driven methodology that utilises
individual answers to questions as input to the model. The modeling results show usefulness of those
test to differentiate programming and non-programming course performance using both methods.
Furthermore, method 2 demonstrates much higher superiority over method 1. The initial results
show that those tests correlate highly to introductory programming grades as indicators/estimators
to programming performance in early CS majors. We also observed a negative correlation of those
tests and performance in non-programming courses which manifests the usefulness of those tests as a
differentiator for different specializations. This has the benefit of helping students in choosing their
career and it also helps learning institutions to segment students according to their programming
performance to create customized learning experience.

Text in English, abstracts in English and Arabic