The use of evidence is a growing theme in educational discourse. One type of review of the scientific literature that is strongly associated with it is the meta-analysis. Meta-analysis combines results from a variety of research studies on the same topic into a single study in order to identify a central trend expressed as an effect size. However, meta-analysis is dependent on the quality and quantity of the research that is selected.
The use of evidence is a growing theme in educational discourse. One type of review of the scientific literature that is strongly associated with it is the meta-analysis. Meta-analysis combines results from a variety of research studies on the same topic into a single study in order to identify a central trend expressed as an effect size. However, meta-analysis is dependent on the quality and quantity of the research that is selected. Unfortunately, many meta-analyses in education use a limited number of selection criteria, resulting in the retention of low-quality studies with significant methodological flaws. As a result, the results of these meta-analyses can lead to misrepresentations and even to a blinding mirage of certainty. In this regard Bissonnette and Boyer (2021) analyzed the studies included in Tingir, Cavlazoglu, Caliskan, Koklus, and Intepe-Tingir's (2017) meta-analysis of the purported positive effects of mobile devices on student achievement.
Tingir and colleagues (2017) select a total of 14 scientific research, distributed as follows: three (3) research in reading, three (3) research in mathematics and eight (8) research in science. In leaving, the small number of studies in total, and particularly in reading and mathematics, is insufficient and worrisome, especially since this amount of research does not allow for rigorous analysis of results based on grade levels or other variables.
Surprisingly, the authors of the meta-analysis include a study that measured the performance of adult post-secondary students (see de-Marcos et al., 2010) in their meta-analysis, even though it is dedicated to students from preschool to high school, as the title of their article explicitly states (Effects of mobile devices on K-12 students' achievement: a meta-analysis).
Another weakness concern learning content. Some of the selected research focuses on topics that are clearly non-representative of usual and common classroom instruction such as learning to interpret traditional Chinese poems, a brief visit to a museum or religious temple (see: Billings & Mathison, 2012; Hwang, Hwang, Wu, Zhuang, & Huang, 2013; Yang, Tseng, Liao, & Liang, 2013a).
The experimental time in several research is less than 240 minutes, which is clearly insufficient and unrepresentative of school reality (see: Ahmed & Parsons, 2013; Huang, Lin, & Cheng, 2010; Hwang et al., 2013; Yang et al., 2013a; Yang, Hwang, Hung, & Tseng, 2013b). Such a short duration greatly limits the generalizability of observed effects (Slavin, 1986).
In terms of the number of subjects per research, of the 14 research analyzed by Tingir et al. (2017), there are only two (2) research that include 250 or more subjects, and half of the research have less than 100 subjects in their sample. Again, this significant flaw severely restricts the generalization of possible conclusions from these research. Moreover, Cheung and Slavin (2016) recommend selecting only studies with a minimum of 250 subjects since research with fewer subjects artificially produces effect sizes twice as large.
Finally, several research show other major methodological flaws, such as the lack of a control group (see Nedungadi and Raman, 2012), uncontrolled non-equivalence of the experimental and control group (see Carr, 2012) as well as non-teaching in the control group of the content covered in the experimental group (see Riconscente, 2013 ; Varma, 2014 - for example, fractions are taught in the experimental group but not in the control group, even though the posttest, to determine the effect of the wearable devices, covers fractions). In Tingir and team's (2017) meta-analysis, in short, twelve out of fourteen (12/14) research studies have such major methodological flaws that they should not have been included.
Tinger and colleagues' (2017) meta-analysis misleads the reader into thinking, based on the results presented, that the use of portable digital devices is effective in teaching. The poor quality of the research selected in this meta-analysis does not support such a conclusion. As mentioned, the quality of a meta-analysis is based on the studies that were selected in the first place, according to the criteria chosen by its authors. Consequently, the rigour inevitably varies from one meta-analysis to another.
Ahmed, S., & Parsons, D. (2013). Abductive science inquiry using mobile devices in the classroom. Computers and Education, 63, 62–72.
Billings, E. S., & Mathison, C. (2012). I get to use an iPod in school? Using technology-based advance organizers to support the academic success of English learners. Journal of Science Education and Technology, 21, 494–503.
Bissonnette, S., & Boyer, C. (2021). A review of the meta-analysis by Tingir and colleagues (2017) on the effects of mobile devices on learning. Journal of Computer Assisted Learning, 1-5
Carr, J. M. (2012). Does math achievement h'APP'en when iPads and game-based learning are incorporated into fifth grade mathematics instruction? Journal of Information Technology Education, 11, 269–286.
Cheung, A. C., & Slavin, R. E. (2016). How methodological features affect effect sizes in education. Educational Researcher, 45(5), 283–292.
de-Marcos, L., Hilera, J. R., Barchino, R., Jiménez, L., Martínez, J. J., Gutiérrez, J. A., Gutiérrez, J. M., & Oton, S. (2010). An experiment for improving students' performance in secondary and tertiary education by means of m-learning auto-assessment. Computers and Education, 55, 1069–1079.
Huang, Y.-M., Lin, Y.-T., & Cheng, S.-C. (2010). Effectiveness of a mobile plant learning system in a science curriculum in Taiwanese elementary education. Computers and Education, 54, 47–58.
Hwang, G. J., Wu, P. W., Zhuang, Y. Y., & Huang, Y. M. (2013). Effects of the inquiry-based mobile learning model on the cognitive load and learning achievement of students. Interactive Learning Environments, 21(4), 338–354.
Nedungadi, P., & Raman, R. (2012). A new approach to personalization: Integrating e-learning and m-learning. Educational Technology Research and Development, 60, 659–678.
Riconscente, M. M. (2013). Results from a controlled study of the iPad fractions game motion math. Games and Culture, 8, 186–214
Slavin, R. E. (1986). Best-Evidence Synthesis: An Alternative to Meta-Analytic and Traditional Reviews. Educational Researcher, 15(9), 5–11.
Tingir, S., Cavlazoglu, B., Caliskan, O., Koklu, O., & Intepe-Tingir, S. (2017). Effects of mobile devices on K-12 students' achievement: A metaanalysis. Journal of Computer Assisted Learning, 33(4), 355–369.
Varma, K. (2014). Supporting scientific experimentation and reasoning in young elementary school students. Journal of Science Education and Technology, 23, 381–397.
Yang, C. C., Hwang, G. J., Hung, C. M., & Tseng, S. S. (2013). An evaluation of the learning effectiveness of concept map-based science book reading via mobile devices. Journal of Educational Technology and Society, 16, 167–178.
Yang, C. C., Tseng, S. S., Liao, A. Y. H., & Liang, T. (2013). Situated poetry learning using multimedia resource sharing approach. Educational Technology & Society, 16(2), 282–295.