Finnish researchers at the University of Turku and Tampere have published to our knowledge the most comprehensive dive into academic research and empirical evidence into gamification.
“The rise of motivational information systems: A review of gamification research” by Jonna Koivisto and Juho Hamari at the Open Access supported International Journal of Information Management also introduces new terminology. Primarily, the concept of Motivational Information Systems. The intent is to display “gameful” LMS within a broader landscape of digital platforms that seek to combine hedonistic and utilitarian elements. But the new term could add more confusion than it’s worth, as is the attempt to define gamification in terms of the “affordances” of the system. It is true that gamification pervades modern application development across all fields. From finance and healthcare, to e-commerce and dating, gamification still proves to be a promising design pattern to increase success in behavioral intervention.
http://lmspulse.com/2017/gradecraft-a-gameful-lms-from-the-university-of-michigan/
The real merit of the endeavor lies in the exhaustive dataset it collected. 787 academic studies published after 2011 and before June 2015 were collected, from which 273 containing empirical data were fully analyzed for the review.
Quest for evidence
The 273 scientific works offering quantitative data on gamification offered data from 276 experiments across a variety of fields. Education\Learning was the most common, with 129 or 46.7% of datasets. Following were Health and Exercise (14.5%), Crowdsourcing and citizen science (9%) and Social behavior (5%). Taking a look at non-empirical studies, including those who lay groundwork for future research, software development and business are the second and third most prolific fields, again behind learning and education.
Quantitative techniques made up 60% of the empirical datasets, split evenly in descriptive (85 datasets) and inferential (80). 17% of the studies were qualitative, and the reset were mixed.
The main data gathering methods were
- Surveys or questionnaires (179)
- Implementation or prototypes (161)
- Experiments or trials (78)
So while the main source of data were surveys, system or log data were also prevalent sources (128 cases). It is worth noting that data from existing systems was only mentioned for 5 cases, suggesting the researchers opted for drawing LMS data very sparingly. Implementing bespoke gamification apps, tools and prototypes were vastly more common choices.
The main analysis methods were
- Quantitative descriptive (136)
- Qualitative (100)
- Quantitative modeling through T-tests, regressions, etc (70)
- Quantitative comparisons using ANOVA or similar (44)
- Associations and correlations through Chi squares, Spearman or other (30)
- Statistical quantitative including Binomial tests, Logistic models, Granger causality test, Clustering, Z-test etc (11)
Finally, going into the gamification design patterns, or what Koivisto and Hamari define as “digital affordances,” the most popular items found to be research subjects were
- Points, scores or “Experience Points” (XP, 138)
- Challenges, quests, missions or similar (91)
- Badges, achievements, medals, trophies (85)
- Leaderboards or rankings (82)
- Levels (59)
- Performance feedback, including real-time stats (46)
- Status or progress bars, skill trees (32)
These reflect interest of designers and academics, as well as ease of deployment and research, rather than reflecting broader interests, let alone effectiveness. In fact, the popularity of the affordance seems to be a function of how easy it is to set up for scientific enquiry. Adding these elements to an app is simple, while “in reality, gamification is difficult to design.”
Key findings
- On its most basic level, gamification helps bridge delayed outcomes with the long-term commitment and perseverance they demand. This could explain why exercise and education are such common implementations.
- The research found often focuses on perceptions of use, experience or satisfaction. Important as they are, they are often considered the basic ground of analytics depth, including the ROI on Analytics Pyramid.
- In the education and learning space, 19 out of 28 papers featuring experimental quantitative studies show positive (10) or “mixed with positive” (9) outcomes of gamification features in learning, for 68% of the subset.
- Among the most popular affordances, Challenges, quests or missions show the most positive results (36.8% in 19 papers), followed by Badges, achievements, medals, trophies (36% of 25) and Points, scores or XP (31.4% of 35).
- Many other showed a lot more success, but in worryingly small samples: Increasing difficulty (3 of 4 papers), Competition (5 in 8), Progress bars (5 in 8), Warnings (1 in 1) and Adaptive difficulty (1 in 1). None of these show null, mixed with negative, or negative results.
- Only 11 papers featured controlled, single factors. They focus on the most popular ones. Only two strongly suggest positive results:
- Leaderboards and rankings contribute to speed, time to course graduation, grades and academic performance.
- Points, scores and XP lead to increased volume of output in social networking settings.
Research agenda: The open world is too broad
The main conclusion is hardly surprising. “Gamification is not a silver bullet.” It depends on the context, and more research is needed.
The main frustration Koivisto and Hamari express across their work is the lack of “agenda, theoretical guidance or a clear picture of the field.” It risks making the results of all these articles harder to compare, if not impossible. It also leads to redundant efforts, such as the construction of the body of theory, they argue. Given the predominance of learning and education on the subject, the field looks the most vulnerable to this theoretical disarray.
It is common for new research topics to show these kinds of issues. But given that the first metascientific inquiries into peer-review literature go as far as 2014, it is not easy to think of gamification as a new concept.
Given that “gamification research still lacks coherence in research models, and a consistency in the variables and theoretical foundations,” Hamari and Koivisto conclude with 15 “research trajectories” for future academics interested in covering gamification, motivational information and “gameful” systems. While other attempts to bring order to the research might exist, Hamari and Koivisto’s exhaustive work gives them some authority on the matter:
- Cooperative and collective gamification
- Diversification of gameful affordances within a broader understanding of gameful experiences
- Widen the domains of gamification
- Potential negative, adverse or non-preferable effects of gamification
- Gamification as an organizational and individual-based practice, as opposed to just a type of human-computer interaction
- Prerequisites of gamification success, perhaps including adoption factors
- Gamification users, goals and individual attributes
- Gamification deployment contexts and link to research models
- Feedback, its types and best gamification affordances to deliver each one
- Dynamic and cyclical perspectives
- Consistency in measurement, research models, depth and scope
- Expansion of controlled, experimental research
- Control for individual factors within a rich implementation
- Long-term spans of research and measured outcomes
- Comprehensive reporting of research
The research is Open Access and was funded by Business Finland, Satakunnan korkeakoulusäätiö (Satakunta University of Applied Sciences: SAMK) and GameCult (Centre of Excellence in Game Culture Studies) at the Academy of Finland. The researchers report no potential conflict of interest. Read or download it in full here. (DOI)
Our coverage on Outcomes for LMS and Learning Systems is supported by eThink Education, a Certified Moodle Partner and Gold Totara Partner that provides a fully-managed Moodle experience including implementation, integration, cloud-hosting, and management services. Learn more about eThink here.
