Next Article in Journal
Enhanced Water Demand Analysis via Symbolic Approximation within an Epidemiology-Based Forecasting Framework
Next Article in Special Issue
Towards a Framework for Designing and Assessing Game-Based Approaches for Sustainable Water Governance
Previous Article in Journal
Hydrological Model Supported by a Step-Wise Calibration against Sub-Flows and Validation of Extreme Flow Events
Previous Article in Special Issue
A Serious Game Designed to Explore and Understand the Complexities of Flood Mitigation Options in Urban–Rural Catchments
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Concept Paper

Conceptualizing Serious Games as a Learning-Based Intervention in the Context of Natural Resources and Environmental Governance

1
Technology and Environmental Studies, School of Natural Sciences, Södertörn University, SE-14189 Huddinge, Sweden
2
Wageningen University & Research, 6700 AH Wageningen, The Netherlands
*
Author to whom correspondence should be addressed.
Water 2019, 11(2), 245; https://doi.org/10.3390/w11020245
Submission received: 27 November 2018 / Revised: 15 December 2018 / Accepted: 14 January 2019 / Published: 31 January 2019

Abstract

:
The use of serious games in the governance of natural resources and the environment is progressively increasing and includes games used for research and data collection, teaching and training, and fostering a change of practices. However, this diversity remains underexplored and underreported. In view of a growing interest in the use of serious games in natural resource and environmental governance, the absence of discussions about how differences in intended use and delivery influence the performance, assessment, and outcomes of games is problematic. Here we present an inventory, and a description, of such different uses then, by focusing on serious games used as interventions, we discuss when, and how, games could be used to generate learning and social learning. To that end we use a narrative review of selected literature, and insight from research on social learning, to develop an inventory of game use, and within that inventory we conceptualize the use of serious games as a social learning intervention. Also, by means of an illustrative case of a serious game (developed as part of the Assessing the Learning Effects of Games on Attitude of Stakeholders toward Sustainable Shrimp Farming – ALEGAMS research project) we reflect on a few key aspects of game use. We suggest that developing a serious game needs several iterations and, although the learning outcomes can be assessed, the impact of games aiming at changes in current practice and policy will likely fall beyond the timespan of usual project periods. This is something future research should consider as it has implications for the research design and methodology.

1. Introduction

Using serious games in natural resource and environmental governance, inclusive of water governance, has been slowly increasing since mid-1990s [1,2,3]. For instance, Flood et al. [1] in their review note a trend of serious games being now developed and meant for use with stakeholders and policy makers. This trend seems to emerge as a response to the limitations ascribed to some participatory activities used in the research field of natural resources and environmental governance. For instance, while expert workshops might work well where participants have comparable levels of education and free speech is the norm, these do not suit contexts where participants might struggle with complex content and do not feel entitled to voice concerns due to cultural norms, power asymmetries, or other [4]. Aware of this, practitioners and researchers are now exploring alternative (participatory) tools, such as serious games [3].
Using games for serious purposes was first introduced for the training of military novices who participated in simulated situations of near-real-world conditions which were (often) modeled and replicated based on close-to-real data and parameters. This allows for players to safely experience and explore the likely implications certain decisions might have, and to learn from that. Today, serious games are used regularly for training in management, medicine, and policy [5,6]; they are gaining ground in education [7]; and furthermore are attracting the attention of researchers and practitioners working in natural resource and environmental governance. There, serious games are used for many, and different, purposes inclusive of games used as interventions intending to resolve or further understand environmental and/or resource issues and challenges. In such cases, when games are an intervention, this is commonly done under the assumption of games having the potential to engender learning and foster transformative change and/or more sustainable practices. Transformative change toward more sustainable alternatives is a very relevant and contemporary subject in which many researchers and practitioners of natural resource and environmental governance have an interest.
However, in a review of 25 serious games on environmental management done by Madani et al. [1] they note that literature on games often does not report on how assumptions about learning, or skill development, informed game development. Their review points at two major shortcomings across the serious games on environmental management that they reviewed: (i) evaluations of a game’s effectiveness were missing; and (ii) the pedagogical foundations on which the games are based were not clear. Madani et al. [1] advance that many games do not seem well-suited for their intended use and conclude that evaluations of serious games’ effectiveness is a critical research frontier. We agree that, in light of a growing interest for serious games in natural resources and environmental governance, more research is needed to meet its potential to perform as a tool for interventions. Except for a few studies [1,2,3], the diversity of serious games found in this field remains underreported and underexplored. Specifically, there is a need for research about the implications arising from differences of intended use e.g., data collection vs. education and training vs. transformative change. A growing interest the absence of discussions about how differences of intended use might influence the development, delivery, assessment, and outcomes of serious games, is problematic.
In this study, we focus on serious games in natural resource and environmental governance to conceptualize and discus differences of use, with a special interest in games as learning-based interventions. Therefore, we do not aim to classify all serious games available across the many scientific fields, but instead we focus on the research field of natural resource and environmental governance. This research field is a problem-solving oriented area of study and practice which we considered in developing our discussion of game use (Figure 1). Moreover, we focus on practitioners (i.e., adults) and so we are not looking at educational games for pupils and children, and we do not review literature on gamification. By extension, since we set out to review selected literature and map trends in a selected field, not reviewing literature on gamification is a further limitation since it relates mainly to enhancing existing education and decision processes with gaming elements.
In the next section, we first present an inventory of uses serious games have found in natural resource and environmental governance. Thereafter, we focus on games used as learning-based interventions in Section 3, and discuss and reflect in Section 4 on the work we did in the Assessing the Learning Effects of Games on Attitude of Stakeholders toward Sustainable Shrimp Farming (ALEGAMS) research project. Based on our current research and collectively accumulated experience, as well as social learning theory and practice, we present a “Development and evaluation framework for serious games used as learning-based interventions” in natural resource and environmental governance.

2. Differences across Serious Games: A Typology of Use in the Context of Natural Resource and Environmental Governance

Games come in many forms, and can be played in numerous ways, from dice, card and board games, computer games, and field games to role-playing (theater) games. However, setting aside genre differences, games are seen to share four defining traits: a goal, rules, a feedback system, and participation [8]. These all are important characteristics; for instance, when the game is compelling, participants/players will go on playing until completion. Much literature on game-play links participants/players’ motivation to completion of the task at hand with the game being perceived as engaging, fun, and compelling. The way a game can trigger participants/players’ interest and motivation will affect the likelihood for the game to perform its intended use at a high level. In environmental and natural resource management, serious games are used for many reasons, and these differences impact not only on how interest and motivation should be built into game-play, but differences of intended use have implications also on evaluating games and their outcomes [9].
Based on available literature, we now summarize and describe differences in using serious games found in environmental and natural resource management. In this we consider three aspects characterizing this field: (i) propensity to explore with, and use of, (participatory) tools that fit within local contexts; (ii) interest in engaging with and seeking input from different profiles, as locals, practitioners, and community members (all being adults); and (iii) a tradition of problem-solving in relation to resource and environmental issues. Therefore, assuming that games in this field are used to address given needs and issues, we focus on two core aspects (i) purpose and (ii) expected outcomes (Figure 1) to distinguish between different typologies of game use. We acknowledge that others before have looked at the purpose of games and offered different descriptions, for instance Djaouti et al. [10]. However, to the best of our knowledge, such an exercise has not been carried out yet for the field of natural resource and environmental governance.
Following McGonigal, [8] we suggest that while the four defining traits of a serious game she outlined (goal, rules, a feedback system, and participation) may stay constant, it is also important to acknowledge differences of intended game use (e.g., games for data collection vs. education; training vs. transformative change) which are likely to have implications for game development, test, use, and evaluation.

2.1. Games Used for Research

The first group of literature we identify reports on games used as a tool to collect data in order to study aspects of interest. This literature often approaches games as an “experimental method” meant to reproduce near-real-world systems that participants/players engage in while researchers observe them and record/collect data produced during the game. Often these studies focus on social interaction and other social processes as there are group dynamics, decision-making, or behavior of interest. Several teams, as well as scholars working on commons and common-pool resources, have used games as an “experimental method” to study competition, collaboration and decision-making in resource use and management [11]. Janssen et al. [12] note how those early works were informed by experimental economics (e.g., public good, prisoner’s dilemma, and bargaining games). A limitation of these studies was that by primarily focusing on decision-making, these did not include ecological dynamics. This aspect was picked up by later research that extended the use of games (type and number of parameters used), but also how games were administrated (on-campus experimental environment with students vs. field data collection with locals). This created new opportunities, such as the collection of data, that are contextualized (i.e., specific to a locality) and are more robust (i.e., better representing current resource use in the field). As a result, the literature reporting on games used to collect data in the field with local stakeholders (e.g., villagers, inhabitants, farmers) is increasing. For instance, Vieira Pak et al. [13] report on research where role-playing games were used to collect data about decision-making processes about land use in the Colombian Amazonian frontier. They reflect on how role-playing games allowed for discussions during which players shared information that the research team needed and that would not be effectively communicated in an interview. Lebel [14] used a serious game in a similar way—albeit combined with other methods of data collection—in a case of fish farming in Thailand. Meinzen-Dick et al. [15] used serious games in field experiments in India, while Bell et al. [16] used (a tablet-PC-based) serious games in field experiments about an irrigation system in Pakistan.
Furthermore, we identify an additional subgroup of literature that reports on serious games used in combination with computer models [17,18]. Often this use is called companion modeling (ComMOD). In contrast with the literature cited above, the main objective in companion modeling is not to collect data to answer research questions; rather, games are used to build and validate models that researchers have developed, or to provide additional information about scenarios these models reproduce. Étienne [19] notes that the model sometimes simulates the dynamics of a natural resource and provides a dynamic basis for a negotiating game, while in other cases the model is transposed into a game to validate and/or to explain the strategies stakeholders use. In this case a role-playing game facilitated a collective construction of the model and helps to discuss scenarios. For instance, Campo et al. [20] used a serious game with villagers in the Philippines to validate their model and report on how the information generated in this way revealed some model elements to be revised or corrected. Joffre et al. [21] have used a role-playing game to calibrate and validate a model representing different aquaculture scenarios, and they reported that farmers’ involvement with the role-playing game helped them to fine-tune the model and identify key parameters and drivers in farmers’ decision-making. In both cases, the game is instrumental to the research process and is one among several steps and tools used to achieve a much broader objective [18].

2.2. Educational Games

A second group of literature reports on games used as a teaching tool within formal curricula [3,22,23] targeting students undergoing formal education. Among the first, and perhaps best-known example of serious games used for natural resource management curricula in the 1970s is the Fish Banks game, which was used to educate players about dilemmas arising through the joint management of open access resources [3,24]. Using games as a teaching tool is a well-discussed topic, and as a teaching method serious games are generally preferred by students over more traditional lectures. These games put students into a more active role, allowing learning by doing and ownership of their role in ways that lectures do not [23]. Given the complexity of contemporary issues in resource management and environmental governance, which require the integration of multiple disciplinary knowledge, as well as good understanding of social and political processes, using games integrating scientific information with information about contemporary policy processes seems to hold promise for education.
Games are assumed to support a deeper understanding of contemporary environmental challenges [25]. For instance, Flood et al. [1] in their review of serious games on climate change, find that besides educational outcomes, often these games also lead to improved awareness. Flood et al. [1] comment that, during game-play, students’ work with data and descriptions/simulations of real-world policy processes helps them in their learning processes, but also allows them to have a close-to-real-world experience. Chen and Martin [24] discuss the potential of role-playing as an educational tool for transformative learning. They anchor their assumptions in the way players engage with each other, and experience different situations during game-play. They use the Climate Change Policy course (for management students in the Community of European Management Schools, CEMS) network as reported by Paschall and Wüstenhagen [26] to provide an example of how such a (two-day intensive role-play) experience can (potentially) foster transformative learning. In addition to scientific facts, Chen and Martin [24] see value in participants having to choose a role to play (i.e., they need to take a stand/adopt a perspective), and then play and advocate for it. This, they assume, requires higher order thinking, reflection, and examination of both internal and external influences on environmental attitude and behavior.
Also, some literature, albeit much less, reports on serious games for training and education of stakeholders, farmers, and other practitioners. For instance, Pruksakorn et al. [27] report on using a role-playing game named “Soil analysis and appropriate fertilizer use game” (SAAFU) designed for farmers with a low formal education to learn about soil analysis before planting, and, by extension, to improve the appropriate chemical fertilizer usage. Ferrero et al. [28] report on using a serious game for water professionals to complement existing training materials for them to better understand the problem at hand, to explore with strategic decisions financial investments, and to see how stakeholder collaboration impacts water supply safety management. Complementing this report, Erisman et al. [29] report on a game named NitroGenious, developed and used to improve understanding among policy makers about the nitrogen pollution problems in a selected area in The Netherlands.
However, we note that literature reporting on games used for training of practitioners, and other stakeholders, is less frequent compared to literature on games used for the teaching of students undergoing education. This could be linked to differences of context within which these two groups are located, and to differences of opportunities educators have to assess and reply of the needs learners have. Teachers/educators have daily contact and a good overview of student performance in class. That knowledge might allow them to better grasp what needs students have and search /choose among alternative pedagogical methods/tools. While the extensions agents, and related agencies, know well what information and content should be delivered on a subject matter, they might not have an overview of practitioners’ learning needs and preferences. Nevertheless, practitioners’ are a more heterogeneous group (e.g., age, education, interests) compared to students. These are, however, tentative assumptions given that, to the best of our knowledge, no overview of used serious games for training practitioners / stakeholders is available.
In contrast, literature reporting on serious games used as part of formal education on natural resource and environmental governance is thoroughly reviewed [2,3,30]. These reviews, however, find that studies often fail to establish links between the learning theory used to inform game development, expectations in terms of outcomes, and frameworks used to establish the game effectively and deliver learning outcomes [2]. Much of the literature on educational games seems to agree that the core objective of these games is to educate, train and help students acquire knowledge. However, these aspects often go unreported. For instance, Chen and Martin [24] discuss the potential of role-playing games, and in so doing use as an example the game developed by Paschall and Wüstenhagen [26] which conceptualizing it as a tool for transformative learning. The discussion by Chen and Martin [24] is relevant and points to several aspects of interest pertaining to game-play. However, that they remain silent on the hypothesis and objectives, in terms of transformative change, as applied by those who developed the game. Albeit, of interest is insight and suggestions they offer on how different aspects of game-play (e.g., discussion, confrontation with opponents, etc.) provide space for participants’ transformative learning and the resulting outcomes. Mezirow [31] assumed that transformative learning results in lasting change processes, yet Chen and Martin [24] and Paschall and Wüstenhagen [26] report on this not being evident for the stated educational game on climate change.

2.3. Games as Interventions

A third group of literature reports on games used as an intervention under the assumption that these contribute to change processes that go beyond the individual (often referred to by current literature as social learning processes). Such use of games might have emerged as a response to the limitations ascribed to participatory interventions e.g., workshops and/or focus groups, that some research critiques do not fit well in all local contexts [1,3]. Serious games are described by some as participatory tools able to engage people (e.g., problem owners) in debate and activity that can accommodate their needs and create certain opportunities in different ways. For instance, Flood et al. [1], while focusing on climate change, propose serious games as tools that can trigger learning about the subject at hand, and assume this could give way to change of practices and behaviors. Interestingly, several other papers assume games to have potential in terms of social learning and to bring about transformative change [15,32,33,34,35,36]. This literature advances specific assumptions, as summarized below, about what a serious game can do, and what its outcomes can be, which contrasts within the literature outlined above describing educational games, or games for research (Figure 1).
First, educational games are broader in scope and often are about a general issue, at a larger geographical and/or administrative scale, e.g., climate change [1], and over-fishing [25]. However, games meant to perform as interventions often target a rather narrowly identified issue in natural resource and environmental governance, and this much often at a local scale [15,28,32,37,38,39,40]. Serious games meant to perform as interventions often integrate contextual features and have finer levels of detail, to ensure that local stakeholders, or problem owners, get motivated and engaged in game-play and become part of the change processes the research team envisaged.
The second difference is notable in the assumptions and expectations about what a game-play can trigger. Educational games, while targeting individual “learners” seek to inform, deliver information, and share knowledge on a subject, whereas research games seek to gather or validate data. In contrast, games used as interventions place the attention on elements/aspects regarded (also from a theoretical view point) to have an important role in fostering the transformative change of interest that goes beyond the individual player. Players are then often understood and described in their role of “stakeholders” rather than “learners” (e.g., educational games), or “testers” or “data points” (e.g., research games). Souchère et al. [41] reported a role-playing game created with groups of local stakeholders, meant to facilitate negotiations on the future management of erosive runoff in the Pays de Caux (France). In this, the objective was to create opportunities for co-operation between stakeholders, and this was done based on theoretically grounded assumptions about participatory and collaborative activities. Souchère et al. [41] defined the game as a tool for mediation and collective thinking that allowed players to see the collective consequences of their individual decisions, and to share knowledge, ideas and their vision of the problem, in order to test different strategies. Rebolledo-Mendez et al. [42] reported on research in which they sought to explore the effectiveness of serious games in improving the awareness on flooding among the general public in the UK; they considered the game as a tool helping participants acquire information and a better understanding of the system.
Third, when serious games are used as interventions, substantial effort is also put into integrating updated scientific data, contextual features, and local circumstances which together can best simulate the current state of affairs. This is often done with the ambition to recreate near-real-world circumstances for the game-play to be a meaningful activity to the problem owners. The recreation of a near-real, meaningful experience is assumed to help players to maintain high motivation, complete the activity, and be influenced by the experience to which they were exposed.
However, for games used as interventions with the intention of pursuing and/or fostering change processes, a question that emerges is to what extent such assumptions are validated in practice. The latter question, however, is poorly understood and underreported. With very few exceptions [42], only few studies provide robust evidence of game-play-induced change. Future research should try to bridge that gap and include methods that assess the effectiveness of serious games when used as interventions. Assessments need to go beyond verifying whether the game is an accurate representation of the reality it represents, and beyond reports of a player’s subjective experience. Rather, assessments need to explore and test the assumptions made about whether, how, and when the serious game has delivered on the expected outcomes, i.e., change processes and actions. For instance, the assessment of the FloodSim serious game offers useful input here. Rebolledo-Mendez et al. [42] note that FloodSim was designed with a stated aim to raise awareness about flooding policy, however, their assessment shows that most players who were interviewed gained only a superficial level of awareness, and not of the kind that would allow them to engage with and understand the policy debate on flood and government expenditure. They conclude for FloodSim that game design and development should have been more interdisciplinary and should have better integrated well-established frameworks and knowledge about the human behavior that is assumed to change. Establishing a conceptual and methodological link between the expected outcomes with the different traits of game-play [8] early in the process is a recommendation from which future projects, interested to use serious games as intervention, could benefit.
To conclude this section we want to clarify that we do not suggest the above being a rigid categorization, and/or mutually exclusive, of game use in the governance of natural resources and the environment. Rather we suggest the typology (Figure 1), and corresponding descriptions, as a support to those needing to plan and deliver games in a local context. Attention to purpose and expected outcomes may guide researchers and practitioners over choices of game development, test, use, and evaluation—inclusive of theoretical and analytical frameworks. In the next chapter, we report and reflect on the work done by our team on developing a serious game meant to perform as a learning-based intervention. In building on this experience, in Section 4 we then consider social learning research and practice and suggest a “Development and evaluation framework for serious games used as learning-based interventions”.

3. Developing a Serious Game within the Assessing the Learning Effects of Games on Attitude of Stakeholders toward Sustainable Shrimp Farming (ALEGAMS) Research Project

The development and use of serious games meant as interventions is often done as part of larger projects needing to achieve broader objectives. Some projects might focus on the development and prototyping of a serious game only—then feature it as an output for others to use—while other projects might develop a game as an intervention/tool and use it to achieve pre-defined outcomes e.g., change processes. The ALEGAMS project (Box 1) aimed for both: the design and development of a serious game meant as a learning-based intervention seeking to generate social learning across local communities in three pilot sites. Below, we summarize our experience and work, including the challenges and constraints encountered during the project.
Box 1. The “Assessing the Learning Effects of Games on Attitude of Stakeholders toward Sustainable Shrimp Farming” research project (acronym: ALEGAMS).
  • Background: Over recent years mangrove forest, along the Vietnamese Mekong Delta, is under pressure from the expansion of aquaculture, shrimp farming more specifically [30]. This is a problem recognized by policy and practice, among whom most agree that loss of mangrove is an issue since these forests perform important functions in regards to climate change adaptation and local economy. Dedicated efforts to restore and maintain mangrove have been implemented in Vietnam, inclusive of a policy program supporting integrated mangrove shrimp farming (IMS). IMS can be economically viable and allow for sufficient ecological quality [43], particularly compared to extensive shrimp systems [30]. However, it has been observed that farmers who practice extensive farming do not adopt IMS voluntarily.
  • Objectives: ALEGAMS aimed to design, test, and use a serious game (i.e., board role-play game) as an intervention. The game reproduces, through game-play, the dynamics of the main different shrimp farming systems present in the region. This has the aim to help farmers/players to understand the implications of the different shrimp farming systems, and of their mix, applied in the Vietnamese Mekong Delta.
  • Output: A serious game (board game) to be handed over to the International Union for the Conservation of Nature - IUCN Asia (Figure S2).
  • Outcome: Exchange and learning about risk and IMS by shrimp farmers across selected cases in the Mekong Delta.
  • Impact: Social learning manifested in increased knowledge and change of attitude on shrimp farming systems. Increased awareness on IMS which will support farmers in decisions to shift toward IMS. Eventually, an increase of IMS farms in two provinces is envisaged.

3.1. Summary of the Work Done and Steps Taken for Developing the ALEGAMS Learning-Based Game Intervention

In the following section, we summarize and reflect on the work done during the development of the ALEGAMS serious game in order to extract input useful for future research. Methodologically speaking, ours is a reflective exercise and in hindsight we see our work articulated in four phases, which we name and use to guide the summary presented below.

3.1.1. Conceptualization in Context

ALEGAMS focused on documented issues and challenges in the fast-growing intensive shrimp aquaculture in the Mekong Delta. This game intervention targets shrimp farmers and, to a degree, also other decision-makers active in this sector. This idea emerged during an earlier project, when an earlier version of the serious game was used for research to collect data for the calibration of an agent-based simulation model [21].
In ALEGAMS, we built on that experience and designed a game that would help shrimp farmers to learn about the risk and financial performance of three shrimp farming systems, a game that would allow them to explore and experience the ups and downs of each in the face of real-life events, with a focus on shrimp disease outbreaks. We assumed that learning about the risk and financial performance of shrimp farming systems could help farmers to recognize the risk of monocultural, intensive shrimp farming and the value of integrated mangrove shrimp farming, and subsequently increase their willingness to maintain, or shift to more sustainable practice. We conceptualized a game where players, by starting at either an integrated mangrove or extensive farm on plots with different area and value (considering infrastructure) could experience (mixes of) different farming systems and evaluate which turned out to be the most profitable. This, we hypothesized, would allow players/farmers to weigh the (high) risks and rewards of intensive farming versus the (medium) risks and rewards of integrated farming which is bringing about other advantages (timber, ecosystem services, etc).
A board game was considered a suitable tool to bring farmers together for learning within a mimicked local context. Computer games would not suit this context.

3.1.2. Iterative Game Design and Testing

After having formulated assumptions about what the ALEGAMS game should achieve, and locating these in a socio-economic but also biophysical context, we expanded on the work reported by Joffre et al. [21] while focusing on the impact of shrimp disease outbreaks. A first test of the game-play was done with students and researchers to check on clarity of the instructions, ease of application, and financial aspects regarding cost and benefits of farming. Feedback on game-play was collected during several iterations with different players (both Vietnamese and European) and then integrated into a prototype. A group of six shrimp farmers were invited to join us at a project progress meeting held in September 2016 in Can Tho, where they played the game and then reported their feedback orally to the ALEGAMS project team. This included also their views about non-technical matters. For instance, farmers reported not finding the game enjoyable and lacking motivation to complete all the rounds planned to deliver the information about differences between the farming systems. Farmers commented on what they identified as a mismatch with reality: the game did not allow farmers to reduce the risk of shrimp disease outbreaks by investing in technology and/or by collaborating. The increased cost of the former would be outweighed by the extra revenues, and the latter was assumed to reduce cost and increase margins.
This input was considered with care, and our team agreed to develop “risk cards” intended to be introduced during game-play in the form of a card with written text about real-life circumstances relating to risks. One set of cards reflected communal risks, such as flooding, storms, and new or more virulent shrimp diseases, and one card was drawn in each round and applied to all players. The second set of cards was drawn individually at each turn and primarily concerned household matters. Both sets of cards were carefully balanced in numbers having positive and negative impact on the player’s game result. Moreover, the option to buy technology and become a member of a cluster (‘cooperative’) was added, both are decisions players can make to reduce risk.
The background data was updated and contextualized to make the identified parameters reflect reality (i.e., cost and income, and risk, for each farming system). We assumed these changes would make the game-play feel closer to real circumstances since farmers pointed to these as important elements to maintain their motivation, while also making the game more dynamic and interesting.
We tested the new version of the game multiple times: first, with students and researchers familiar with the shrimp sector in Vietnam; then, in The Netherlands with a group of researchers to check the game flow, and the new instructions; then, also with farmers in Vietnam. We integrated all input emerging from these sessions. Then we repeated the tests and ran a pilot with shrimp farmers, i.e., profiles matching our target group, who also offered valuable feedback.

3.1.3. Game Use in the Field

When a project aims to develop and use a game in the field, certain questions need to be addressed. For instance, what is the minimum number of rounds and how to acquaint the players with the rules and to engage them in learning, respectively. The timespan for concentration and availability to complete the gameplay is to be considered. In our case, the numbers of rounds was related to the number of playing rounds/cycles needed before an effect could be observed on the results (bank account) and of the new technologies, to all hazards cards had been turned, and players felt to have had a fair chance to obtain one good dice score. The total time available to farmers was limited to a morning or afternoon since we wanted them to play at least two sessions. For the ALEGAMS game, we saw the need for a first session to make farmers acquainted with all the rules and a second to have them play independently of the moderator.

3.1.4. Evaluation of the Game Intervention

The final prototype of the game was produced by a professional company in Vietnam (Figures S1 and S2). We agreed on a longitudinal design early in the process and developed an evaluation strategy meant to collect data about game-play and learning, along with the post-game outcomes at different points of time. ALEGAMS intended to trigger change in a semi-experimental design to collect data about learning and attitude, and repeated exposure (one or two times) to the play. The game was used with local shrimp farmers and practitioners on two occasions at several months’ interval in three coastal locations (Figure S1). This sample excluded those who had played the prototype version.
Data on the learning outcomes are being analyzed and will be published later. The intended impact, i.e., an increase in applying integrated mangrove shrimp systems, was not aimed to be assessed within the three-year project period.

4. Development and Evaluation Framework for Games as Learning-Based Interventions

In a narrative review of literature in natural resource and environmental governance, we identified eight studies reporting on games used as learning-based interventions seeking to promote social learning and so contributing at changing in how things are done [32,33,34,35,36,44,45]. These studies differ in the way social learning is defined and operationalized, but share assumptions about game-play offering opportunities for (i) debate and interaction with other players/participants, (ii) critical reflection, either alone or collectively, (iii) development of a shared perception on the problem domain, (iv) development of a shared stock of knowledge, and (v) seeing the issue from a different perspective. All are process and outcome features discussed in the literature on social learning [46,47] that partially agree with the operationalization of social learning within three dimensions: relational, cognitive, and moral/ normative [47,48,49,50]. Based on this compilation of papers, on broader literature on social learning, and most important, informed by work done on the ALEGAMS project, we now propose a “Development and evaluation framework for serious games meant as learning-based interventions”.

4.1. Conceptualization in Context

When serious games are utilized as interventions, a first step is the identification and description of the problem, or issue, games are expected to contribute at resolving and these issues likely are about social matters. Games are about the engagement of people, and it is important that the theoretical basis operationalizing social processes under study (i.e., learning and transformative change) is chosen, described, and made operational early on. Literature on social learning assumes that social learning can be operationalized within cognitive, behavioral, normative, and relational aspects [46,47,50]. Drawing links between these and ways in which game-play exerts an influence on them are relevant. Two recent reviews of the learning literature may help to reflect upon, identify and select process and outcome features assumed to be most relevant to a project [46,47].
Also, the assumed underlying mechanism and linkages between these (social) processes and the problem issue at hand should be explicated, evidence for these collected and considered, and the expected short and longer effects of game-play listed. This may build on well-established theories about learning and literature in psychology (e.g., when attitudes are under focus) and on social learning literature. One can reach out also to more informal evidence/local expertise regarding pressures, or local beliefs, assumed to promote or inhibit (behavioral) change. Current empirical literature on this subject has acknowledged benefiting from informal input as well [35,36].
This work, inclusive of the assessment and choice of a best-suited theory, will help preliminarily to identify a suitable game typology (computer-based vs. board game; group game-play vs. solo-play: role-playing etc.) and study designs (case study; quasi-experiment; controlled study; ethnographic research; etc.) [51,52,53]. The choice of game typology should fit well to local circumstances inclusive of its needs, expectations, and practices and socio-economic characteristics, all of which must be identified at an early stage. This will affect choices on whether to use a computer-based vs. board game; group game-play vs. solo-play, or others. It is useful to note that the choice of a game type will have implications for the (social) processes the game-play will bring about. De Vente et al. [4] addressed the question of how the context and design of participatory processes affect their outcomes more broadly and that discussion can offer some useful input on that aspects. Furthermore, literature focused on social learning offers input and informs us that activities fostering collaboration between the participants overall seem to perform differently on learning compared to activities fostering competition [54]. Studies have, for instance, reported on activities undertaken as part of group work where deliberation, and talking over different viewpoints, resulted in developing a shared understanding [48,49].

4.2. Iterative Game Design and Testing

After having addressed theoretical questions and developed a sharper perspective on the intervention needed, the components the game intervention is expected to have are listed (i.e., opportunity for interaction, for reflection), and how these are inter-related and will lead to desired outcomes outlined. Input on these aspects can be sought in studies reporting on and discussing implications about how choices of design (of participatory processes) impact on learning and interaction. Studies have also discussed how that experience may affect (enhance or hinder) social learning after the activity also [52,53,55]. At this stage, a paper-and-pencil plan can describe the idea and assumptions behind the intervention, or this can be done with a preliminary computer simulations, or modeling. Where needed, and possible, testing of assumptions regarded as not well-supported by available scientific evidence can be done in focus groups, surveys, or shorter observational studies. This work will result in a game-play conceptual idea.
Serious games used as interventions should represent the reality these seek to mimic. When this is not the case, research shows that incentives and motivation drop. Players (particularly when well-informed on the topic) struggle to find meaning, become disengaged, or drop out before completing the rounds. The team needs to populate the conceptual idea of the game intervention with current data and information in order to safeguard meaning and purpose. Games are expected to be an engaging experience also; thus, adequate incentives, matching well with the target group profile, should be considered and integrated into the game (e.g., suitable degree of difficulty, challenge, entertainment features). This step may result in a rudimentary prototype.
Testing the game intervention allows for checking how different components are received and how these are appreciated. The way the prototype is perceived and first assessed by players can be established with short post-test questionnaires, interviews, and/or observation of the game-play. The data collected will help to identify areas to be adjusted/strengthened, or features to be added. For instance, identify issues with the representation/recognition of the natural environment, of the problem(s) at hand and the goal(s) to accomplish, and identification of processes/events present in reality but missing in the game. Several re-tests are needed, and suitable testers should recruited [56]. Where needed, control groups and outcome measures could be used. As the prototype is being refined and tested, there are opportunities to verify alternative ideas. After several iterations, when the majority of the testers no longer signal significant shortcomings, and the research team is satisfied with the learning outcomes observed, the game can regarded as fit for purpose.

4.3. Game Use in the Field

Projects aiming to develop a game intervention to be used in the field and to foster change processes are advised to consider constrains and opportunities for learning and social learning present in a local context. In this there is a need to acknowledge differences about what is to be looked at more precisely, and evaluated (the game vs. long terms change processes). For instance, evaluation of the game as an output, developed as part to a project, can be done within the project lifespan and can involve self-reported assessments the players fill out after completing game-play. However, assessments aiming to assess the impact in terms of change at a more meta level might to be based on objective measures. Also, since change processes are best observed over longer time periods, the questions of when is a good time to collect evidence, and for how long, become relevant.

4.4. Evaluation of the Game Intervention

A game intervention is developed with the intent to be used and put to work. When implementation is scheduled as part of the core project, that might also include evaluation. Choice of a suitable evaluation method is a challenge, given that change of practice and transformative processes require longer observational periods to be detected and recorded [53]. A suitable evaluation method, as some critical papers noted, needs to be chosen at an early stage, and revised when needed [54]. Evaluating learning and other processes at the level of individual players is recommended, and can be done with different post-test data collection tools and approaches. The broader applicability of the game intervention outside the context where it was developed may also be considered early in the processes. Appropriate arrangements are needed for handover and implementation when this is done by a different organization than that developing it.
To conclude, we want to note this framework should not be taken as a rigid progression, and not all research teams need to move hierarchically, and some projects might proceed fast, skip steps, or reiterate. But we expect this overview can offer some support to project teams in embedding game development and its delivery within social learning theory and practice, and within learning-led change process.

5. Conclusions

The inventory of literature conducted suggests that serious games are mainly used and studied for educational purposes, while their use to support research and interventions has emerged more recently. We noted that several studies report on games used for more than one purpose, e.g., to collect data or as an educational tool. However, the assessment of the outcomes games were expected to deliver is not always undertaken. In this paper, we aimed to conceptualize serous games as a learning-based intervention and then have used the illustrative case to consider a few aspects, including the assessment of expected outcomes.
In our case, developing a serious game with the objective to serve as a learning-based intervention within the ALEGAMS research project needed the articulation of several steps. The game was developed in several iterations; each version of the game, updated with a main new element, was tested with the target group (farmers). We observed that having adequate research design that allows flexibility, and complementary revision, is paramount. Based on literature, we developed a framework that included assumptions about learning outcomes the game would deliver. In ALEGAMS the degree to which the learning outcomes were reached was assessed within the project timeline. The latter helped for work to be framed within a period useful to make timely recommendations and thus contribute to innovation of serious games as a participatory tool. However, we recognize this might not always be possible for all projects. Projects aiming at a change of practice, or policy, should consider that this change can only be assessed over a longer while, which most likely goes beyond short projects.
Future research could try to add to this debate by discussing the challenges and issues faced with serious games in natural resource management. Active debate, based on empirical studies and experience, about the extent to which serious games can foster learning and social learning and then bring about change, should form the basis of this field of research.

Supplementary Materials

The following are available online at https://www.mdpi.com/2073-4441/11/2/245/s1. Figure S1: The players, Figure S2: The game.

Author Contributions

The idea for this conceptual paper originates in the work done as part to the ALEGAMS project for the “Research Protocol about the Assessment of the Potential for Role-play Games to Foster Learning” prepared by the first author with other authors. The conceptualization of the paper and review of selected literature has been done by R.R. Preparation of the illustrative case has been done by R.R., R.B. and A.L.

Funding

The ALEGAMS project was funded by IUCN and the WOTRO founding scheme of the Dutch Research Agency (NWO).

Acknowledgments

The authors wish to acknowledge the contributions during the game development of the project director Arnold Bregt, and the Vietnamese team members Tran Thi Phung Ha, Nguyen Huynh Thi Phuong and Tong Quoc Hiep.

Conflicts of Interest

The authors declare no conflict of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript; or in the decision to publish the results.

Abbreviations

ALEGAMSAssessing the Learning Effects of Games on Attitude of Stakeholders toward Sustainable Shrimp Farming
IMSIntegrated mangrove shrimp farming
IUCNInternational Union for the Conservation of Nature

References

  1. Flood, S.; Cradock-Henry, N.A.; Blackett, P.; Edwards, P. Adaptive and interactive climate futures: Systematic review of ‘serious games’ for engagement and decision-making. Environ. Res. Lett. 2018, 13, 063005. [Google Scholar] [CrossRef]
  2. Madani, K.; Pierce, T.W.; Mirchi, A. Serious games on environmental management. Sustain. Cities Soc. 2017, 29, 1–11. [Google Scholar] [CrossRef] [Green Version]
  3. Wesselow, M.; Stoll-Kleemann, S. Role-playing games in natural resource management and research: Lessons learned from theory and practice. Geogr. J. 2018, 184, 298–309. [Google Scholar] [CrossRef] [Green Version]
  4. de Vente, J.; Reed, M.S.; Stringer, L.C.; Valente, S.; Newig, J. How does the context and design of participatory decision making processes affect their outcomes? Evidence from sustainable land management in global drylands. Ecol. Soc. 2016, 21, 24. [Google Scholar] [CrossRef]
  5. Mayer, I.; Bekebrede, G.; Harteveld, C.; Warmelink, H.; Zhou, Q.; van Ruijven, T.; Lo, J.; Kortmann, R.; Wenzler, I. The research and evaluation of serious games: Toward a comprehensive methodology. Br. J. Educ. Technol. 2014, 45, 502–527. [Google Scholar] [CrossRef]
  6. Neys, J.; Jansz, J. Political Internet games: Engaging an audience. Eur. J. Commun. 2010, 25, 227–241. [Google Scholar] [CrossRef]
  7. Cai, Y.; Goei, S.L.; Trooster, W. Simulation and Serious Games for Education; Springer: Berlin, Germany, 2016. [Google Scholar]
  8. McGonigal, J. Reality Is Broken, Why Games Make Us Better and How They Can Change the World; The Penguin Press: New York, NY, USA, 2011. [Google Scholar]
  9. Wilson, K.A.; Bedwell, W.; Lazzara, E.; Salas, E.; Burke, S.; Estock, J.; Orvis, K.; Conkey, C. Relationships Between Game Attributes and Learning Outcomes: Review and Research Proposals. Simul. Gaming 2009, 40, 217–266. [Google Scholar] [CrossRef]
  10. Djaouti, D.; Alvarez, J.; Jessel, J.P. Classifying Serious Games: The G/P/S Model; IGI Global: Hershey, PA, USA, 2011. [Google Scholar]
  11. Baggio, J.A.; Rollins, N.D.; Pérez, I.; Janssen, M.A. Irrigation experiments in the lab: Trust, environmental variability, and collective action. Ecol. Soc. 2015, 20, 12. [Google Scholar] [CrossRef]
  12. Janssen, M.A.; Lee, A.; Waring, T.M. Experimental platforms for behavioral experiments on social-ecological systems. Ecol. Soc. 2014, 19, 20. [Google Scholar] [CrossRef]
  13. Vieira Pak, M.; Castillo Brieva, D. Designing and implementing a Role-Playing Game: A tool to explain factors, decision making and landscape transformation. Environ. Model. Softw. 2010, 25, 1322–1333. [Google Scholar] [CrossRef]
  14. Lebel, P.; Sriyasak, P.; Kallayanamitra, C.; Duangsuwan, C.; Lebel, L. Learning about climate-related risks: Decisions of Northern Thailand fish farmers in a role-playing simulation game. Reg. Environ. Chang. 2016, 16, 1481–1494. [Google Scholar] [CrossRef]
  15. Meinzen-Dick, R.; Janssen, M.A.; Kandikuppa, S.; Chaturvedi, R.; Rao, K.; Theis, S. Playing games to save water: Collective action games for groundwater management in Andhra Pradesh, India. World Dev. 2018, 107, 40–53. [Google Scholar] [CrossRef]
  16. Bell, A.R.; Ward, P.S.; Shah, M.A.A. Increased water charges improve efficiency and equity in an irrigation system. Ecol. Soc. 2016, 21, 23. [Google Scholar] [CrossRef]
  17. D’aquino, P.; Le Page, C.; Bousquet, F.; Bah, A. Using self-designed role-playing games and a multi-agent system to empower a local decision-making process for land use management: The SelfCormas experiment in Senegal. J. Artif. Soc. Soc. Simul. 2013, 6, 5. [Google Scholar]
  18. Ligtenberg, A.; van Lammeren, R.J.A.; Bregt, A.K.; Beulens, A.J.M. Validation of an agent-based model for spatial planning: A role-playing approach. Comput. Environ. Urban Syst. 2010, 34, 424–434. [Google Scholar] [CrossRef]
  19. Étienne, M. Sylvopast: A multiple target role-playing game to assess negotiation processes in sylvopastoral management planning. J. Artif. Soc. Soc. Simul. 2003, 6, 5. [Google Scholar]
  20. Campo, P.; Bousquet, F.; Villanueva, T. Modelling with stakeholders within a development project. Environ. Model. Softw. 2010, 25, 1302–1321. [Google Scholar] [CrossRef]
  21. Joffre, O.M.; Bosma, R.H.; Ligtenberg, A.; Tri, V.P.D.; Ha, T.T.P.; Bregt, A.K. Combining participatory approaches and an agent-based model for better planning shrimp aquaculture. Agric. Syst. 2015, 141, 149–159. [Google Scholar] [CrossRef]
  22. Gordon, S.; Thomas, I. ‘The learning sticks’: Reflections on a case study of role-playing for sustainability. Environ. Educ. Res. 2018, 24, 172–190. [Google Scholar] [CrossRef]
  23. Hoekstra, A.Y. Computer-supported games and role plays in teaching water management. Hydrol. Earth Syst. Sci. 2012, 16, 2985–2994. [Google Scholar] [CrossRef] [Green Version]
  24. Chen, J.C.; Martin, A.R. Role-Play Simulations as a Transformative Methodology in Environmental Education. J. Transform. Educ. 2015, 13, 85–102. [Google Scholar] [CrossRef]
  25. Meadows, D.L. Learning to Be Simple: My Odyssey with Games. Simul. Gaming 1999, 30, 342–351. [Google Scholar] [CrossRef]
  26. Paschall, M.; Wüstenhagen, R. More Than a Game: Learning About Climate Change Through Role-Play. J. Manag. Educ. 2011, 36, 510–543. [Google Scholar] [CrossRef]
  27. Pruksakorn, S.; Kiratiprayoon, S.; Uttaranakorn, S.; Sukreeyapongse, O.; Dumrongrojwatthana, P. Game for Low-Formal Education Farmers to Learn How to Improve Soil Quality. Simul. Gaming 2018, 49, 146–167. [Google Scholar] [CrossRef]
  28. Ferrero, G.; Bichai, F.; Rusca, M. Experiential Learning through Role-Playing: Enhancing Stakeholder Collaboration in Water Safety Plans. Water 2018, 10, 227. [Google Scholar] [CrossRef]
  29. Erisman, J.W.; Hensen, A.; de Vries, W.; Kros, H.; van de Wal, T.; de Winter, W.; Wien, J.E.; Elswijk, M.V.; Maat, M.; Sanders, K. NitroGenius: A Nitrogen Decision Support System. J. Hum. Environ. 2002, 31, 190–196. [Google Scholar] [CrossRef]
  30. Tran, T.P.H. Resilience and Livelihoods Dynamics of Shrimp Farmer and Fishers in the Mekong Delta, Vietnam. Ph.D. Thesis, Wageningen University, Wageningen, The Netherlands, 2012; pp. 117–138. [Google Scholar]
  31. Mezirow, J. Transformative Learning as Discourse. J. Transform. Educ. 2003, 1, 58–63. [Google Scholar] [CrossRef]
  32. Becu, N.; Amalric, M.; Anselme, B.; Beck, E.; Bertin, X.; Delay, E.; Long, N.; Marilleau, N.; Pignon-Mussaud, C.; Rousseaux, F. Participatory simulation to foster social learning on coastal flooding prevention. Environ. Model. Softw. 2017, 98, 1–11. [Google Scholar] [CrossRef]
  33. Medema, W.; Furber, A.; Adamowski, J.; Zhou, Q.; Mayer, I. Exploring the Potential Impact of Serious Games on Social Learning and Stakeholder Collaborations for Transboundary Watershed Management of the St. Lawrence River Basin. Water 2016, 8, 175. [Google Scholar] [CrossRef]
  34. Magnuszewski, P.; Królikowska, K.; Koch, A.; Pająk, M.; Allen, C.; Chraibi, V.; Giri, A.; Haak, D.; Hart, N.; Hellman, M.; et al. Exploring the Role of Relational Practices in Water Governance Using a Game-Based Approach. Water 2018, 10, 346. [Google Scholar] [CrossRef]
  35. Salvini, G.; van Paassen, A.; Ligtenberg, A.; Carrero, G.C.; Bregt, A.K. A role-playing game as a tool to facilitate social learning and collective action towards Climate Smart Agriculture: Lessons learned from Apuí, Brazil. Environ. Sci. Policy 2016, 63, 113–121. [Google Scholar] [CrossRef]
  36. Villamor, G.B.; Badmos, B.K. Grazing game: A learning tool for adaptive management in response to climate variability in semiarid areas of Ghana. Ecol. Soc. 2016, 21, 39. [Google Scholar] [CrossRef]
  37. Bourgoin, J.; Castella, J.-C. “PLUP FICTION”: Landscape Simulation for Participatory Land Use Planning in Northern Lao PDR. Mt. Res. Dev. 2011, 31, 78–88. [Google Scholar] [CrossRef]
  38. Craven, J.; Angarita, H.; Corzo Perez, G.A.; Vasquez, D. Development and testing of a river basin management simulation game for integrated management of the Magdalena-Cauca river basin. Environ. Model. Softw. 2017, 90, 78–88. [Google Scholar] [CrossRef]
  39. Ducrot, R.; van Paassen, A.; Barban, V.; Daré, W.S.; Gramaglia, C. Learning integrative negotiation to manage complex environmental issues: Example of a gaming approach in the peri-urban catchment of São Paulo, Brazil. Reg. Environ. Chang. 2015, 15, 67–78. [Google Scholar] [CrossRef]
  40. Gurung, T.R.; Bousquet, F.; Trébuil, G. Companion modeling, conflict resolution, and institution building: Sharing irrigation water in the Lingmuteychu Watershed, Bhutan. Ecol. Soc. 2006, 11, 36. [Google Scholar] [CrossRef]
  41. Souchère, V.; Millair, L.; Echeverria, J.; Bousquet, F.; Le Page, C.; Étienne, M. Co-constructing with stakeholders a role-playing game to initiate collective management of erosive runoff risks at the watershed scale. Environ. Model. Softw. 2010, 25, 1359–1370. [Google Scholar] [CrossRef]
  42. Rebolledo-Mendez, G.; Avramides, K.; De Freitas, S.; Memarzia, K. Societal impact of a serious game on raising public awareness: The case of FloodSim. Sandbox 2009, 9, 15–22. [Google Scholar]
  43. Bosma, R.H.; Nguyen, T.H.; Siahainenia, A.J.; Tran, H.T.; Tran, H.N. Shrimp-based livelihoods in mangrove silvo-aquaculture farming systems. Rev. Aquac. 2014, 6, 43–60. [Google Scholar] [CrossRef]
  44. Carson, A.; Windsor, M.; Hill, H.; Haigh, T.; Wall, N.; Smith, J.; Olsen, R.; Bathke, D.; Demir, I.; Muste, M. Serious gaming for participatory planning of multi-hazard mitigation. Int. J. River Basin Manag. 2018, 16, 379–391. [Google Scholar] [CrossRef]
  45. Van der Wal, M.M.; de Kraker, J.; Kroeze, C.; Kirschner, P.A.; Valkering, P. Can computer models be used for social learning? A serious game in water management. Environ. Model. Softw. 2016, 75, 119–132. [Google Scholar] [CrossRef] [Green Version]
  46. Gerlak, A.K.; Heikkila, T.; Smolinski, S.L.; Huitema, D.; Armitage, D. Learning our way out of environmental policy problems: A review of the scholarship. Policy Sci. 2017, 51, 335–371. [Google Scholar] [CrossRef]
  47. Suškevičs, M.; Hahn, T.; Rodela, R.; Macura, B.; Pahl-Wostl, C. Learning for social-ecological change: A qualitative review of outcomes across empirical literature in natural resource management. J. Environ. Plan. Manag. 2018, 1, 1085–1112. [Google Scholar] [CrossRef]
  48. Baird, J.; Plummer, R.; Haug, C.; Huitema, D. Learning effects of interactive decision-making processes for climate change adaptation. Glob. Environ. Chang. 2014, 27, 51–63. [Google Scholar] [CrossRef]
  49. Ensor, J.; Harvey, B. Social learning and climate change adaptation: Evidence for international development practice. Wiley Interdiscip. Rev. Clim. Chang. 2015, 6, 509–522. [Google Scholar]
  50. Rodela, R. Social Learning, Natural Resource Management, and Participatory Activities: A reflection on construct development and testing. Wagening. J. Life Sci. 2014, 69, 15–22. [Google Scholar] [Green Version]
  51. Rodela, R.; Cundill, G.; Wals, A.E.J. An analysis of the methodological underpinnings of social learning research in natural resource management. Ecol. Econ. 2012, 77, 16–26. [Google Scholar] [CrossRef]
  52. Pahl-Wostl, C.; Hare, M. Processes of Social Learning in Integrated Resources Management. J. Community Appl. Soc. Psychol. 2004, 14, 193–206. [Google Scholar] [CrossRef]
  53. Shackleton, C.M.; Cundill, G.; Knight, A.T. Beyond Just Research: Experiences from Southern Africa in Developing Social Learning Partnerships for Resource Conservation Initiatives. Biotropica 2009, 41, 563–570. [Google Scholar] [CrossRef]
  54. Cundill, G.; Rodela, R. A search for coherence: Social learning in natural resource management. Rev. Soc. Learn. Lit. 2012, 5, 31–38. [Google Scholar]
  55. Vinke-de Kruijf, J.; Bressers, H.; Augustijn, D.C.M. How social learning influences further collaboration: Experiences from an international collaborative water project. Ecol. Soc. 2014, 19, 61. [Google Scholar] [CrossRef]
  56. Rodela, R.; Pérez-Soba, M.; Bregt, A.; Verweij, P. Spatial decision support systems: Exploring differences in pilot-testing with students vs. professionals. Comput. Environ. Urban Syst. 2018, 72, 204–211. [Google Scholar] [CrossRef]
Figure 1. Differences in the use of serious games in natural resource and environmental governance.
Figure 1. Differences in the use of serious games in natural resource and environmental governance.
Water 11 00245 g001

Share and Cite

MDPI and ACS Style

Rodela, R.; Ligtenberg, A.; Bosma, R. Conceptualizing Serious Games as a Learning-Based Intervention in the Context of Natural Resources and Environmental Governance. Water 2019, 11, 245. https://doi.org/10.3390/w11020245

AMA Style

Rodela R, Ligtenberg A, Bosma R. Conceptualizing Serious Games as a Learning-Based Intervention in the Context of Natural Resources and Environmental Governance. Water. 2019; 11(2):245. https://doi.org/10.3390/w11020245

Chicago/Turabian Style

Rodela, Romina, Arend Ligtenberg, and Roel Bosma. 2019. "Conceptualizing Serious Games as a Learning-Based Intervention in the Context of Natural Resources and Environmental Governance" Water 11, no. 2: 245. https://doi.org/10.3390/w11020245

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop