Abstract
Prevention curricula rely on audience engagement to effectively communicate their messages. However, to date, measurement of engagement has primarily focused on self-report that is often an indicator of liking or satisfaction. Emerging technologies for intervention delivery hold promise not only for additional engagement indicators but also for dissemination outside of traditional vehicles such as classroom delivery. The present study, grounded in the theory of active involvement (Greene 2013), explores the role of engagement (as measured by self-report, program analytics, and observation) with short-term substance use prevention outcomes such as self-efficacy to counter-argue and descriptive and injunctive norms. The study tracks 4-H youth (N = 310) engaged with a media literacy focused e-learning substance prevention curriculum, REAL media. Results indicate that self-reports of engagement predicted self-efficacy to counter-argue, but a program-analytic indicator of dosage predicted lower injunctive and descriptive norms, all at 3 months. The observational indicator was correlated with self-efficacy to counter-argue but not significant in the predictive models. The implications and directions for future research regarding how engagement is measured in prevention and included in studying program effects are discussed. Clinical trial: NCT03157700, May 2017.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Appleton, J. J., Christenson, S. L., Kim, D., & Reschly, A. L. (2006). Measuring cognitive and psychological engagement: Validation of the student engagement instrument. Journal of School Psychology, 44, 427–445. https://doi.org/10.1016/j.jsp.2006.04.002.
Appleton, J. J., Christenson, S. L., & Furlong, M. J. (2008). Student engagement with school: Critical conceptual and methodological issues of the construct. Psychology in the Schools, 45, 369–386. https://doi.org/10.1002/pits.20303.
Arnett, J. (1992). Reckless behavior in adolescence: A developmental perspective. Developmental Review, 12, 339–373. https://doi.org/10.1016/0273-2297(92)90013-R.
Bandura, A. (1986). The explanatory and predictive scope of self-efficacy theory. Journal of Social and Clinical Psychology, 4, 359–373. https://doi.org/10.1521/jscp.1986.4.3.359.
Banerjee, S. C., & Greene, K. (2007). Anti-smoking initiatives: Effects of analysis versus production media literacy interventions on smoking-related attitude, norm, and behavioral intention. Health Communication, 22, 37–48. https://doi.org/10.1080/10410230701310281.
Banerjee, S. C., Greene, K., Magsamen-Conrad, K., Elek, E., & Hecht, M. L. (2015). Interpersonal communication outcomes of a media literacy alcohol prevention curriculum. Translational Behavioral Medicine, 5, 425–432. https://doi.org/10.1007/s13142-015-0329-9.
Berkel, C., Mauricio, A. M., Schoenfelder, E., & Sandler, I. N. (2011). Putting the pieces together: An integrated model of program implementation. Prevention Science, 12, 23–33. https://doi.org/10.1007/s11121-010-0186-1.
Burleson, B. R., & Waltman, M. S. (1988). Complexity: Using the Role Category Questionnaire measure. In C. Tardy (Ed.), A handbook for the study of human communication: Methods and instruments for observing, measuring, and assessing communication processes (pp. 1–35). Norwood, NJ: Abex Publishing.
Colby, M., Hecht, M. L., Miller-Day, M., Krieger, J. R., Syverstsen, A. K., Graham, J. W., & Pettigrew, J. (2013). Adapting school-based substance use prevention curriculum through cultural grounding: An exemplar of adaptation processes for rural schools. American Journal of Community Psychology, 51, 190–205. https://doi.org/10.1007/s10464-012-9524-8.
Colquitt, J. A., LePine, J. A., & Noe, R. A. (2000). Toward an integrative theory of training motivation: A meta-analytic path analysis of 20 years of research. Journal of Applied Psychology, 85, 678–707.
Crockett, W. H. (1965). Cognitive complexity and impression formation. In B. E. Maher (Ed.), Progress in experimental personality research (Vol. 2, pp. 74–90). New York: Academic Press.
Davis, K., Sridharan, H., Koepke, L., Singh, S., & Boiko, R. (2018). Learning and engagement in a gamified course: Investigating the effects of student characteristics. Journal of Computer Assisted Learning, 34, 492–503. https://doi.org/10.1111/jcal.12254.
Dopp, A. R., Parisi, K. E., Munson, S. A., & Lyon, A. R. (2019). A glossary of user-centered design strategies for implementation experts. Translational Behavioral Medicine, 9, 1057–1064. https://doi.org/10.1093/tbm/iby119.
Durlak, J., Weissberg, R., & Pachan, M. (2010). A meta-analysis of after-school programs that seek to promote personal and social skills in children and adolescents. American Journal of Community Psychology, 45, 294–309. https://doi.org/10.1007/s10464-010-9300-6.
Dusenbury, L., Brannigan, R., Hansen, W. B., Walsh, J., & Falco, M. (2005). Quality of implementation: Developing measures crucial to understanding the diffusion of preventive interventions. Health Education Research, 20, 308–313. https://doi.org/10.1093/her/cyg134.
Dusenbury, L., Hansen, W. B., Jackson-Newsom, J., Pittman, D., Wilson, C., Simley, K., et al. (2010). Coaching to enhance quality of implementation in prevention. Health Education, 110, 43–60. https://doi.org/10.1108/09654281011008744.
Elek, E., Miller-Day, M., & Hecht, M. L. (2006). Influences of personal, injunctive, and descriptive norms on early adolescent substance use. Journal of Drug Issues, 36, 147–171. https://doi.org/10.1177/002204260603600107.
Fredricks, J. A., & McColskey, W. (2012). The measurement of student engagement: A comparative analysis of various methods and student self-report instruments. In S. L. Christenson, A. L. Reschly, & C. Wylie (Eds.), Handbook of research on student engagement (pp. 763–782). Boston, MA: Springer US. https://doi.org/10.1007/978-1-4614-2018-7_37.
Fredricks, J. A., Blumenfeld, P. C., & Paris, A. H. (2004). School engagement: Potential of the concept, state of the evidence. Review of Educational Research, 74, 59–109. https://doi.org/10.3102/00346543074001059.
Gottfredson, D. C., Cross, A., Wilson, D., Rorie, M., & Connell, N. (2010). An experimental evaluation of the all stars prevention curriculum in a community after school setting. Prevention Science, 11, 142–154. https://doi.org/10.1007/s11121-009-0156-7.
Greene, K. (2013). The theory of active involvement: Processes underlying interventions that engage adolescents in message planning and/or production. Health Communication, 28, 644–656. https://doi.org/10.1080/10410236.2012.762824.
Greene, K., & Hecht, M. L. (2013). Introduction for symposium on engaging youth in prevention message creation: The theory and practice of active involvement interventions. Health Communication, 28, 641–643. https://doi.org/10.1080/10410236.2012.762825.
Greene, K., Yanovitzky, I., Carpenter, A., Banerjee, S. C., Magsamen-Conrad, K., Hecht, M. L., & Elek, E. (2015). A theory-grounded measure of adolescents’ response to a media literacy intervention. Journal of Media Literacy Education, 7, 35–49.
Greene, K., Catona, D., Elek, E., Magsamen-Conrad, K., Banerjee, S. C., & Hecht, M. L. (2016). Improving prevention curricula: Lessons learned through formative research on the Youth Message Development Curriculum. Journal of Health Communication, 21, 1071–1078. https://doi.org/10.1080/10810730.2016.1222029.
Greene, K., Ray, A. E., Choi, H. J., Glenn, S. D., Lyons, R. E., & Hecht, M. L. (2020). Short-term effects of the REAL media e-learning media literacy substance prevention curriculum: An RCT of adolescents disseminated through a community organization. Drug and Alcohol Dependence, 214. https://doi.org/10.1016/j.drugalcdep.2020.108170.
Hansen, W. B., Graham, J. W., Wolkenstein, B. H., Lundy, B. Z., Pearson, J., Flay, B. R., & Anderson, J. (1998). Differential impact of three alcohol prevention curricula on hypothesized mediating variables. Journal of Drug Education, 18, 143–153. https://doi.org/10.2190/FLQ5-9KNJ-92TH-WCDF.
Hansen, W. B., Fleming, C. B., & Scheier, L. (2019). Self-reported engagement in a drug prevention program: Individual and classroom effects on proximal and behavioral outcomes. Journal of Primary Prevention, 40, 5–34. https://doi.org/10.1007/s10935-018-00532-1.
Henrie, C., Halverson, L. R., & Graham, C. R. (2015). Measuring student engagement in technology-mediated learning: A review. Computers & Education, 90, 36–53. https://doi.org/10.1016/j.compedu.2015.09.005.
Hochheiser, H., & Lazar, J. (2007). HCI and societal issues: A framework for engagement. International Journal of Human-Computer Interaction, 23, 339–374. https://doi.org/10.1080/10447310701702717.
Li, J., Wong, S. C., Yang, X., & Bell, A. (2020). Using feedback to promote student participation in online learning programs: Evidence from a quasi-experimental study. Educational Technology Research and Development, 68, 485–510. https://doi.org/10.1007/s11423-019-09709-9.
Lillehoj, C. J., Griffin, K. W., & Spoth, R. (2004). Program provider and observer ratings of school-based preventive intervention implementation: Agreement and relation to youth outcomes. Health Education & Behavior, 31, 242–257. https://doi.org/10.1177/1090198103260514.
Low, S., Van Ryzin, M. J., Brown, E. C., Smith, B. H., & Haggerty, K. P. (2014). Engagement matters: Lessons from assessing classroom implementation of steps to respect: A bullying prevention program over a one-year period. Prevention Science, 15, 165–176. https://doi.org/10.1007/s11121-012-0359-1.
Lyon, A. R., & Bruns, E. J. (2019). User-centered redesign of evidence-based psychosocial interventions to enhance implementation: Hospitable soil or better seeds? JAMA Psychiatry, 76, 3–4. https://doi.org/10.1001/jamapsychiatry.2018.3060.
Ma, L., & Lee, C. S. (2018). Investigating the adoption of MOOCs: A technology-user-environment perspective. Journal of Computer Assisted Learning, 34, 1–10. https://doi.org/10.1111/jcal.12314.
Macfadyen, L. P., & Dawson, S. (2012). Numbers are not enough. Why e-learning analytics failed to inform an institutional strategic plan. Journal of Educational Technology and Society, 15, 149–163.
Marsch, L. A., & Borodovsky, J. T. (2016). Technology-based interventions for preventing and treating substance use among youth. Child and Adolescent Psychiatric Clinics of North America, 25, 755–768. https://doi.org/10.1016/j.chc.2016.06.005.
Morris, L. V., Finnegan, C., & Wu, S. S. (2005). Tracking student behaviours, persistence, and achievement in online courses. Internet and Higher Education, 8, 221–231. https://doi.org/10.1016/j.iheduc.2005.06.009.
Muench, F. (2014). The promises and pitfalls of digital technology in its application to alcohol treatment. Alcohol Research: Current Reviews, 36, 131–142.
O’Keefe, D. J., & Sypher, H. E. (1981). Complexity measures and the relationship of complexity to communication. Human Communication Research, 8, 72–92. https://doi.org/10.1111/j.1468-2958.1981.tb00657.x.
Patrick, K., Hekler, E. B., Estrin, D., Mohr, D. C., Riper, H., Crane, D., et al. (2016). The pace of technologic change: Implications for digital health behavior intervention research. American Journal of Preventive Medicine, 51, 816–824. https://doi.org/10.1016/j.amepre.2016.05.001.
Peled, A., & Rashty, D. (1999). Logging for success: Advancing the use of the WWW logs to improve computer mediated distance learning. Journal of Educational Computing Research, 21, 413–431. https://doi.org/10.2190/NLR6-K355-LAQY-U01D.
Pettigrew, J., & Hecht, M. L. (2015). Developing prevention curricula. In K. Bosworth (Ed.), Prevention science in school settings: Complex relationships and processes (pp. 151–174). NY: Springer.
Pettigrew, J., Graham, J. W., Miller-Day, M., Hecht, M. L., Krieger, J. L., & Shin, Y. J. (2015). Adherence and delivery: Implementation quality and program outcomes for the 7th-grade Keepin’ it REAL program. Prevention Science, 16, 90–99. https://doi.org/10.1007/s11121-014-0459-1.
Pössel, P., Baldus, C., Horn, A. B., Groen, G., & Hautzinger, M. (2005). Influence of general self-efficacy on the effects of a school-based universal primary prevention program of depressive symptoms in adolescents: A randomized and controlled follow-up study. Journal of Child Psychology and Psychiatry, 46, 982–994. https://doi.org/10.1111/j.1469-7610.2004.00395.x.
Pradhan, A. M., Park, L., Shaya, F. T., & Finkelstein, J. (2019). Consumer health information technology in the prevention of substance abuse: Scoping review. Journal of Medical Internet Research: Formative Research, 21, e11297. https://doi.org/10.2196/11297.
Ray, A. E., Greene, K., Hecht, M. L., Barriage, S. C., Miller-Day, M., Glenn, S. D., et al. (2019). An e-learning adaptation of an evidence-based media literacy curriculum to prevent youth substance use in community groups: Development and feasibility of REAL media. Journal of Medical Internet Research: Formative Research, 3, e12134. https://doi.org/10.2196/12132.
Ray, A. E., Greene, K., Pristavec, T., Miller-Day, M. A., Banerjee, S. C., & Hecht, M. L. (2020). Exploring indicators of engagement in online learning as applied to adolescent health prevention: A pilot study of REAL media. Educational Technology Research and Development. https://doi.org/10.1007/s11423-020-09813-1.
Reynolds, R., & Chiu, M. M. (2016). Reducing digital divide effects through student engagement in coordinated game design, online resource use, and social computing activities in school. JASIST, 67, 1822–1835.
Salomon, G. (1984). Television is "easy" and print is “tough”: The differential investment of mental effort in learning as a function of perceptions and attributions. Journal of Educational Psychology, 76, 647–658. https://doi.org/10.1037/0022-0663.76.4.647.
Soffer, T., & Nachmias, R. (2018). Effectiveness of learning in online academic courses compared with face-to-face courses in higher education. Journal of Computer Assisted Learning, 34, 534–543.
Tobler, N. S., Roona, M. R., Ochshorn, P., Marshall, D. G., Streke, A. V., & Stackpole, K. M. (2000). School-based adolescent drug prevention programs: 1998 meta-analysis. The Journal of Primary Prevention, 20, 275–336. https://doi.org/10.1023/A:1021314704811.
Wagner, P., Schober, B., & Spiel, C. (2008). Time students spend working at home for school. Learning and Instruction, 18, 309–320. https://doi.org/10.1016/j.learninstruc.2007.03.002.
Wellman, G. S., & Marcinkiewicz, H. (2004). Online learning and time-on-task: Impact of proctored vs. un-proctored testing. Journal for Asynchronous Learning Networks, 8, 93–104.
Acknowledgments
We gratefully acknowledge the contributions of the 4-H clubs and their members, particularly Rachel E. Lyons of 4-H and Rutgers University.
Funding
This study is supported by NIDA/NIH (R41DA039595, R42DA039595).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Ethics Approval
All procedures involving human participants were in accordance with institutional review board ethical standards and with the 1964 Helsinki declaration and its later amendments. IRB protocol approval #15-544Rc (Rutgers University).
Disclosure of Potential Conflicts of Interest
Kathryn Greene and Michael Hecht disclose intellectual property interests in the REAL media curriculum.
Informed Consent
We obtained written parental consent and online youth assent for all participants.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Electronic Supplementary Material
ESM 1
(DOCX 84 kb)
Rights and permissions
About this article
Cite this article
Greene, K., Choi, H.J., Glenn, S.D. et al. The Role of Engagement in Effective, Digital Prevention Interventions: the Function of Engagement in the REAL Media Substance Use Prevention Curriculum. Prev Sci 22, 247–258 (2021). https://doi.org/10.1007/s11121-020-01181-9
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11121-020-01181-9