Measurement and Evaluation

Section

Toolkit Home Brief Orientation Key Language and Concepts Key Literature Community and Partner Engagement Reflectivity in Research Practice Implementation Science Frameworks Measurement and Evaluation Data Collection and Analysis Implementation Trials

Section lead: Jinying Chen, PhD

Identifying Relevant Methods and Study Designs for Investigating Health Equity in Implementation Science

This section gives a brief overview of methods and study designs that were used or recommended for use by investigators that integrate a primary or secondary focus on health equity in implementation science. It is not an exhaustive list and is meant to be a resource for people new to this research area. Following McNulty and colleagues’ guidance in “Implementation Research Methodologies for Achieving Scientific Equity and Health Equity,”1 we categorized methods and study designs in Table 4 into three paradigms: 1) research using existing data to assess health inequity and related factors; 2) research including populations experiencing health inequities; and 3) research focusing on addressing health equity. Some study designs and methods may apply to all the three research paradigms; while others may be more relevant to a single paradigm than others.

Table 4. Methods and Study Designs for Investigating Health Equity in Implementation Science

Methods/Designs Definition or Representative Methods Uses Citations for Example Studies
Quantitative methods: Mostly applied in paradigm 1 and outcome evaluation in paradigms 2 and 3
Descriptive analysis Presenting the adoption rates of EBIs or outcomes from IS studies for underrepresented population against a standard or a control group Assess the extent of population-level disparities Glasgow et al., 20132
Neighbors et al., 20073
Analytic epidemiologic methods Regression analysis to assess the association between social determinants of health and outcomes from IS studies; includes regression model, mixed effects model, moderated regression, and mediation analysis Assess factors associated with health disparities or mechanisms that can explain them Morgan et al., 20184
Agent-based modeling Building computational models to simulate the actions and interactions between entities (called “agents”) of a complex system Assess the likely impact of specific implementation strategies on reducing disparities Brown et al., 20155
Brown et al., 20136
Janulis et al., 20187
Methods for community input/needs assessment: Mostly used in paradigms 2 and 3
Qualitative evaluation Semi-structured interviews, focus-group discussions Understand the needs of marginalized communities or contextual factors that contributed to health disparity Shelton et al., 20228
Ramanadhan et al., 20219
Allen et al., 202110
Community-based participatory research (CBPR) A collaborative approach that involves community partners in all phases of the research process, aiming to increase local relevance and reduce health disparities11; CBPR adopted a variety of research methods, such as intervention mapping, transcreation, and implementation mapping Understand the health equity-related issues in the local context and identify strategies or interventions to address these issues Tomayko et al., 201912
Intervention mapping A planning framework that “provides a systematic process and detailed protocol for effective, step-by-step decision-making for intervention development, implementation, and evaluation”13 Provide guidance on how and when to use evidence, theory, and community-based participation during the implementation process Holcomb et al., 202114
Implementation mapping A systematic process, by extending intervention mapping, for planning or selecting implementation strategies 15 Same as above Ibekwe et al., 202216
Study designs: Paradigms 2 and 3
Hybrid effectiveness-implementation trial A study design that “takes a dual focus a priori in assessing clinical effectiveness and implementation”, including three hybrid types I-III17 Assess effectiveness and the implementation process and outcomes of interventions that address health disparities Smith et al., 201818
Pragmatic non-randomized trial Quasi-experimental study designs that compare effects of interventions between non-randomized intervention group and control group Compare effects of implementation programs or interventions that aim to address health disparity in the clinical setting Cykert et al., 202019
Sequential multiple assignment randomized trial (SMART) A randomized experimental design for building time-varying adaptive interventions20 Compare combinations of interventions and implementation strategies for underserved populations Johnson et al., 201821
Mixed methods design A study design that combines quantitative and qualitative research methods22 Assess barriers and facilitators to the implementation of EBI or adaptation of implementation programs in underserved populations Shelton et al., 202123

References

  1. McNulty M, Smith JD, Villamar J, et al. Implementation research methodologies for achieving scientific equity and health equity. Ethn Dis. 2019;29(Suppl 1):83-92. doi:10.18865/ed.29.S1.83
  2. Glasgow RE, Askew S, Purcell P, et al. Use of RE-AIM to address health inequities: Application in a low-income community health center-based weight loss and hypertension self-management program. Transl Behav Med. 2013;3(2):200-210. doi:10.1007/s13142-013-0201-8
  3. Neighbors HW, Caldwell C, Williams DR, et al. Race, ethnicity, and the use of services for mental disorders: Results from the National Survey of American Life. Arch Gen Psychiatry. 2007;64(4):485-494. doi:10.1001/archpsyc.64.4.485
  4. Morgan E, Moran K, Ryan DT, Mustanski B, Newcomb ME. Threefold increase in PrEP uptake over time with high adherence among young men who have sex with men in Chicago. AIDS Behav. 2018;22(11):3637-3644. doi:10.1007/s10461-018-2122-5
  5. Brown CH, PoVey C, Hjorth A, Gallo CG, Wilensky U, Villamar J. Computational and technical approaches to improve the implementation of prevention programs. Paper presented at: Implementation Science; 2015.
  6. Brown CH, Mohr DC, Gallo CG, et al. A computational future for preventing HIV in minority communities: How advanced technology can improve implementation of effective programs. J Acquir Immune Defic Syndr. 2013;63(Suppl 1):S72-S84. doi:10.1097/QAI.0b013e31829372bd
  7. Janulis P, Phillips G, Birkett M, Mustanski B. Sexual networks of racially diverse young MSM differ in racial homophily but not concurrency. J Acquir Immune Defic Syndr. 2018;77(5):459-466. doi:10.1097/QAI.0000000000001620
  8. Shelton RC, Philbin MM, Ramanadhan S. Qualitative research methods in chronic disease: Introduction and opportunities to promote health equity. Annu Rev Public Health. 2022;43:37-57. doi:10.1146/annurev-publhealth-012420-105104
  9. Ramanadhan S, Revette AC, Lee RM, Aveling EL. Pragmatic approaches to analyzing qualitative data for implementation science: An introduction. Implement Sci Commun. 2021;2(1):70. doi:10.1186/s43058-021-00174-1
  10. Allen M, Wilhelm A, Ortega LE, Pergament S, Bates N, Cunningham, B. Applying a race(ism)-conscious adaptation of the CFIR framework to understand implementation of a school-based equity-oriented intervention. Ethn Dis. 2021;31(Suppl 1):375-388. doi:10.18865/ed.31.S1.375
  11. Ramanadhan S, Davis MM, Armstrong R, et al. Participatory implementation science to increase the impact of evidence-based cancer prevention and control. Cancer Causes Control. 2018;29(3):363-369. doi:10.1007/s10552-018-1008-1
  12. Tomayko EJ, Prince RJ, Cronin KA, Kim K, Parker T, Adams AK. The Healthy Children, Strong Families 2 (HCSF2) randomized controlled trial improved healthy behaviors in American Indian families with young children. Curr Dev Nutr. 2019;3(Suppl 2):53-62. doi:10.1093/cdn/nzy087
  13. Fernandez ME, Ruiter RAC, Markham CM, Kok G. Intervention mapping: Theory- and evidence-based health promotion program planning: Perspective and examples. Front Public Health. 2019;7:209. doi:10.3389/fpubh.2019.00209
  14. Holcomb J, Ferguson GM, Sun J, Walton GH, Highfield L. Stakeholder engagement in adoption, implementation, and sustainment of an evidence-based intervention to increase mammography adherence among low-income women. J Cancer Educ. 2021;37:1486-1495. doi:10.1007/s13187-021-01988-2
  15. Fernandez ME, ten Hoor GA, van Lieshout S, et al. Implementation mapping: Using intervention mapping to develop implementation strategies. Front Public Health. 2019;7:158. doi:10.3389/fpubh.2019.00158
  16. Ibekwe LN, Walker TJ, Ebunlomo E, et al. Using implementation mapping to develop implementation strategies for the delivery of a cancer prevention and control phone navigation program: A collaboration with 2-1-1. Health Promot Pract. 2022;23(1):86-97. 10.1177/1524839920957979
  17. Curran GM, Bauer M, Mittman B, Pyne JM, Stetler C. Effectiveness-implementation hybrid designs: Combining elements of clinical effectiveness and implementation research to enhance public health impact. Med Care. 2012;50(3):217-226. 10.1097/MLR.0b013e3182408812
  18. Smith JD, Berkel C, Jordan N, et al. An individually tailored family-centered intervention for pediatric obesity in primary care: study protocol of a randomized type II hybrid effectiveness-implementation trial (Raising Healthy Children study). Implement Sci. 2018;13(1):11. doi:10.1186/s13012-017-0697-2
  19. Cykert S, Eng E, Manning MA, et al. A multi-faceted intervention aimed at Black-White disparities in the treatment of early stage cancers: The ACCURE Pragmatic Quality Improvement trial. J Natl Med Assoc. 2020;112(5):468-477. doi:10.1016/j.jnma.2019.03.001
  20. Collins LM, Murphy SA, Strecher V. The multiphase optimization strategy (MOST) and the sequential multiple assignment randomized trial (SMART): New methods for more potent eHealth interventions. Am J Prev Med. 2007;32(5 Suppl):S112-S118. doi:10.1016/j.amepre.2007.01.022
  21. Johnson JE, Wiltsey-Stirman S, Sikorskii A, et al. Protocol for the ROSE sustainment (ROSES) study, a sequential multiple assignment randomized trial to determine the minimum necessary intervention to maintain a postpartum depression prevention program in prenatal clinics serving low-income women. Implement Sci. 2018;13(1):115. doi:10.1186/s13012-018-0807-9
  22. Palinkas LA, Aarons GA, Horwitz S, Chamberlain P, Hurlburt M, Landsverk J. Mixed method designs in implementation research. Adm Policy Ment Health. 2011;38(1):44-53. doi:10.1007/s10488-010-0314-z
  23. Shelton RC, Brotzman LE, Johnson D, Erwin D. Trust and mistrust in shaping adaptation and de-implementation in the context of changing screening guidelines. Ethn Dis. 2021;31(1):119-132. doi:10.18865/ed.31.1.119