ClinicalTrials.gg
Enrolling by invitationNCT04356274Updated May 4, 2026

Local Participatory Systems Dynamics to Increase Reach of Evidence Based Addiction and Mental Health Care

An interventional study of Participatory System Dynamics (PSD) and Audit and Feedback (AF) in PTSD, Depression and Alcohol Use Disorder, sponsored by Palo Alto Veterans Institute for Research. Enrolling by invitation at 1 site in United States. Open to participants aged 18 Years and older, including healthy volunteers. Per ClinicalTrials.gov, last updated 2026-05-04.

Sponsored by Palo Alto Veterans Institute for Research · Not applicable, Interventional, and Health services research

From the registry’s dates

  • Primary completion was expected by Jul 2026, 3 months ago, but the record still lists the study as enrolling by invitation.
Phase
Not applicable
Study type
Interventional
Enrollment
720
Allocation
Randomized
Ages
18 Years and older
Sex
All
01

Study summary

The most common reasons Veterans seek VA addiction and mental health care is for help with opioid and alcohol misuse, depression and PTSD. Research evidence has established highly effective treatments that prevent relapse, overdose and suicide, but even with policy mandates, performance metrics, and electronic health records to fix the problem, these treatments may only reach 3-28% of patients. This study tests participatory business engineering methods (Participatory System Dynamics) that engage patients, providers and policy makers against the status quo approaches, such as data review, and will determine if participatory system dynamics works, why it works, and whether it can be applied in many health care settings to guarantee patient access to the highest quality care and better meet the addiction and mental health needs of Veterans and the U.S. population.

Read the detailed description

The broad aim is to empower all healthcare stakeholders to provide the highest quality care to all patients. The specific aims address the complexities and tradeoffs associated with implementing evidence-based practices (EBPs) in outpatient addiction and mental health systems. There is scientific consensus about the best evidence-based psychotherapies and pharmacotherapies (EBPs) to meet the needs of patients with opioid and alcohol use disorder, PTSD and depression. However, EBP coordination over time, within and across multidisciplinary teams of providers, is complex and constantly changing. Veterans Health Administration (VA) policy mandates, national training programs, and incentivized quality measures, have been insufficient for reaching more than 3 to 28% of patients with the highest quality treatments. In fact, limited EBP reach is common in health systems and the field of implementation science seeks to address it. One routine strategy is data auditing with provider feedback (audit-and-feedback; AF), however, the impact is highly variable. As an alternative, participatory system dynamics (PSD) has been used to explain causes of complex problems in business management for 60 years. Partnering with frontline staff using PSD to determine how EBP reach emerges from local resources and constraints, and is determined by system dynamics, such as delays and feedback. The dynamics of EBP reach were formally specified in differential equation models, and tested against VA data drawn from a national VA SQL database. Using existing enterprise data to tailor model parameters to each care team. PSD models were made accessible via a 'Modeling to Learn' interface and training, during which teams safely evaluated local change scenarios via simulation to find the highest yield options for meeting Veterans' needs. PSD learning simulations produce immediate, real-time feedback to the teams who coordinate care, improving day-to-day decisions and long-term improvement plans. The study design is a two-arm, 24-site (12 sites/arm) cluster randomized trial to test the effectiveness of PSD simulation as compared to more standard team AF data review. The hypothesis is that PSD will be superior to AF for improving EBP initiation and dose (Aim 1). The investigators will test the PSD theory of change that the effect of PSD on improved EBP reach is explained by improvement in team systems thinking (Aim 2). To confirm the potential for widespread usefulness of PSD, by also testing the generalizability of PSD causal dynamics across PSD and AF arms (Aim 3). This study has the potential to inform a new paradigm, by determining what works to improve health system quality defined as EBP reach, why it works, and under what conditions. If PSD is effective, study activities will address a national priority to improve Veterans' addiction and mental health care to prevent chronic symptoms, relapse, suicide and overdose. Findings from the proposed tests of effectiveness, causality, and generality, could also catalyze future applications to make a significant public health impact across the continuum of healthcare.

02

Conditions studied

  • PTSD
  • Depression
  • Alcohol Use Disorder
  • Opioid Use Disorder

Keywords

  • implementation science
  • quality improvement
  • participatory system dynamics
  • audit and feedback
  • evidence-based psychotherapy
  • evidence-based pharmacotherapy
  • addiction
  • mental health/behavioral health
03

In context

Stress Disorders, Post-Traumatic

2,682 studies on the registry are indexed under Stress Disorders, Post-Traumatic; 601 are open to participants now.

This study's planned enrollment of 720 is above the median of 69 across 2,196 interventional studies indexed under Stress Disorders, Post-Traumatic.

Browse Stress Disorders, Post-Traumatic studies →

Lead sponsor

Palo Alto Veterans Institute for Research is the lead sponsor of 53 studies on the registry; 10 are open to participants now.

Counted across the registry records on this site, refreshed daily.

04

Who can participate

Ages eligible
18 Years and older
Sexes eligible
All
Accepts healthy volunteers
Yes

Go through the criteria one by one — yes, no or not sure — and take the list to your doctor or the study team. Kept in this browser only.

Inclusion criteria

  • VA divisions and community-based outpatient clinics (CBOCs) or 'clinics' from regional VA health systems
  • Must be below the overall VA quality median (as assessed by the Strategic Analytics for Improvement and Learning or SAIL), which includes 3 of 8 SAIL measures associated with four evidence-based psychotherapies and three evidence-based pharmacotherapies for depression, PTSD, and opioid use disorder.

Exclusion criteria

Exclusion Criteria Clinics:

  • clinics with less than 12 months of data in 2018
  • clinics already involved in Office of Veterans Access to Care (OVACS) quality improvement program at baseline.
  • clinics where the VA Cerner electronic health record (EHR) implementation rollout will occur during the project period (Veterans Integrated Services Networks (VISNs) 20, 21 ,22, and 7)
  • clinics who serve less than 122 unique patients each month on average
  • clinics without an onsite multidisciplinary team of mental health or addiction service providers (minimum required: 1 psychiatrist, 1 psychologist, 1 social worker onsite)
05

Study design

Phase
Not applicable
Primary purpose
Health services research
Allocation
Randomized
Intervention model
Parallel assignment
Masking
None (open label)
Enrollment
720 participants (estimated)

Study arms

  • Experimental
    Participatory System Dynamics (PSD)

    12 clinics assigned to PSD

    Other: Participatory System Dynamics (PSD)

  • Experimental
    Audit and Feedback (AF)

    12 clinics assigned to AF

    Other: Audit and Feedback (AF)

Interventions

  • OtherParticipatory System Dynamics (PSD)

    Participatory system dynamics is a facilitated health care quality improvement or evidence-based practice implementation strategy that includes frontline addiction and mental health staff running simulations of clinic improvement strategies to find the best approaches for improving the reach of evidence-based psychotherapy and evidence-based pharmacotherapy.

  • OtherAudit and Feedback (AF)

    Audit and feedback is a health care quality improvement or evidence-based practice implementation strategy that includes frontline addiction and mental health staff reviewing clinical care team data to find the best approaches for improving the reach of evidence-based psychotherapy and evidence-based pharmacotherapy.

06

What researchers measure

Primary outcomes

  1. Proportion of patients diagnosed with alcohol use disorder, depression, opioid use disorder, or PTSD who meet evidence-based psychotherapy and pharmacotherapy initiation and course measures divided by total number of patients with these diagnoses

    Initiation of an evidence-based practice is indicated by an evidence-based psychotherapy template or evidence-based pharmacotherapy prescription after intake. Adequate course is based on receiving an adequate number of evidence-based psychotherapy sessions to be a "completer" (typically 8 sessions) or enough refills for a guideline-recommended adequate trial of each medication (varies by medication). Data is gathered based on electronic health record data from the VA Corporate Data Warehouse (CDW).

    Time frame: Pre-/Post- 12-month period average of evidence-based practice reach (24 months total observation)

  2. Proportion of completed evidence-based practice templates during sessions with a relevant CPT code

    We will study 5 evidence-based psychotherapies: 3 for depression (Cognitive Behavior Therapy (CBT-D), Acceptance and Commitment Therapy (ACT), and Interpersonal Psychotherapy (IPT)) and 2 for PTSD (Prolonged Exposure (PE) and Cognitive Processing Therapy (CPT)). Data is gathered based on electronic health record data from the Corporate Data Warehouse (CDW).

    Time frame: Pre-/Post- 12-month period average of evidence-based practice reach (24 months total observation)

  3. Proportion of combination of prescriptions placed with the VA pharmacy and sessions with a relevant CPT code

    We will study 8 evidence-based pharmacotherapies: 2 for depression (84 and 180 days therapeutic continuity at new antidepressant start), 2 for Opioid Use Disorder (OUD) (methadone and buprenorphine), and 4 for Alcohol Use Disorder (AUD) (Acamprosate, Disulfiram, Naltrexone, and Topiramate). Data is gathered based on electronic health record data from the Corporate Data Warehouse (CDW).

    Time frame: Pre-/Post- 12-month period average of evidence-based practice reach (24 months total observation)

Secondary outcomes

  1. Degree of acceptability of intervention assessed by the Acceptability of Intervention Measure (AIM) [followed by its scale information in the Description]

    Assesses degree of differences in team perceptions of PSD and AF on a survey with 4 items. Scale: 1-5, in 1 point increments (1 = completely disagree, 2 = disagree, 3 = neither agree nor disagree, 4 = agree, 5 = completely agree)

    Time frame: At 6 months

  2. Degree of appropriateness of intervention assessed by the Intervention Appropriateness Measure (IAM) [followed by its scale information in the Description]

    Assesses degree of for differences in team perceptions of PSD and AF on a survey with 4 items. Scale: 1-5, in 1 point increments (1 = completely disagree, 2 = disagree, 3 = neither agree nor disagree, 4 = agree, 5 = completely agree)

    Time frame: At 6 months

  3. Degree of feasibility of intervention assessed by the Feasibility of Intervention Measure (FIM) [followed by its scale information in the Description]

    Assesses degree of differences in team perceptions of PSD and AF on a survey with 4 items. Scale: 1-5, in 1 point increments (1 = completely disagree, 2 = disagree, 3 = neither agree nor disagree, 4 = agree, 5 = completely agree)

    Time frame: At 6 months

  4. Patient Aligned Care team Burnout Measure (PACT) [followed by its scale information in the Description]

    Quality of work satisfaction and burnout in a 4-item descriptive survey with measures from VA team-based primary care that tracks 1) years of experience with the team, 2) working on more than one team, 3) turnover/change in team staff, 4) team overwork, and the single-item 5) self-reported burnout (sensitivity 83.2% and specificity 87.4%) (Question 1) Answered in # of years (Question 2-3) Yes or No (Question 4-5) Scale: 1-5, in 1 point increments (1 = Never, 2 = Rarely, 3 = Sometimes, 4 = Very Often, 5 = Always)

    Time frame: At baseline and 6 months

  5. Learning Organization Survey (LOS-27) [followed by its scale information in the Description]

    Psychological safety in the workplace using a 27-item survey developed out of the learning organization tradition and demonstrated to have good psychometric properties during VA validation which assesses 7 clinic context factors: a) supportive learning environment (including psychological safety), b) leadership that reinforces learning, c) experimentation, d) training, e) knowledge acquisition, f) time for reflection, and g) performance monitoring (Questions 1-23) Scale: 0-4, in 1 point increments (0 = Never, 4 = Always) (Questions 24-27) Scale: 0 to 7, in 1 point increments (0 = Highly Inaccurate, 7 = Highly Accurate)

    Time frame: At baseline and 6 months

  6. Team Decision Making Questionnaire (TDMQ) [followed by its scale information in the Description]

    Team dynamics in the workplace using a four factor scale survey validated to assess the impact of a team intervention on team decision-making, support learning and development of quality services Scale: 1-7, in 1 point increments incl N/A (N/A = Not Applicable, 1 = Not at all, 2 = To a very small extent, 3 = To a small extent, 4 = To a moderate extent, 5 = To a great extent, 6 = To a very great extent, 7 = To a vast extent)

    Time frame: At 6 months

  7. Systems Thinking Scale (STS) [followed by its scale information in the Description]

    Use of systems thinking in the work place and the ability to recognize, understand, and synthesize interactions and interdependencies, including how actions and components can reinforce or counteract each other. Scale: 1-5, in 1 point increments (1 = Never, 2 = Seldom, 3 = Some of the time, 4 = Often, 5 = Most of the time)

    Time frame: At baseline and 6 months

  8. Systems Thinking Codebook and Session Observations [followed by its scale information in the Description]

    Observation of systems thinking in language/explanations, and performance demonstrating system thinking skills (competence) measured on four constructs: Complex, Feedback, Behavior, Time Scale: Level 1-4 in 1 point increments (1 = Construct is demonstrated at most simple level, 4 = Construct is fully demonstrated at most complex level)

    Time frame: Over 6 months

  9. Facilitator Fidelity to Intervention Guides and Theory of Change

    Review fidelity with qualitative checks against AF/PSD facilitator scripts for session learning objectives, 'key idea' and 'definitions,' including tracking the proportion of AF/PSD session activities (in minutes) on these components.

    Time frame: Over 6 months

  10. Demographic Measures

    4 item measure assessing ethnic (Hispanic, Latino, Latina, or Latinx), racial (American Indian/Alaska Native, Asian, Native Hawaiian or Other Pacific Islander, Black or African American, White, More than One Race) and gender (Man, Woman, Non-binary) identity of respondent. All items include a "Prefer not to say" option. Categories for demographic measures determined based on NIH reporting standards.

    Time frame: At baseline and 6 months

07

Study locations

1 site
08

References and documents

Publications

  • Elbogen EB, Wagner HR, Johnson SC, Kinneer P, Kang H, Vasterling JJ, Timko C, Beckham JC. Are Iraq and Afghanistan veterans using mental health services? New data from a national random-sample survey. Psychiatr Serv. 2013 Feb 1;64(2):134-41. doi: 10.1176/appi.ps.004792011. PubMed 23475498 ↗
  • Hermes ED, Hoff R, Rosenheck RA. Sources of the increasing number of Vietnam era veterans with a diagnosis of PTSD using VHA services. Psychiatr Serv. 2014 Jun 1;65(6):830-2. doi: 10.1176/appi.ps.201300232. PubMed 24733559 ↗
  • Mott JM, Grubbs KM, Sansgiry S, Fortney JC, Cully JA. Psychotherapy Utilization Among Rural and Urban Veterans From 2007 to 2010. J Rural Health. 2015 Summer;31(3):235-43. doi: 10.1111/jrh.12099. Epub 2014 Dec 3. PubMed 25471067 ↗
  • Hankin CS, Spiro A 3rd, Miller DR, Kazis L. Mental disorders and mental health treatment among U.S. Department of Veterans Affairs outpatients: the Veterans Health Study. Am J Psychiatry. 1999 Dec;156(12):1924-30. doi: 10.1176/ajp.156.12.1924. PubMed 10588406 ↗
  • Hoggatt KJ, Williams EC, Der-Martirosian C, Yano EM, Washington DL. National prevalence and correlates of alcohol misuse in women veterans. J Subst Abuse Treat. 2015 May;52:10-6. doi: 10.1016/j.jsat.2014.12.003. Epub 2014 Dec 19. PubMed 25661517 ↗
  • Fulton JJ, Calhoun PS, Wagner HR, Schry AR, Hair LP, Feeling N, Elbogen E, Beckham JC. The prevalence of posttraumatic stress disorder in Operation Enduring Freedom/Operation Iraqi Freedom (OEF/OIF) Veterans: a meta-analysis. J Anxiety Disord. 2015 Apr;31:98-107. doi: 10.1016/j.janxdis.2015.02.003. Epub 2015 Feb 19. PubMed 25768399 ↗
  • Karlin BE, Cross G. From the laboratory to the therapy room: National dissemination and implementation of evidence-based psychotherapies in the U.S. Department of Veterans Affairs Health Care System. Am Psychol. 2014 Jan;69(1):19-33. doi: 10.1037/a0033888. Epub 2013 Sep 2. PubMed 24001035 ↗
  • Karlin BE, Brown GK, Trockel M, Cunning D, Zeiss AM, Taylor CB. National dissemination of cognitive behavioral therapy for depression in the Department of Veterans Affairs health care system: therapist and patient-level outcomes. J Consult Clin Psychol. 2012 Oct;80(5):707-18. doi: 10.1037/a0029328. Epub 2012 Jul 23. PubMed 22823859 ↗
  • Ruzek JI, Karlin BE, & Zeiss AM. Implementation of Evidence-Based Psychological Treatments in the Veterans Health Administration. In: McHugh RK, Barlow DH, eds. Dissemination of evidence-based psychological treatments. New York, NY: Oxford University Press. , 2012.
  • Eftekhari A, Ruzek JI, Crowley JJ, Rosen CS, Greenbaum MA, Karlin BE. Effectiveness of national implementation of prolonged exposure therapy in Veterans Affairs care. JAMA Psychiatry. 2013 Sep;70(9):949-55. doi: 10.1001/jamapsychiatry.2013.36. PubMed 23863892 ↗
  • Watts BV, Shiner B, Zubkoff L, Carpenter-Song E, Ronconi JM, Coldwell CM. Implementation of evidence-based psychotherapies for posttraumatic stress disorder in VA specialty clinics. Psychiatr Serv. 2014 May 1;65(5):648-53. doi: 10.1176/appi.ps.201300176. PubMed 24430622 ↗
  • Department of Defense, & Department of Veterans Affairs. The management of MDD Working Group. VA/DOD clinical practice guideline for management of major depressive disorder (MDD). , 2009.
  • Department of Veterans Affairs, & Department of Defense. VA/DoD Clinical practice guideline for the management of post-traumatic stress. , 2010.
  • Department of Veterans Affairs, & Department of Defense. VA/DoD Clinical practice guideline for the management of substance use disorders. , 2009.
  • Lemke S, Boden MT, Kearney LK, Krahn DD, Neuman MJ, Schmidt EM, Trafton JA. Measurement-based management of mental health quality and access in VHA: SAIL mental health domain. Psychol Serv. 2017 Feb;14(1):1-12. doi: 10.1037/ser0000097. PubMed 28134552 ↗
  • Harris AH, Humphreys K, Bowe T, Kivlahan DR, Finney JW. Measuring the quality of substance use disorder treatment: evaluating the validity of the Department of Veterans Affairs continuity of care performance measure. J Subst Abuse Treat. 2009 Apr;36(3):294-305. doi: 10.1016/j.jsat.2008.05.011. Epub 2008 Oct 5. PubMed 18835678 ↗
  • Rosen CS, Matthieu MM, Wiltsey Stirman S, Cook JM, Landes S, Bernardy NC, Chard KM, Crowley J, Eftekhari A, Finley EP, Hamblen JL, Harik JM, Kehle-Forbes SM, Meis LA, Osei-Bonsu PE, Rodriguez AL, Ruggiero KJ, Ruzek JI, Smith BN, Trent L, Watts BV. A Review of Studies on the System-Wide Implementation of Evidence-Based Psychotherapies for Posttraumatic Stress Disorder in the Veterans Health Administration. Adm Policy Ment Health. 2016 Nov;43(6):957-977. doi: 10.1007/s10488-016-0755-0. PubMed 27474040 ↗
  • Seal KH, Maguen S, Cohen B, Gima KS, Metzler TJ, Ren L, Bertenthal D, Marmar CR. VA mental health services utilization in Iraq and Afghanistan veterans in the first year of receiving new mental health diagnoses. J Trauma Stress. 2010 Feb;23(1):5-16. doi: 10.1002/jts.20493. PubMed 20146392 ↗
  • Mott JM, Mondragon S, Hundt NE, Beason-Smith M, Grady RH, Teng EJ. Characteristics of U.S. veterans who begin and complete prolonged exposure and cognitive processing therapy for PTSD. J Trauma Stress. 2014 Jun;27(3):265-73. doi: 10.1002/jts.21927. PubMed 24948535 ↗
  • Harpaz-Rotem I, Rosenheck RA. Serving those who served: retention of newly returning veterans from Iraq and Afghanistan in mental health treatment. Psychiatr Serv. 2011 Jan;62(1):22-7. doi: 10.1176/ps.62.1.pss6201_0022. PubMed 21209295 ↗
  • Lin LA, Bohnert AS, Ilgen MA, Pfeiffer PN, Ganoczy D, Blow FC. Outpatient Provider Contact Prior to Unintentional Opioid Overdose Among VHA Service Users. Psychiatr Serv. 2015 Nov;66(11):1149-54. doi: 10.1176/appi.ps.201400194. Epub 2015 Jul 1. PubMed 26129993 ↗
  • Harris AH, Bowe T, Del Re AC, Finlay AK, Oliva E, Myrick HL, Rubinsky AD. Extended release naltrexone for alcohol use disorders: quasi-experimental effects on mortality and subsequent detoxification episodes. Alcohol Clin Exp Res. 2015 Jan;39(1):79-83. doi: 10.1111/acer.12597. PubMed 25623408 ↗
  • Degenhardt L, Bucello C, Mathers B, Briegleb C, Ali H, Hickman M, McLaren J. Mortality among regular or dependent users of heroin and other opioids: a systematic review and meta-analysis of cohort studies. Addiction. 2011 Jan;106(1):32-51. doi: 10.1111/j.1360-0443.2010.03140.x. Epub 2010 Nov 4. PubMed 21054613 ↗
  • Kaplan MS, Huguet N, McFarland BH, Newsom JT. Suicide among male veterans: a prospective population-based study. J Epidemiol Community Health. 2007 Jul;61(7):619-24. doi: 10.1136/jech.2006.054346. PubMed 17568055 ↗
  • Desai RA, Dausey DJ, Rosenheck RA. Mental health service delivery and suicide risk: the role of individual patient and facility factors. Am J Psychiatry. 2005 Feb;162(2):311-8. doi: 10.1176/appi.ajp.162.2.311. PubMed 15677596 ↗
  • Gradus JL, Suvak MK, Wisco BE, Marx BP, Resick PA. Treatment of posttraumatic stress disorder reduces suicidal ideation. Depress Anxiety. 2013 Oct;30(10):1046-53. doi: 10.1002/da.22117. Epub 2013 May 1. PubMed 23636925 ↗
  • Hysong SJ. Meta-analysis: audit and feedback features impact effectiveness on care quality. Med Care. 2009 Mar;47(3):356-63. doi: 10.1097/MLR.0b013e3181893f6b. PubMed 19194332 ↗
  • Foy R, Eccles MP, Jamtvedt G, Young J, Grimshaw JM, Baker R. What do we know about how to do audit and feedback? Pitfalls in applying evidence from a systematic review. BMC Health Serv Res. 2005 Jul 13;5:50. doi: 10.1186/1472-6963-5-50. PubMed 16011811 ↗
  • Ivers N, Jamtvedt G, Flottorp S, Young JM, Odgaard-Jensen J, French SD, O'Brien MA, Johansen M, Grimshaw J, Oxman AD. Audit and feedback: effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev. 2012 Jun 13;2012(6):CD000259. doi: 10.1002/14651858.CD000259.pub3. PubMed 22696318 ↗
  • Flottorp SA, Jamtvedt G, Gibis B, et al. Using audit and feedback to health professionals to improve the quality and safety of health care. World Health Organ. 2010.
  • Ivers NM, Sales A, Colquhoun H, Michie S, Foy R, Francis JJ, Grimshaw JM. No more 'business as usual' with audit and feedback interventions: towards an agenda for a reinvigorated intervention. Implement Sci. 2014 Jan 17;9:14. doi: 10.1186/1748-5908-9-14. PubMed 24438584 ↗
  • Colquhoun HL, Carroll K, Eva KW, Grimshaw JM, Ivers N, Michie S, Sales A, Brehaut JC. Advancing the literature on designing audit and feedback interventions: identifying theory-informed hypotheses. Implement Sci. 2017 Sep 29;12(1):117. doi: 10.1186/s13012-017-0646-0. PubMed 28962632 ↗
  • Colquhoun H, Michie S, Sales A, Ivers N, Grimshaw JM, Carroll K, Chalifoux M, Eva K, Brehaut J. Reporting and design elements of audit and feedback interventions: a secondary review. BMJ Qual Saf. 2017 Jan;26(1):54-60. doi: 10.1136/bmjqs-2015-005004. Epub 2016 Jan 25. PubMed 26811541 ↗
  • Landis-Lewis Z, Brehaut JC, Hochheiser H, Douglas GP, Jacobson RS. Computer-supported feedback message tailoring: theory-informed adaptation of clinical audit and feedback for learning and behavior change. Implement Sci. 2015 Jan 21;10:12. doi: 10.1186/s13012-014-0203-z. PubMed 25603806 ↗
  • Atkinson JA, Wells R, Page A, Dominello A, Haines M, Wilson A. Applications of system dynamics modelling to support health policy. Public Health Res Pract. 2015 Jul 9;25(3):e2531531. doi: 10.17061/phrp2531531. PubMed 26243490 ↗
  • Diehl E, & Sterman JD. Effects of feedback complexity on dynamic decision making. Organ Behav Hum Decis Process. 1995;62:198-215.
  • Sterman JD. Modeling managerial behavior: Misperceptions of feedback in a dynamic decision making experiment. Manag Sci. 1989;35:321-339.
  • Cronin MA, Gonzalez C, & Sterman JD. Why don't well-educated adults understand accumulation? A challenge to researchers, educators, and citizens. Organ Behav Hum Decis Process. 2009;108:116-130.
  • Sterman JD. Learning from evidence in a complex world. Am J Public Health. 2006 Mar;96(3):505-14. doi: 10.2105/AJPH.2005.066043. Epub 2006 Jan 31. PubMed 16449579 ↗
  • Sterman JD. Learning in and about complex systems. Syst Dyn Rev. 1994;10:291-330.
  • Andersen DF, Vennix JA, Richardson GP, et al. Group model building: Problem structing, policy simulation and decision support. J Oper Res Soc. 2007;:691-694.
  • Vennix JAM. Group model building: facilitating team learning using system dynamics. Chichester ; New York: J. Wiley, 1996: 1-297.
  • Rouwette EAJA, Vennix JAM, & Mullekom T van. Group model building effectiveness: A review of assessment studies. Syst Dyn Rev. 2002;18:5-45.
  • Bendoly E. System dynamics understanding in projects: Information sharing, psychological safety, and performance effects. Prod Oper Manag. 2014;23:1352-1369.
  • Sterman JD. Does formal system dynamics training improve people's understanding of accumulation? Syst Dyn Rev. 2010;26:316-334.
  • Simon HA. Bounded rationality and organizational learning. Organ Sci. 1991;2:125-134.
  • Rahmandad H, Repenning N, & Sterman J. Effects of feedback delay on learning. Syst Dyn Rev. 2009;25:309-338.
  • Zimmerman L, Lounsbury DW, Rosen CS, Kimerling R, Trafton JA, Lindley SE. Participatory System Dynamics Modeling: Increasing Stakeholder Engagement and Precision to Improve Implementation Planning in Systems. Adm Policy Ment Health. 2016 Nov;43(6):834-849. doi: 10.1007/s10488-016-0754-1. PubMed 27480546 ↗
  • Oliva R. Model calibration as a testing strategy for system dynamics models. Eur J Oper Res. 2003;151:552-568.
  • Trafton JA, Greenberg G, Harris AH, Tavakoli S, Kearney L, McCarthy J, Blow F, Hoff R, Schohn M. VHA mental health information system: applying health information technology to monitor and facilitate implementation of VHA Uniform Mental Health Services Handbook requirements. Med Care. 2013 Mar;51(3 Suppl 1):S29-36. doi: 10.1097/MLR.0b013e31827da836. PubMed 23407008 ↗
  • Barlas Y. Formal aspects of model validity and validation in system dynamics. Syst Dyn Rev. 1996;12:183-210.
  • Hirsch G, Homer J, Trogdon J, Wile K, Orenstein D. Using simulation to compare 4 categories of intervention for reducing cardiovascular disease risks. Am J Public Health. 2014 Jul;104(7):1187-95. doi: 10.2105/AJPH.2013.301816. Epub 2014 May 15. PubMed 24832142 ↗
  • Lich KH, Tian Y, Beadles CA, Williams LS, Bravata DM, Cheng EM, Bosworth HB, Homer JB, Matchar DB. Strategic planning to reduce the burden of stroke among veterans: using simulation modeling to inform decision making. Stroke. 2014 Jul;45(7):2078-84. doi: 10.1161/STROKEAHA.114.004694. Epub 2014 Jun 12. PubMed 24923722 ↗
  • Forrester, J.W. The model versus a modeling process. Syst Dyn Rev. 1985;:133-134.
  • Derickson R, Fishman J, Osatuke K, Teclaw R, Ramsel D. Psychological safety and error reporting within Veterans Health Administration hospitals. J Patient Saf. 2015 Mar;11(1):60-6. doi: 10.1097/PTS.0000000000000082. PubMed 24583957 ↗
  • Edmondson A. Psychological safety and learning behavior in work teams. Adm Sci Q. 1999;44:350-383.
  • Campbell MJ, & Walters SJ. How to Design, Analyise and Report Cluster Randomised Trials in Medicine and Health Related Research. : Wiley, 2014: 1-264.
  • Forrester JW. System dynamics, systems thinking, and soft OR. Syst Dyn Rev. 1994;10:245-256.
  • Forrester JW. Some basic concepts in system dynamics. Sloan Sch Manag Mass Inst Technol Camb MA. 2009.
  • Hong G. Ratio of mediator probability weighting for estimating natural direct and indirect effects. In: Proceedings of the American Statistical Association, Biometrics Section. : American Statistical Association Alexandria, VA, 2010: 2401-2415.
  • Hong G, Deutsch J, & Hill HD. Ratio-of-Mediator-Probability Weighting for Causal Mediation Analysis in the Presence of Treatment-by-Mediator Interaction. J Educ Behav Stat. 2015;40:307-340.
  • Qin X, & Hong G. Causal mediation analysis in multi-site trials: An application of ratio-of-mediatorprobability weighting to the Head Start Impact Study. JSM Proc Soc Stat Sect. 2014;:912-926.
  • Homer JB. Partial-model testing as a validation tool for system dynamics (1983). Syst Dyn Rev. 2012;28:281-294.
  • Rahmandad H, & Sterman JD. Reporting guidelines for simulation-based research in social sciences. Syst Dyn Rev. 2012;28:396-411.
  • Singer SJ, Moore SC, Meterko M, Williams S. Development of a short-form Learning Organization Survey: the LOS-27. Med Care Res Rev. 2012 Aug;69(4):432-59. doi: 10.1177/1077558712448135. Epub 2012 May 25. PubMed 22635274 ↗
  • Batorowicz B, Shepherd TA. Measuring the quality of transdisciplinary teams. J Interprof Care. 2008 Dec;22(6):612-20. doi: 10.1080/13561820802303664. PubMed 19012141 ↗
  • Dolan ED, Mohr D, Lempa M, Joos S, Fihn SD, Nelson KM, Helfrich CD. Using a single item to measure burnout in primary care staff: a psychometric evaluation. J Gen Intern Med. 2015 May;30(5):582-7. doi: 10.1007/s11606-014-3112-6. Epub 2014 Dec 2. PubMed 25451989 ↗
  • Helfrich CD, Simonetti JA, Clinton WL, Wood GB, Taylor L, Schectman G, Stark R, Rubenstein LV, Fihn SD, Nelson KM. The Association of Team-Specific Workload and Staffing with Odds of Burnout Among VA Primary Care Team Members. J Gen Intern Med. 2017 Jul;32(7):760-766. doi: 10.1007/s11606-017-4011-4. Epub 2017 Feb 23. PubMed 28233221 ↗
  • Nelson KM, Helfrich C, Sun H, Hebert PL, Liu CF, Dolan E, Taylor L, Wong E, Maynard C, Hernandez SE, Sanders W, Randall I, Curtis I, Schectman G, Stark R, Fihn SD. Implementation of the patient-centered medical home in the Veterans Health Administration: associations with patient satisfaction, quality of care, staff burnout, and hospital and emergency department use. JAMA Intern Med. 2014 Aug;174(8):1350-8. doi: 10.1001/jamainternmed.2014.2488. PubMed 25055197 ↗
  • Helfrich CD, Dolan ED, Simonetti J, Reid RJ, Joos S, Wakefield BJ, Schectman G, Stark R, Fihn SD, Harvey HB, Nelson K. Elements of team-based care in a patient-centered medical home are associated with lower burnout among VA primary care employees. J Gen Intern Med. 2014 Jul;29 Suppl 2(Suppl 2):S659-66. doi: 10.1007/s11606-013-2702-z. PubMed 24715396 ↗
  • Weiner BJ, Lewis CC, Stanick C, Powell BJ, Dorsey CN, Clary AS, Boynton MH, Halko H. Psychometric assessment of three newly developed implementation outcome measures. Implement Sci. 2017 Aug 29;12(1):108. doi: 10.1186/s13012-017-0635-3. PubMed 28851459 ↗
  • Sweeney LB, & Sterman JD. Bathtub dynamics: Initial results of a systems thinking inventory. Syst Dyn Rev. 2000;16:249-286.
  • Liberati EG, Ruggiero F, Galuppo L, Gorli M, Gonzalez-Lorenzo M, Maraldi M, Ruggieri P, Polo Friz H, Scaratti G, Kwag KH, Vespignani R, Moja L. What hinders the uptake of computerized decision support systems in hospitals? A qualitative study and framework for implementation. Implement Sci. 2017 Sep 15;12(1):113. doi: 10.1186/s13012-017-0644-2. PubMed 28915822 ↗
  • Centers for Medicare & Medicaid Services Alliance to Modernize Healthcare (CAMH). Independent Assessment of the Health Care Delivery Systems and Management Processes of the Department of Veterans Affairs (Volume 1: Integrated Report).
  • Department of Veterans Affairs. Uniform mental health services in VA medical centers and clinics. Washington DC: Veterans Health Administration, 2008.
  • Powers MB, Halpern JM, Ferenschak MP, Gillihan SJ, Foa EB. A meta-analytic review of prolonged exposure for posttraumatic stress disorder. Clin Psychol Rev. 2010 Aug;30(6):635-41. doi: 10.1016/j.cpr.2010.04.007. Epub 2010 May 2. PubMed 20546985 ↗
  • Bradley R, Greene J, Russ E, Dutra L, Westen D. A multidimensional meta-analysis of psychotherapy for PTSD. Am J Psychiatry. 2005 Feb;162(2):214-27. doi: 10.1176/appi.ajp.162.2.214. PubMed 15677582 ↗
  • Powers MB, Zum Vorde Sive Vording MB, Emmelkamp PM. Acceptance and commitment therapy: a meta-analytic review. Psychother Psychosom. 2009;78(2):73-80. doi: 10.1159/000190790. Epub 2009 Jan 14. PubMed 19142046 ↗
  • Walser RD, Karlin BE, Trockel M, Mazina B, Barr Taylor C. Training in and implementation of Acceptance and Commitment Therapy for depression in the Veterans Health Administration: therapist and patient outcomes. Behav Res Ther. 2013 Sep;51(9):555-63. doi: 10.1016/j.brat.2013.05.009. Epub 2013 Jun 3. PubMed 23851161 ↗
  • Stewart MO, Raffa SD, Steele JL, Miller SA, Clougherty KF, Hinrichsen GA, Karlin BE. National dissemination of interpersonal psychotherapy for depression in veterans: therapist and patient-level outcomes. J Consult Clin Psychol. 2014 Dec;82(6):1201-6. doi: 10.1037/a0037410. Epub 2014 Jul 21. PubMed 25045906 ↗
  • Cuijpers P, Geraedts AS, van Oppen P, Andersson G, Markowitz JC, van Straten A. Interpersonal psychotherapy for depression: a meta-analysis. Am J Psychiatry. 2011 Jun;168(6):581-92. doi: 10.1176/appi.ajp.2010.10101411. Epub 2011 Mar 1. PubMed 21362740 ↗
  • Fournier JC, DeRubeis RJ, Hollon SD, Dimidjian S, Amsterdam JD, Shelton RC, Fawcett J. Antidepressant drug effects and depression severity: a patient-level meta-analysis. JAMA. 2010 Jan 6;303(1):47-53. doi: 10.1001/jama.2009.1943. PubMed 20051569 ↗
  • Knudsen HK, Roman PM. The Diffusion of Acamprosate for the Treatment of Alcohol Use Disorder: Results From a National Longitudinal Study. J Subst Abuse Treat. 2016 Mar;62:62-7. doi: 10.1016/j.jsat.2015.10.005. Epub 2015 Dec 1. PubMed 26689318 ↗
  • Knudsen HK, Roman PM. Service delivery and pharmacotherapy for alcohol use disorder in the era of health reform: Data from a national sample of treatment organizations. Subst Abus. 2016;37(1):230-7. doi: 10.1080/08897077.2015.1028699. Epub 2015 Apr 20. PubMed 25893539 ↗
  • Pettinati HM, O'Brien CP, Rabinowitz AR, Wortman SP, Oslin DW, Kampman KM, Dackis CA. The status of naltrexone in the treatment of alcohol dependence: specific effects on heavy drinking. J Clin Psychopharmacol. 2006 Dec;26(6):610-25. doi: 10.1097/01.jcp.0000245566.52401.20. PubMed 17110818 ↗
  • Anton RF, O'Malley SS, Ciraulo DA, Cisler RA, Couper D, Donovan DM, Gastfriend DR, Hosking JD, Johnson BA, LoCastro JS, Longabaugh R, Mason BJ, Mattson ME, Miller WR, Pettinati HM, Randall CL, Swift R, Weiss RD, Williams LD, Zweben A; COMBINE Study Research Group. Combined pharmacotherapies and behavioral interventions for alcohol dependence: the COMBINE study: a randomized controlled trial. JAMA. 2006 May 3;295(17):2003-17. doi: 10.1001/jama.295.17.2003. PubMed 16670409 ↗
  • Barnett PG, Rodgers JH, Bloch DA. A meta-analysis comparing buprenorphine to methadone for treatment of opiate dependence. Addiction. 2001 May;96(5):683-90. doi: 10.1046/j.1360-0443.2001.9656834.x. PubMed 11331027 ↗
  • West SL, O'Neal KK, Graham CW. A meta-analysis comparing the effectiveness of buprenorphine and methadone. J Subst Abuse. 2000;12(4):405-14. doi: 10.1016/s0899-3289(01)00054-2. PubMed 11452842 ↗
  • Hysong SJ, Kell HJ, Petersen LA, Campbell BA, Trautner BW. Theory-based and evidence-based design of audit and feedback programmes: examples from two clinical intervention studies. BMJ Qual Saf. 2017 Apr;26(4):323-334. doi: 10.1136/bmjqs-2015-004796. Epub 2016 Jun 10. PubMed 27288054 ↗
  • Hysong SJ, Best RG, Pugh JA. Audit and feedback and clinical practice guideline adherence: making feedback actionable. Implement Sci. 2006 Apr 28;1:9. doi: 10.1186/1748-5908-1-9. PubMed 16722539 ↗
  • Hysong SJ, Teal CR, Khan MJ, Haidet P. Improving quality of care through improved audit and feedback. Implement Sci. 2012 May 18;7:45. doi: 10.1186/1748-5908-7-45. PubMed 22607640 ↗
  • Shiner B, D'Avolio LW, Nguyen TM, Zayed MH, Young-Xu Y, Desai RA, Schnurr PP, Fiore LD, Watts BV. Measuring use of evidence based psychotherapy for posttraumatic stress disorder. Adm Policy Ment Health. 2013 Jul;40(4):311-8. doi: 10.1007/s10488-012-0421-0. PubMed 22535469 ↗
  • Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health. 1999 Sep;89(9):1322-7. doi: 10.2105/ajph.89.9.1322. PubMed 10474547 ↗
  • Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, Griffey R, Hensley M. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health. 2011 Mar;38(2):65-76. doi: 10.1007/s10488-010-0319-7. PubMed 20957426 ↗
  • Brownson RC, Colditz GA, & Proctor EK. Dissemination and Implementation Research in Health: Translating Science to Practice. Oxford, UK: Oxford, 2012.
  • Tabak RG, Khoong EC, Chambers DA, Brownson RC. Bridging research and practice: models for dissemination and implementation research. Am J Prev Med. 2012 Sep;43(3):337-50. doi: 10.1016/j.amepre.2012.05.024. PubMed 22898128 ↗
  • Tabak RG, Khoong EC, Chambers D, et al. Models in dissemination and implementation research: useful tools in public health services and systems research. Front Public Health Serv Syst Res. 2013;2:8.
  • Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009 Aug 7;4:50. doi: 10.1186/1748-5908-4-50. PubMed 19664226 ↗
  • Powell BJ, Waltz TJ, Chinman MJ, Damschroder LJ, Smith JL, Matthieu MM, Proctor EK, Kirchner JE. A refined compilation of implementation strategies: results from the Expert Recommendations for Implementing Change (ERIC) project. Implement Sci. 2015 Feb 12;10:21. doi: 10.1186/s13012-015-0209-1. PubMed 25889199 ↗
  • Chambers DA, Glasgow RE, Stange KC. The dynamic sustainability framework: addressing the paradox of sustainment amid ongoing change. Implement Sci. 2013 Oct 2;8:117. doi: 10.1186/1748-5908-8-117. PubMed 24088228 ↗
  • Breuer E, Lee L, De Silva M, Lund C. Using theory of change to design and evaluate public health interventions: a systematic review. Implement Sci. 2016 May 6;11:63. doi: 10.1186/s13012-016-0422-6. PubMed 27153985 ↗
  • Schierhout G, Hains J, Si D, Kennedy C, Cox R, Kwedza R, O'Donoghue L, Fittock M, Brands J, Lonergan K, Dowden M, Bailie R. Evaluating the effectiveness of a multifaceted, multilevel continuous quality improvement program in primary health care: developing a realist theory of change. Implement Sci. 2013 Oct 8;8:119. doi: 10.1186/1748-5908-8-119. PubMed 24098940 ↗
  • Nilsen P. Making sense of implementation theories, models and frameworks. Implement Sci. 2015 Apr 21;10:53. doi: 10.1186/s13012-015-0242-0. PubMed 25895742 ↗
  • Sales A, Smith J, Curran G, Kochevar L. Models, strategies, and tools. Theory in implementing evidence-based findings into health care practice. J Gen Intern Med. 2006 Feb;21 Suppl 2(Suppl 2):S43-9. doi: 10.1111/j.1525-1497.2006.00362.x. PubMed 16637960 ↗
  • Damschroder LJ, & Lowery JC. Efficient Synthesis: Using Qualitative Comparative Analysis (QCA) and the CFIR across Diverse Studies. 2015.
  • Baumgartner M, & Thiem A. Often Trusted but Never (Properly) Tested: Evaluating Qualitative Comparative Analysis. Sociol Methods Res. 2017;:4912411770148.
  • Hug S. Qualitative Comparative Analysis: How Inductive Use and Measurement Error Lead to Problematic Inference. Polit Anal. 2013;21:252-265.
  • Krogslund C, Choi DD, & Poertner M. Fuzzy Sets on Shaky Ground: Parameter Sensitivity and Confirmation Bias in fsQCA. Polit Anal. 2015;23:21-41.
  • Braumoeller B. QCAfalsePositive: Tests for Type I Error in Qualitative Comparative Analysis (QCA). 2015.
  • Powell BJ, Proctor EK, Glass JE. A Systematic Review of Strategies for Implementing Empirically Supported Mental Health Interventions. Res Soc Work Pract. 2014 Mar 1;24(2):192-212. doi: 10.1177/1049731513505778. PubMed 24791131 ↗
  • Williams NJ. Multilevel Mechanisms of Implementation Strategies in Mental Health: Integrating Theory, Research, and Practice. Adm Policy Ment Health. 2016 Sep;43(5):783-798. doi: 10.1007/s10488-015-0693-2. PubMed 26474761 ↗
  • Senge P. The Fifth Discipline. : Currency/Doubleday, 1990.
  • Sweeney LB, & Sterman JD. Bathtub dynamics: initial results of a systems thinking inventory. Syst Dyn Rev. 2000;16:249-286.
  • Hysong SJ, Knox MK, Haidet P. Examining clinical performance feedback in Patient-Aligned Care Teams. J Gen Intern Med. 2014 Jul;29 Suppl 2(Suppl 2):S667-74. doi: 10.1007/s11606-013-2707-7. PubMed 24715398 ↗
  • Hysong SJ, Thomas CL, Spitzmuller C, Amspoker AB, Woodard L, Modi V, Naik AD. Linking clinician interaction and coordination to clinical performance in Patient-Aligned Care Teams. Implement Sci. 2016 Jan 15;11:7. doi: 10.1186/s13012-015-0368-0. PubMed 26772972 ↗
  • Richardson GP. Concept models in group model building: G. P. Richardson: Concept Models in Group Model Building. Syst Dyn Rev. 2013;29:42-55.
  • Richardson GP. Reflections on the foundations of system dynamics: Foundations of System Dynamics. Syst Dyn Rev. 2011;27:219-243.
  • Garcia HA, McGeary CA, Finley EP, et al. Burnout among psychiatrists in the Veterans Health Administration. Burn Res. 2015;2:108-114.
  • Garcia HA, Finley EP, Ketchum N, Jakupcak M, Dassori A, Reyes SC. A survey of perceived barriers and attitudes toward mental health care among OEF/OIF veterans at VA outpatient mental health clinics. Mil Med. 2014 Mar;179(3):273-8. doi: 10.7205/MILMED-D-13-00076. PubMed 24594461 ↗
  • Garcia HA, Kelley LP, Rentz TO, et al. Pretreatment predictors of dropout from cognitive behavioral therapy for PTSD in Iraq and Afghanistan war veterans. Psychol Serv. 2011;8:1-11.
  • Senge, P.M. The fifth discipline: The art and practice of the learning organization. : Broadway Business, 2006.
  • Meadows DH. Thinking in systems: A primer. : Chelsea Green Publishing, 2012.
  • Senge PM, ed. The dance of change: the challenges of sustaining momentum in learning organizations, 1st ed. New York: Currency/Doubleday, 1999: 1-596.
  • Sterman JD. Business Dynamics: Systems Thinking and Modeling for a Complex World. : McGraw-Hill Education, 2000: 1-1008.
  • Rouwette EA, & Vennix JA. System dynamics and organizational interventions. Syst Res Behav Sci. 2006;23:451-466.
  • Akhnif E, Macq J, Idrissi Fakhreddine MO, Meessen B. Scoping literature review on the Learning Organisation concept as applied to the health system. Health Res Policy Syst. 2017 Mar 1;15(1):16. doi: 10.1186/s12961-017-0176-x. PubMed 28249608 ↗
  • Morecroft J, & Sherman J. Modeling for learning organizations. Portland OR: Productivity Press, 1994.
  • Argyris C, & Schön DA. Organizational learning: A theory of action perspective. Reis. 1997;:345-348.
  • Forrester JW. Industrial Dynamics. Cambridge, MA: MIT Press, 1961.
  • Simon HA. Models of Man: Social and Rational-Mathematical Essays on Rational Human Behavior in a Social Setting. Oxford, UK: Wiley, 1957.
  • Huz S, Andersen DF, Richardson GP, et al. A framework for evaluating systems thinking interventions: an experimental approach to mental health system change. Syst Dyn Rev. 1997;13:149-169.
  • Evans JS. Dual-processing accounts of reasoning, judgment, and social cognition. Annu Rev Psychol. 2008;59:255-78. doi: 10.1146/annurev.psych.59.103006.093629. PubMed 18154502 ↗
  • Evans JS. In two minds: dual-process accounts of reasoning. Trends Cogn Sci. 2003 Oct;7(10):454-9. doi: 10.1016/j.tics.2003.08.012. PubMed 14550493 ↗
  • Evans JS, Stanovich KE. Dual-Process Theories of Higher Cognition: Advancing the Debate. Perspect Psychol Sci. 2013 May;8(3):223-41. doi: 10.1177/1745691612460685. PubMed 26172965 ↗
  • Dolansky MA, Moore SM, Palmieri PA, Singh MK. Development and Validation of the Systems Thinking Scale. J Gen Intern Med. 2020 Aug;35(8):2314-2320. doi: 10.1007/s11606-020-05830-1. Epub 2020 Apr 27. PubMed 32342481 ↗
  • Hovmand PS. Community Based System Dynamics. New York, NY: Springer New York, 2014.
  • Minkler M, & Wallerstein N. Community-Based Participatory Research for Health: From Process to Outcomes. : John Wiley & Sons, 2011: 1-758.
  • Case AD, Byrd R, Claggett E, DeVeaux S, Perkins R, Huang C, Sernyak MJ, Steiner JL, Cole R, LaPaglia DM, Bailey M, Buchanan C, Johnson A, Kaufman JS. Stakeholders' perspectives on community-based participatory research to enhance mental health services. Am J Community Psychol. 2014 Dec;54(3-4):397-408. doi: 10.1007/s10464-014-9677-8. PubMed 25245601 ↗
  • Israel BA, Coombe CM, Cheezum RR, Schulz AJ, McGranaghan RJ, Lichtenstein R, Reyes AG, Clement J, Burris A. Community-based participatory research: a capacity-building approach for policy advocacy aimed at eliminating health disparities. Am J Public Health. 2010 Nov;100(11):2094-102. doi: 10.2105/AJPH.2009.170506. Epub 2010 Sep 23. PubMed 20864728 ↗
  • Wallerstein NB, Duran B. Using community-based participatory research to address health disparities. Health Promot Pract. 2006 Jul;7(3):312-23. doi: 10.1177/1524839906289376. Epub 2006 Jun 7. PubMed 16760238 ↗
  • Aarons GA. Measuring provider attitudes toward evidence-based practice: consideration of organizational context and individual differences. Child Adolesc Psychiatr Clin N Am. 2005 Apr;14(2):255-71, viii. doi: 10.1016/j.chc.2004.04.008. PubMed 15694785 ↗
  • Aarons GA, Hurlburt M, Horwitz SM. Advancing a conceptual model of evidence-based practice implementation in public service sectors. Adm Policy Ment Health. 2011 Jan;38(1):4-23. doi: 10.1007/s10488-010-0327-7. PubMed 21197565 ↗
  • Aarons GA, Ehrhart MG, Farahnak LR, Sklar M. Aligning leadership across systems and organizations to develop a strategic climate for evidence-based practice implementation. Annu Rev Public Health. 2014;35:255-74. doi: 10.1146/annurev-publhealth-032013-182447. PubMed 24641560 ↗
  • Aarons GA, Green AE, Trott E, Willging CE, Torres EM, Ehrhart MG, Roesch SC. The Roles of System and Organizational Leadership in System-Wide Evidence-Based Intervention Sustainment: A Mixed-Method Study. Adm Policy Ment Health. 2016 Nov;43(6):991-1008. doi: 10.1007/s10488-016-0751-4. PubMed 27439504 ↗
  • Brimhall KC, Fenwick K, Farahnak LR, Hurlburt MS, Roesch SC, Aarons GA. Leadership, Organizational Climate, and Perceived Burden of Evidence-Based Practice in Mental Health Services. Adm Policy Ment Health. 2016 Sep;43(5):629-639. doi: 10.1007/s10488-015-0670-9. PubMed 26152770 ↗
  • Senge P, & Forrester JW. Tests for building confidence in system dynamics models. 1980;14:209-228.
  • Rahmandad H, & Sterman JD. Reporting guidelines for simulation-based research in social sciences: Reporting Guidelines for Simulation-Based Research. Syst Dyn Rev. 2012;28:396-411.
  • Hovmand PS, Andersen DF, Rouwette E, et al. Group model-building "scripts" as a collaborative planning tool: Scripts as a collaborative planning tool. Syst Res Behav Sci. 2012;29:179-193.
  • Andersen D, & Richardson G. Scripts for group model building. Syst Dyn Rev. 1997;13:107-129.
  • Hovmand, Peter, S., et al. Scriptapedia 4.0.6. 2013.
  • McCarthy JT, Hocum CL, Albright RC, Rogers J, Gallaher EJ, Steensma DP, Gudgell SF, Bergstralh EJ, Dillon JC, Hickson LJ, Williams AW, Dingli D. Biomedical system dynamics to improve anemia control with darbepoetin alfa in long-term hemodialysis patients. Mayo Clin Proc. 2014 Jan;89(1):87-94. doi: 10.1016/j.mayocp.2013.10.022. PubMed 24388026 ↗
  • Gallaher E, Steensma DP, Chrisope TR, et al. Individualized Medicine and Biophysical System Dynamics: An Example from Clinical Practice in End Stage Renal Disease. 2011.
  • Homer J. Levels of evidence in system dynamics modeling. Syst Dyn Rev. 2014;30:75-80.
  • Oliva R, & Sterman JD. Death spirals and virtuous cycles. In: Handbook of Service Science. : Springer, 2010: 321-358.
  • Snow G. Package "blockrand." 2015.
  • Holowka DW, Marx BP, Gates MA, Litman HJ, Ranganathan G, Rosen RC, Keane TM. PTSD diagnostic validity in Veterans Affairs electronic records of Iraq and Afghanistan veterans. J Consult Clin Psychol. 2014 Aug;82(4):569-579. doi: 10.1037/a0036347. Epub 2014 Apr 14. PubMed 24731235 ↗
  • Emmons KM, Weiner B, Fernandez ME, Tu SP. Systems antecedents for dissemination and implementation: a review and analysis of measures. Health Educ Behav. 2012 Feb;39(1):87-105. doi: 10.1177/1090198111409748. Epub 2011 Jul 1. PubMed 21724933 ↗
  • Rabin BA, Lewis CC, Norton WE, Neta G, Chambers D, Tobin JN, Brownson RC, Glasgow RE. Measurement resources for dissemination and implementation research in health. Implement Sci. 2016 Mar 22;11:42. doi: 10.1186/s13012-016-0401-y. PubMed 27000147 ↗
  • Lewis CC, Fischer S, Weiner BJ, Stanick C, Kim M, Martinez RG. Outcomes for implementation science: an enhanced systematic review of instruments using evidence-based rating criteria. Implement Sci. 2015 Nov 4;10:155. doi: 10.1186/s13012-015-0342-x. PubMed 26537706 ↗
  • Glasgow RE. What does it mean to be pragmatic? Pragmatic methods, measures, and models to facilitate research translation. Health Educ Behav. 2013 Jun;40(3):257-65. doi: 10.1177/1090198113486805. PubMed 23709579 ↗
  • Glasgow RE, Riley WT. Pragmatic measures: what they are and why we need them. Am J Prev Med. 2013 Aug;45(2):237-43. doi: 10.1016/j.amepre.2013.03.010. PubMed 23867032 ↗
  • Singh MK, Ogrinc G, Cox KR, Dolansky M, Brandt J, Morrison LJ, Harwood B, Petroski G, West A, Headrick LA. The Quality Improvement Knowledge Application Tool Revised (QIKAT-R). Acad Med. 2014 Oct;89(10):1386-91. doi: 10.1097/ACM.0000000000000456. PubMed 25119555 ↗
  • Rosseel Y, Oberski D, Byrnes J, et al. Package "lavaan" Version 0.5-23.1097. 2017.
  • Shavelson RJ, & Webb NM. Generalizability theory: A primer. Newbury Park, CA: Sage., 1991.
  • Millsap RE, Kwok OM. Evaluating the impact of partial factorial invariance on selection in two populations. Psychol Methods. 2004 Mar;9(1):93-115. doi: 10.1037/1082-989X.9.1.93. PubMed 15053721 ↗
  • Wolak M. Package "ICC" Version 2.3.0. 2015.
  • Bates D, Maechler M, Bolker B, et al. Package "lme4" Version 1.1-13. 2017.
  • Rotondi MA. Package "CRTSize" Version 1.0. 2015.
  • Rotondi MA, & Donner A. Sample Size Estimation in Cluster Randomized Educational Trials: An Empirical Bayes Approach. J Educ Behav Stat. 2009;34:229-237.
  • Ventana Systems Inc. Vensim@ Version 6.3. 2014.
  • Hyndman, R.J., & Khandakar, Y. Automatic time series forecasting: The forecast package for R. J Stat Softw. 2008;26.
  • Hyndman R, O'Hara-Wild Mi, Bergmeir C, et al. Package "forecast" Version 8.0. 2017.
  • Mott JM, Hundt NE, Sansgiry S, Mignogna J, Cully JA. Changes in psychotherapy utilization among veterans with depression, anxiety, and PTSD. Psychiatr Serv. 2014 Jan 1;65(1):106-12. doi: 10.1176/appi.ps.201300056. PubMed 24081213 ↗
  • Oliva EM, Trafton JA, Harris AH, Gordon AJ. Trends in opioid agonist therapy in the Veterans Health Administration: is supply keeping up with demand? Am J Drug Alcohol Abuse. 2013 Mar;39(2):103-7. doi: 10.3109/00952990.2012.741167. PubMed 23421571 ↗
  • Diaz M, Neuhauser D. Pasteur and parachutes: when statistical process control is better than a randomized controlled trial. Qual Saf Health Care. 2005 Apr;14(2):140-3. doi: 10.1136/qshc.2005.013763. No abstract available. PubMed 15805461 ↗
  • Duclos A, Voirin N. The p-control chart: a tool for care improvement. Int J Qual Health Care. 2010 Oct;22(5):402-7. doi: 10.1093/intqhc/mzq037. Epub 2010 Jul 30. PubMed 20675711 ↗
  • Ogrinc GS, & Headrick L. Fundamentals of health care improvement: A guide to improving your patients' care. : Joint Commission Resources, 2008.
  • Ogrinc G, Mooney SE, Estrada C, Foster T, Goldmann D, Hall LW, Huizinga MM, Liu SK, Mills P, Neily J, Nelson W, Pronovost PJ, Provost L, Rubenstein LV, Speroff T, Splaine M, Thomson R, Tomolo AM, Watts B. The SQUIRE (Standards for QUality Improvement Reporting Excellence) guidelines for quality improvement reporting: explanation and elaboration. Qual Saf Health Care. 2008 Oct;17 Suppl 1(Suppl_1):i13-32. doi: 10.1136/qshc.2008.029058. PubMed 18836062 ↗
  • Buntin MB, Burke MF, Hoaglin MC, Blumenthal D. The benefits of health information technology: a review of the recent literature shows predominantly positive results. Health Aff (Millwood). 2011 Mar;30(3):464-71. doi: 10.1377/hlthaff.2011.0178. PubMed 21383365 ↗
  • Raffa SD, Maciejewski ML, Zimmerman LE, Damschroder LJ, Estabrooks PA, Ackermann RT, Tsai AG, Histon T, Goldstein MG. A System-Level Approach to Overweight and Obesity in the Veterans Health Administration. J Gen Intern Med. 2017 Apr;32(Suppl 1):79-82. doi: 10.1007/s11606-016-3948-z. PubMed 28271428 ↗
  • Kilbourne AM, Elwy AR, Sales AE, Atkins D. Accelerating Research Impact in a Learning Health Care System: VA's Quality Enhancement Research Initiative in the Choice Act Era. Med Care. 2017 Jul;55 Suppl 7 Suppl 1(7 Suppl 1):S4-S12. doi: 10.1097/MLR.0000000000000683. PubMed 27997456 ↗
  • Mabry PL, Olster DH, Morgan GD, Abrams DB. Interdisciplinarity and systems science to improve population health: a view from the NIH Office of Behavioral and Social Sciences Research. Am J Prev Med. 2008 Aug;35(2 Suppl):S211-24. doi: 10.1016/j.amepre.2008.05.018. PubMed 18619402 ↗
  • Mabry PL, Milstein B, Abraido-Lanza AF, Livingood WC, Allegrante JP. Opening a window on systems science research in health promotion and public health. Health Educ Behav. 2013 Oct;40(1 Suppl):5S-8S. doi: 10.1177/1090198113503343. No abstract available. PubMed 24084401 ↗
  • Luke DA, Stamatakis KA. Systems science methods in public health: dynamics, networks, and agents. Annu Rev Public Health. 2012 Apr;33:357-76. doi: 10.1146/annurev-publhealth-031210-101222. Epub 2012 Jan 3. PubMed 22224885 ↗
  • Moore GF, Evans RE. What theory, for whom and in which context? Reflections on the application of theory in the development and evaluation of complex population health interventions. SSM Popul Health. 2017 Dec;3:132-135. doi: 10.1016/j.ssmph.2016.12.005. PubMed 29302610 ↗
  • Monks T. Operational research as implementation science: definitions, challenges and research priorities. Implement Sci. 2016 Jun 6;11(1):81. doi: 10.1186/s13012-016-0444-0. PubMed 27268021 ↗
  • May CR, Johnson M, Finch T. Implementation, context and complexity. Implement Sci. 2016 Oct 19;11(1):141. doi: 10.1186/s13012-016-0506-3. PubMed 27756414 ↗
  • Brainard J, Hunter PR. Do complexity-informed health interventions work? A scoping review. Implement Sci. 2016 Sep 20;11(1):127. doi: 10.1186/s13012-016-0492-5. PubMed 27647152 ↗

Study documents

PDFs hosted by the registry, ClinicalTrials.gov.

Individual participant data

Plan to share: No

09

Updates

Tracking since Sep 25, 2026
No changes since tracking began. The registry record was last updated on May 4, 2026, before this site started recording changes on Sep 25, 2026. Its history is on ClinicalTrials.gov ↗
10

Registry details

Key details

Study ID
NCT04356274
Lead sponsor
Palo Alto Veterans Institute for Research
Responsible party
Lindsey Zimmerman (Principal Investigator, Palo Alto Veterans Institute for Research) — Principal investigator
First posted
Apr 22, 2020
Start date
Feb 1, 2019
Primary completion
Jul 31, 2026 (estimated)
Completion
Jan 29, 2027 (estimated)
Last update
May 4, 2026

Study contacts

Lindsey E Zimmerman, PhD
principal investigator · National Center for PTSD, Dissemination & Training Division

Oversight

Data monitoring committee
Yes
FDA-regulated drug
No
FDA-regulated device
No
View the source record on ClinicalTrials.gov ↗

Interested in this study?

Eligibility is decided by the study team. Share this record with your doctor or contact the team directly.

No contact was published for this record. The registry link below has the sponsor’s details.

Follow this study

Get an email when the registry record changes — status, dates, results — or when someone posts here.

Sign in to follow

Discussion

Questions and observations about this study, from anyone following it. Not medical advice, and not a channel to the study team — their contact details are on the registry record.

Sign in to join the discussion. Reading takes no account; posting does. You choose a display name, and a pseudonym is the default.

Nothing here yet. If you are running this trial, taking part in it, or weighing whether to, this is the place to say so.

Start the discussion