Research Article Innovative Program Evaluation: Factors Associated with Primary Care Project Completion to Advance Pharmacist Clinical Services Integration
by Marie Smith1*, Brenda Shipley2, Diana M Sobieraj3, Edward Miech4, Mary Mulrooney5
1Henry A. Palmer Endowed Professor of Community Pharmacy Practice and Assistant Dean for Practice & Public Policy Partnerships,
University of Connecticut, School of Pharmacy Storrs, CT, 69 N Eagleville Rd Unit 3092, Storrs, CT 06269, USA
2Research Assistant, University of Connecticut, School of Pharmacy Storrs, CT, USA
3Associate Professor, University of Connecticut, School of Pharmacy Storrs, CT, USA
4Research Scientist, Regenstrief Institute, Center for Health Services Research, Indianapolis, IN, USA
5Former Research Assistant Professor, University of Connecticut, School of Pharmacy, Storrs, CT, USA
*Corresponding author: Marie Smith, Henry A. Palmer Endowed Professor of Community Pharmacy Practice and Assistant Dean for Practice & Public Policy Partnerships, University of Connecticut, School of Pharmacy Storrs, CT, 69 N Eagleville Rd Unit 3092, Storrs, CT 06269, USA
Received Date: 04 September, 2024
Accepted Date: 11 September, 2024
Published Date: 13 September, 2024
Citation: Smith M, Shipley B, Sobieraj DM, Miech E, Mulrooney M (2024) Innovative Program Evaluation: Factors Associated with Primary Care Project Completion to Advance Pharmacist Clinical Services Integration. J Family Med Prim Care Open Acc 8: 269. https://doi.org/10.29011/2688-7460.100269
Abstract
Few Primary Care Organizations (PCO) have access to expertise and resources that can optimize clinical pharmacist integration, identify and select high-value clinical pharmacist services, and evaluate the impact/value (of the clinical pharmacist in the primary care setting. Beyond the presence of having a pharmacist on staff, little is known about the implementation factors associated with PCO project completion to advance pharmacist clinical services integration in primary care teams. Our objective was to assess the plausible implementation factors at the organizational, operational, and pharmacist level to identify difference-makers that contributed to the completion or incompletion of pharmacist projects. The study was designed as a secondary analysis and observational study to assess nine projects that were conducted to advance pharmacist clinical services in 4 PCOs. Coincidence analysis was applied to identify factors related to the completion of pharmacist clinical service projects. Six of the nine projects were completed. The modelling phase identified two pathways to project completion – (1) no involvement outside of the core team (i.e., medical and pharmacy leaders), or (2) multiple sources of evidence showing active participation by pharmacists in project. Three of the nine projects did not reach completion. For the outcome of lack of project completion, two conditions needed to jointly appear together – (1) involvement of members outside of the core team, and (2) absence of multiple sources of evidence showing active participation by pharmacist in the project. Our evaluation found that organizational- and operational-level factors, rather than pharmacist-level factors, were most influential to the successful completion of pharmacist clinical services projects.
Keywords: Implementation science; Coincidence Analysis; Primary care; Pharmacist clinical services; Pharmacist integration; Teams
Introduction
In 2019, the Connecticut Office of Health Strategy contracted with the University of Connecticut Pharmacy Technical Assistance (TA) team to provide a Pharmacy TA Program for clinical pharmacist integration as part of its State Innovation Model initiative funded by The Center for Medicare and Medicaid Innovation. This Pharmacy TA Program was available to all Primary Care Organizations (PCOs) participating in the program as a no-cost opportunity to initiate, optimize, or advance clinical pharmacist capabilities and implement best practices that contribute to complex care management and comprehensive medication management [1,2].
Few PCOs have access to expertise and resources that can optimize clinical pharmacist integration, identify and select highvalue clinical pharmacist services, and evaluate the impact/value (e.g., workflow, role delineation, productivity, patient outcomes, provider clinical workload burden, quality measures) of the clinical pharmacist in the primary care setting. We previously have described a technical assistance program to assist PCOs with the optimization of clinical pharmacist roles and integration in teambased care or population health programs [3].
Beyond the presence of having a pharmacist on staff, little is known about the implementation factors associated with PCO project completion to advance pharmacist clinical services integration in population health or direct care teams [4]. In this study, our objective was to assess the plausible implementation factors at the organizational, operational, and pharmacist level to identify factors that contributed to the completion or incompletion of pharmacist projects.
Methods
The study was designed as a secondary analysis and observational study to assess nine projects that were conducted to advance pharmacist clinical services in 4 PCOs -- an academic health center, federally qualified health center, regional health system, and accountable care organization -- that participated in the Pharmacy TA program.
A logic model was previously published [3] and the pharmacist projects covered a variety of topics that were tailored to the specific needs of each PCO. Project examples included the quality improvement of medication-related outcomes for patients with uncontrolled hypertension and diabetes, comprehensive transition of care medication reviews, development of a strategic roadmap for developing pharmacist clinical services, workflow maps for integrating pharmacist services, developing key performance indicators for pharmacist productivity and return-on-investment, development of a pharmacist workload capacity tool and scalability model, and development of a medication refill service.
Development and Calibration of Factors
During the TA program, data were collected from multiple sources including PCO demographic data; discussions and meetings with PCO medical, pharmacy, and administrative leaders; on-site workflow observations; and pharmacist coaching sessions.
The research team sought to identify which factor(s) best explained the outcome of PCO project completion. Factor selection and calibration was an iterative process. Two researchers (MAS, BS) were key members of the Pharmacy TA team. They first identified candidate factors based on their knowledge of organizational readiness for change models, implementation science frameworks, and literature. A third researcher (DS) completed an intensive training program on coincidence analysis (CNA ) [5] and guided the team through the application of the CNA analysis. Because this research team member (DS) was not involved in the Pharmacy TA Program, she also served as an objective reviewer and discussant during the factor selection and calibration process.
Next, each candidate factor was further discussed and critically assessed by the research team to establish if the factor was relevant to the outcome of interest and if so, we created a definition that captured the intention of the factor. This process occurred iteratively over several meetings until there was agreement amongst the team.
Using this process, the research team selected and defined 45 factors and subsequently calibrated each factor across the 9 projects (Table 1). Two research team members (MS, BS) independently assigned each factor a value for each project; factors were coded as either dichotomous or multivalue. These calibrations were based on team member observations and information gathered from meetings and discussions with PCO staff and leadership during the project phase. Disagreement was resolved through discussion facilitated by the third team member, until consensus was reached.
|
Outcome Factor = Pharmacist Project Completion |
|
Organizational Level (19 Factors) |
|
Administrative Factors |
|
Organizational structure Number of value-based contracts Type of value-based contracts Approach to decision-making Ongoing organizational changes Planned use of project results Organizational impact of pharmacist project |
|
Experience with Clinical Pharmacist |
|
Previous experience with clinical pharmacist Clinical pharmacist role was delineated in job descriptions Organizational expectations of clinical pharmacist capacity Clinical pharmacist tracked activities/performance metrics Stage of pharmacist clinical services integration at beginning of TA services Extent of pharmacist role integration in the organization* |
|
Management Involvement |
|
Senior administrative and clinical leaders’ participation in pharmacist TA discussions Pharmacy management engagement in project oversight Extent of TA service utilization to inform project priorities |
|
Primary Care Collaboration |
|
PCP(s) have expressed value of pharmacist as a care team member |
|
Physician leaders collaborate with the pharmacist to enhance services outside of TA |
|
Collaborative Practice Agreements are established with PCPs |
|
Operational Level (17 Factors) |
|
Utilization of TA Services |
|
Extent of TA service utilization to inform project priorities Total hours of PCO engagement in TA % of weeks with project meetings from orientation Number of on-site Pharmacist Coaching sessions PCO utilization of TA Learning Community Learning |
|
Pharmacist Staffing |
|
Defined clinical pharmacist service model Percent of pharmacist time dedicated to clinical pharmacy services Pharmacist new in current role Pharmacist new to PCO Pharmacist retention through end of TA services* |
|
Project Characteristics |
|
Number of PCO team members actively involved in project Project required involvement beyond core team members Pharmacist workflows for the project had been established Clinical workflow impact of the project had been determined |
|
Project Resources |
|
Pharmacist use of drug information resources (e.g. Lexicomp, Micromedex) Pharmacist use of business intelligence reports* Pharmacist had access to document notes in EHR |
|
Pharmacist Level (8 Factors) |
|
Pharmacist Credentials |
|
PharmD degree Advanced Training/Pharmacy Specialization (e.g., Residency Training, BCACP, BCGP, BCPS) |
|
Pharmacist Clinical Experience |
|
>5 years of clinical experience in primary care Prior experience working under a Collaborative Practice Agreement Prior experience providing CMM for chronic diseases >2 years of work experience at current site Skillset for integrated clinical services (e.g., actionable notes in EHR, critical thinking, building relationships with clinicians, leadership, selfstarter, problem solving) Serves as a pharmacist resident preceptor |
|
KEY: Bold-text factors contributed to completion or incompletion pathway; *factor was combined into a metafactor |
Table 1: List of Factors Used in Data Analysis.
Data Analysis
Coincidence Analysis (CNA) was applied to identify factors related to the PCO’s completion of pharmacy projects. CNA is a configurational comparative method, an established group of analytic approaches based upon Boolean algebra, a regularity theory of causation, and causal inference.
CNA identifies necessary and sufficient conditions linked to a specific outcome (e.g., PCO completion of a project) and can be used to identify difference-making “bundles” of factors that uniquely distinguish one group of projects that were completed from those projects that were not completed [6,7]. Furthermore, CNA allows for conjunctivity and disjunctivity; that is, respectively, when multiple conditions may need be jointly present in order for the outcome to be present, or when multiple paths lead to the same outcome. Unlike traditional regression methods, CNA handles each project as a whole entity rather than deconstructing the project into individual components to analyze in relation to a dependent variable [5,7]. Thus, CNA is particularly suitable for this analysis of projects in order to retain their complex and unique structure as it relates to the outcome.
Most of the 45 candidate factors were multivalue and coded with multiple possible answers. To reduce dimensionality of our dataset and as a first step in creating our analytic dataset, we performed exploratory analysis and used a configurational approach to factor selection that has been described in detail in prior publications [812]. In brief, we applied the minimally sufficient conditions “msc” function in the Coincidence Analysis (“cna”) package (Version 3.4.0) [13] in R (Version 4.2.2) and identified all one, two- and three-condition configurations instantiated within the dataset that met a pre-specified consistency threshold and ran five iterations at five different consistency levels: 75%, 80%, 85%, 90%, 95%. We reviewed the CNA output and looked for configurations that met all of the following criteria: satisfied the consistency threshold; had “best in class” coverage (i.e., higher coverage scores than any other configuration with the same complexity level); aligned with prior theory and expert knowledge; and where the same set of factors -- when taking on different values-- were involved in explaining both the presence and absence of the outcome. Using this configurational output, we identified a smaller subset of four factors to use in the subsequent modelling phase of the configurational analysis.
In the next phase of the analysis, models were developed using the cna package (Version 3.4.0) [13] in R (Version 4.2.2) and RStudio (Version 2023.03.1+446). We modelled the presence and absence of the outcome separately. Final model selection was based on the following criteria: consistency (number of projects covered by the solution that also had the outcome present divided by the total number of projects covered by the solution) of ≥80%; coverage (number of projects covered by the solution that also had the outcome present divided by the total number of projects with the outcome present) of ≥80%; having a common set of factors involved in explaining both the presence and absence of the outcome (i.e., these factors were consistently linked with the presence of the outcome when they took on certain values, and consistently linked with the absence of the outcome when they took on other values); and alignment with theory, project knowledge and subject matter expertise.
The University of Connecticut Institutional Review Board determined this study was not considered human subjects research.
Results
At the consistency threshold of 95%, the exploratory analysis revealed four factors with strong connections to both the presence and the absence of the outcome related to project completion:
Table 2: Solution Visualization for Project Completion (model consistency and coverage=100%). |
||||||||||||||||||||||||||||||||||||||||||||
(1) involvement of team members outside of the core team (i.e., medical and pharmacy leaders), (2) level of pharmacist integration, (3) retention of the pharmacist through the TA services, and (4) pharmacist use of business intelligence reports (Table 2). The data reduction process identified the factor value of “no team involvement outside of the core team” as a potential differencemaker, allowing us to recode this multivalue factor dichotomously as a yes/no variable. Finally, we created a metafactor [5] by combining the information from three of the four remaining factors, which had similar or identical values across projects: level of pharmacist integration (high versus low), pharmacist retention through the TA services (yes or no), and pharmacist use of business intelligence reports (had access and used them vs. had access but didn’t use them). This further reduced dimensionality in our dataset while allowing us to retain the collective contribution of these factors to the outcome. This metafactor is subsequently referred to as “multiple sources of evidence showing active participation by pharmacists,” meaning the presence of high pharmacist integration, pharmacist retention through the TA services, and pharmacist use of business intelligence reports.
Project Completion Pathway
Six of the nine projects were completed. The modelling phase identified two pathways to project completion, with 100% consistency and 100% coverage and no model ambiguity:
1. No involvement outside of the core team OR
2. Multiple sources of evidence showing active participation by pharmacists in the project.
Incomplete Project Pathway
Three of the nine projects did not reach completion. The modelling phase identified a single conjunct that consistently distinguished projects that did not reach completion, with 100% consistency and 100% coverage and no model ambiguity. For the outcome of lack of project completion, both conditions needed to jointly appear together:
- Involvement of members outside of the core team, combined with
- Absence of multiple sources of evidence showing active participation by pharmacists in the project.
Discussion
Project completion: Six projects were completed and included various topics to guide the role and responsibilities of the integrated pharmacist, analyze pharmacist workflows and capacity, build a strategic roadmap for pharmacist services, and assess the impact of the pharmacist clinical services on the clinical outcomes of patients with uncontrolled diabetes and hypertension.
Two unique factors contributed to pharmacist project completion -- no involvement outside of the PCO core team or multiple sources of evidence showing active participation by pharmacist in the project.
The type and scope of the completed projects could largely be addressed by the PCO core team members without a high need for involvement of other parties for successful completion. The core team members included pharmacy, medical, and administrative leaders who were actively involved in identifying project topics and scope. They had the decision-making authority needed to keep project plans on task, gather necessary resources, and they met on a regular basis to review project progress. In addition, they had a vested interest in seeing the project completed so that results could be incorporated into strategic business or sustainability planning.
Even if a project did require the involvement of PCO staff outside the core team (e.g., medication management workflow changes that required the input of practice-level medical and nursing staff), the presence of three other factors that demonstrated active pharmacist participation contributed to successful project completion. These factors include presence of high pharmacist integration, pharmacist retention through the TA services, and pharmacist use of business intelligence reports. These factors indicate that the pharmacists’ leadership in conducting the projects was critical for project completion. The pharmacist was instrumental in gathering necessary data (i.e., quality improvement reports), meeting with the PCO staff to explain the pharmacist workflow, discuss the impact on changes in other staff members’ workflow, and determine workable solutions.
Lack of project completion: Three projects were not completed and can be categorized as projects that involved pharmacistprovided comprehensive medication management for patients with uncontrolled hypertension and medication refill renewals, as well as conducting focus groups with physicians and practice managers to promote the uptake of pharmacist clinical services.
Two combined factors contributed to the lack of project completion: required the involvement of members outside of the PCO core team, combined with the absence of multiple sources of evidence showing active participation by pharmacist in the project. Both conditions needed to jointly appear together.
The project with pharmacist-provided medication management services for patients with uncontrolled hypertension required the involvement of physicians, nurses, medical assistants, and scheduling staff members. Also, it required a change in workflow for PCPs and other staff members that was a change from their established daily routines. In addition, the pharmacist was not fully aware of the integration challenges and left the practice.
For the project with establishing a new medication refill renewal service, this involved a change in the usual workflow of PCPs and nurses. Since the pharmacist was not fully integrated as a team member and eventually left the practice, the project stalled.
The project with focus groups to promote the uptake of pharmacist clinical services required the participation of PCPs and patients. In addition, the pharmacist was not well-known to PCPs or patients, which contributed to a low level of pharmacists’ integration. Finally, the pharmacist had access to, yet did not use, business intelligence reports to identify PCPs’ and patients’ use of the pharmacist clinical services.
In this study, we did not find any pharmacist-level factors (e.g., training, clinical experience) that contributed to the successful completion or lack of completion for pharmacist clinical service projects.
Limitations
This study has some limitations that are commonly related to retrospective data analysis, which by nature is limited to the information obtained during the TA program. PCOs were not surveyed to validate all the factors that led to or prevented them from successfully completing all the projects. The research team may not have been aware of all the factors that weighed into the pharmacists’ or practices’ success in project completion. In addition, the TA program was offered to the PCOs as a no-cost opportunity. It is unknown whether the PCOs would have prioritized project completion if they had to pay for the TA program. The retention of the pharmacist coincided with the project completion timeframe; it is not known if the pharmacist would have left the program regardless of successfully completing projects.
Conclusion
Our evaluation found that an organizational factor and 3 operational-level factors were most influential to the successful completion of pharmacist clinical services projects. At the organizational level, the most important factor was the extent of pharmacist role integration in the organization. Three operational factors influenced the successful completion of pharmacist clinical services projects: no involvement outside of the PCO core team, pharmacist retention through end of TA services, and pharmacist use of business intelligence reports.
Understanding factors that contributed to the successful completion or incompletion of pharmacist clinical services projects can inform future efforts to engage primary care organizations and PCP leaders to advance pharmacist integration on primary care and population health teams. Projects that can be led by the pharmacist, especially when the pharmacist is integrated on the primary care team, are more likely to be successfully completed.
In addition, this study can inform implementation aspects of policy efforts related to the integration of pharmacist clinical services with expanded primary care teams and building a strategic roadmap for pharmacist-led interventions in primary care practices.
Acknowledgments
The authors would like to acknowledge the contributions of the primary care leaders at four CT primary care organizations that participated in the Connecticut State Innovation Model Pharmacy Technical Assistance Program.
This work was partially funded by Henry A. Palmer Endowed Professorship, US Department of Health and Human Services/CMS State Innovation Model Award Number 1G1CMS33160-02-00, and UConn School of Pharmacy. Funders did not have a role in the study design, data collection/interpretation, or manuscript development.
Author Conflicts of Interest: The authors declare no relevant conflicts of interest or financial relationships
References
- Smith M, Bates DW, Bodenheimer TS (2013) Pharmacists belong in accountable care organizations and integrated care teams. Health Aff (Millwood) 32: 1963-1970.
- Smith MA (2017) Implementing primary care pharmacist services: Go upstream in the world of value-based payment models. Res Social Adm Pharm 13: 892-895.
- Smith MA, Mulrooney M, Shipley B, Soberiaj D (2023) A technical assistance program approach for pharmacist clinical services integration in primary care organizations. J Am Pharm Assoc (2003) 63: 952-960.
- Mulrooney M, Smith M, Sobieraj D, Shipley B, Miech EJ (2022) Factors influencing primary care organization commitment to technical assistance services for clinical pharmacist integration using configurational comparative methods. J Am Pharm Assoc (2003) 62: 1564-1571.
- Whitaker RG, Sperber N, Baumgartner M, Thiem A, Cragun D, et al. (2020) Coincidence analysis: a new method for causal inference in implementation science. Implement Sci 15: 108.
- Thiem A (2017) Conducting configurational comparative research with qualitative comparative analysis: a hands-on tutorial for applied evaluation scholars and practitioners. Am J Eval 38: 420-433.
- Baumgartner M, Falk C (2023) Boolean difference-making: a modern regularity theory of causation. The British Journal for the Philosophy of Science 74.
- Adams KK, Miech EJ, Sobieraj DM (2022) Factors that distinguish opioid withdrawal during induction with buprenorphine microdosing: a configurational analysis. Addict Sci Clin Pract 17: 55.
- Dodge JR, Youles B, Caldararo J, Sears ED, Tanner J, et al. (2023) Engaging operational partners is critical for successful implementation of research products: a coincidence analysis of access-related projects in the Veterans Affairs Healthcare System. J Gen Intern Med 38: 923-930.
- Dodge J, Sullivan K, Miech E, Clomax A, Riviere L, et al. (2024) Exploring the social determinants of mental health by race and ethnicity in Army wives. J Racial Ethn Health Disparities 11: 669-684.
- Knott CL, Miech EJ, Woodward N, Huq M (2023) The role of organizational capacity in intervention efficacy in a church-based cancer education program: A configurational analysis. Glob Implement Res Appl 3: 284-294.
- Miech EJ, Freitag MB, Evans RR, Burns JA, Wiitala WL, et al. (2021) Facility-level conditions leading to higher reach: Aa configurational analysis of national VA weight management programming. BMC Health Serv Res 21: 797.
- Ambuehl M, Baumgartner M, Epple R, Parkkinen VP, Thiem A (2021) Causal Modeling with Coincidence Analysis. R Package Version 3.2.0. 2021.
© by the Authors & Gavin Publishers. This is an Open Access Journal Article Published Under Attribution-Share Alike CC BY-SA: Creative Commons Attribution-Share Alike 4.0 International License. Read More About Open Access Policy.