Introduction

In December 2019, a statement was released by the Wuhan region of China describing cases of viral pneumonia. The causative virus was identified as SARS-CoV-2, the novel coronavirus that causes COVID-19 [1]. As of December 16, 2022, there have been nearly 650 million confirmed cases of COVID-19 globally, and over 6 million deaths worldwide. The first case of COVID-19 in New York State was identified on March 1, 2020, and the first case in Westchester County was identified two days later [2]. During the ensuing days and weeks, there was rapid community spread of SARS-CoV-2, and COVID-19 was declared a pandemic by the World Health Organization on March 11, 2020. On March 15, 2020, NYMC made the decision to suspend all clinical rotations for third- and fourth-year medical students. In response to this decision, clerkship leadership was tasked with designing new curricula that fit within these evolving guidelines. Within 24 hours of this announcement, the virtual pediatric clerkship was designed and implemented, and modifications were made throughout the clerkship. In July 2020, medical students were partially allowed back into the clinical arena; the hybrid clerkship was developed.

To our knowledge, there are no studies comparing different curricular models of a single clerkship in response to the COVID-19 pandemic. Previous studies have described changes in clinical education due to COVID-19 restrictions in many fields [313], most of which have been brief and largely descriptive. Some authors have performed systematic reviews identifying challenges to medical education due to the pandemic and innovative solutions [14], others have assessed the perceptions and attitudes of medical students toward clinical training during the pandemic [15,16], while still others have argued that innovations in medical education due to COVID-19 should be studied as quality improvement [17]. Few studies have been performed describing changes to the pediatric clerkship due to COVID-19; one evaluated the virtual curriculum but did not compare it to other curricular models [18]. In this study, we analyze a single clerkship’s response to the COVID-19 pandemic over time, and use student perceptions to evaluate three distinct curricular models.

We hypothesized that medical student perspectives on the three clerkship models would differ according to the changes made in response to COVID-19, particularly regarding changes to time spent in direct patient care and virtual learning. In our study, we aim to identify strengths and weaknesses of the different clerkship models, and use medical student feedback to further improve the pediatric clerkship.

Methods

Description of clerkship models

Traditional: The traditional (pre-COVID-19) pediatric clerkship at NYMC was six weeks weeks in duration. Students spent three weeks on inpatient teams, one week in the emergency department (ED), one week in an outpatient primary care clinic, and one week in the well-baby nursery. In addition to clinical work, students participated in weekly small group sessions during which they had the opportunity to practice their presentation and clinical reasoning skills, as well as simulation days.

 Virtual: The pediatric clerkship was fully virtual for two rotations, between April and June 2020. The duration of these rotations varied from 4-5 weeks. Students spent time on different “clinicalvirtual experiences”: inpatient pediatrics for approximately half of the rotation, with the remainder divided between ED, outpatient and nursery. Students engaged in self-directed learning, including mandatory interactive virtual cases using the Aquifer platform, as well as textbook and online learning. Students also had virtual sessions with clerkship leadership during which they discussed clinical aspects of pediatrics pertinent to their clinicalvirtual experience. In addition, students continued to participate, now virtually, in weekly small group sessions, as well as virtual simulation days. To supplement the virtual clerkship, students later returned to the clinical arena during their fourth year; the survey was filled out before this return.

 Hybrid: In July 2020, medical students were allowed back into the clinical arena, with modifications. Clerkship leadership designed a 6-week hybrid curriculum, blending clinical and virtual elements. All students spent two weeks on inpatient and one week in the well-baby nursery. Only a minority of students were able to participate in outpatient clinics, and time in the ED was limited. Students spent one week engaged in virtual learning, completing mandatory Aquifer cases and meeting virtually with clerkship leadership, and those students who did not participate in outpatient clinics had a virtual outpatient week. During the hybrid clerkship, students were not allowed to see COVID-positive or COVIDsuspected patients, including those with multisystem inflammatory syndrome in children (MIS-C). In addition to clinical and virtual work, students also participated, still virtually, in weekly small group sessions and simulation days.

Participants

Medical students who rotated through the pediatric clerkship during the 2019-2020 and 2020-2021 academic years were invited to participate in the study. Students were recruited to participate in the online survey via email. Inclusion criteria were students participating in the traditional clerkship (July 2019-February 2020), virtual clerkship (April-June 2020) and hybrid clerkship (July-December 2020). Students who rotated through the pediatric clerkship from mid-February to March 2020, whose clerkship model was traditional for the first four weeks and virtual for the last two weeks, were excluded from the study.

The study was deemed exempt by the institutional review board at NYMC. Students indicated their willingness to participate in the survey electronically, upon initiation of the survey.

Measures

Surveys

Surveys were created using Qualtrics software (Qualtrics, Provo, UT) and sent to students via email. Survey questions were divided into five categories: questions related to clerkship objectives, general questions, and questions related to clinical experience and skills, readiness, and wellness. There were 3-7 questions in each category, 25 questions in total, and questions were rated on five-point Likert scales. Anchors for clerkship objectives questions ranged from 1 = not well at all to 5 = extremely well, and those for all other questions ranged from 1 = strongly disagree to 5 = strongly agree. Answers to survey questions remained anonymous. Survey data were pooled for analysis and kept on a password-protected online database on NYMC’s secure network.

Post-Clerkship Evaluation Comments

It is NYMC’s standard practice to send formal course evaluation forms to each student at the completion of a clerkship. This post-clerkship evaluation is optional and anonymous, and includes both Likert scale questions and space for the student to provide comments. Comments completed by students in all curricular models were reviewed, pooled and analyzed for themes. Statistical Analysis

All statistical analysis was done using Microsoft Excel 2021.

Standard descriptive statistics were used to assess participants’ responses to survey questions. Responses were then compared by group based on clerkship model. Post-clerkship evaluation comments were reviewed individually by all three authors and analyzed for themes. Results

Survey Data

Forty-eight students completed the survey: 18 students in the traditional clerkship model, 7 in the virtual model and 23 in the hybrid model. Completion rate was 13% for traditional and virtual students and 22% for hybrid students.

Survey questions and results are reported in (Table 1) and grouped by category. Throughout their responses, student ratings followed a consistent pattern. Average scores in all categories were consistently highest for traditional students, lowest for virtual students, and returning almost to the level of traditional for hybrid students.

Table 1: Survey questions and results.

To help visualize this pattern, data from the Clerkship Objectives portion of the survey are shown in the Figure. Students rated each objective as being met best in the traditional clerkship (average 4.2) and worst in the virtual clerkship (average 2.8), and then ratings for the hybrid model increased (average 3.7). For the first of the clerkship objectives, “Demonstrate proficiency in pediatric history taking and exam skills for evaluation of children of various ages”, the percentage of students reporting that this objective was met extremely well or very well ranged from 83% for traditional students, 0% for virtual students, and 48% for hybrid students. Similarly, the percent of students reporting that the objective “Employ strategies for health promotion as well as disease and injury prevention” was achieved extremely well or very well ranged from 83% for the traditional clerkship, 29% for the virtual clerkship, and 52% for the hybrid clerkship.

Regarding general questions about the clerkship, average score for traditional students was 4.6, average for virtual students was 3.5 and average for hybrid students was 4.2.

In the Clinical Experience and Skills category of the survey, 89% of students in the traditional clerkship strongly agreed or somewhat agreed “The clerkship allowed me to practice and improve oral presentation skills”, 71% of students in the virtual clerkship strongly or somewhat agreed, and 78% of students in the hybrid curriculum strongly or somewhat agreed.

With respect to readiness, average score was 4.5 for traditional students, 3.8 for virtual students, and 4.1 for those rotating in the hybrid clerkship.

From a wellness perspective, 83% of traditional students, 43% of virtual students and 87% of hybrid students strongly or somewhat agreed that “My voice was heard during this clerkship.”

Figure 1: Survey results for Clerkship objectives:

Figure 1a: Survey results for Clerkship objectives for students in the traditional clerkship model.

Figure 1b: Survey results for Clerkship objectives for students in the traditional clerkship model.

Figure 1c: Survey results for Clerkship objectives for students in the hybrid clerkship model.

Post-Clerkship Evaluation Comments

Five key themes emerged from analysis of the post-clerkship evaluation comments (See Box). These themes included: (1) Participation in interdisciplinary teams, (2) Direct patient care, (3) Virtual learning material as supplements to clinical experiences, (4) Faculty teaching, feedback, and support, and (5) Self-directed study time.

Theme 1: Participation in interdisciplinary teams.

Students in both the traditional and hybrid clerkship models voiced appreciation for having as much experience working with interdisciplinary teams as possible. Those in the virtual clerkship did not comment on this, as they did not have the opportunity to interact with clinical teams.

Theme 2: Direct patient care

Students participating in both the virtual and hybrid models expressed their desire to participate in direct patient care. Students in the virtual clerkship understood that direct patient care was not possible during their clerkship, while still citing it as a deficiency in their experience. While those in the hybrid curriculum had more patient contact than those in the virtual curriculum, they still reported the decreased amount of direct patient contact as a deficiency, particularly the lack of in-person outpatient experience.

Theme 3: Virtual learning material as supplements to clinical experiences.

Student perspectives on virtual learning materials were overall positive regardless of clerkship model. Although students appreciated the virtual case-based experiences, they reported a plateau in their educational value in the absence of direct patient care.

Theme 4: Faculty teaching, feedback, and support.

Regardless of curricular model, students appreciated having quality time with clerkship leadership and faculty. They particularly enjoyed small group settings, both in-person and virtual.

Theme 5: Self-directed study time.

Students reported that they appreciated extra study time afforded to them during virtual portions of the virtual and hybrid clerkship models. However, overall, they still preferred direct patient interaction.

Discussion

The COVID-19 pandemic has significantly disrupted medical education, particularly the clinical years. The initial dismissal of medical students from their clinical duties was nearly universal, due both to a desire to protect medical students from a novel virus about which little was known and to concerns about the availability of personal protective equipment. After these first few weeks and months, however, the decision of how to reintegrate medical students into the clinical arena varied.

This study shows the evolution of our pediatric clerkship through the lens of medical students. While students identified deficiencies in the initial virtual clerkship model compared with the traditional model, implementation of the hybrid curriculum improved student perspectives on many aspects of the clerkship to levels approaching the traditional model.

Despite significant challenges, the pediatric clerkship was able to function effectively in a virtual format. Pediatrics was taught by a combination of self-directed learning and virtual meetings with faculty; focus was placed on clinical reasoning skills, patient presentations, and building differential diagnoses.

Students identified certain clerkship objectives that were more difficult to meet in a virtual format, particularly history taking, physical exam skills, and collaboration with interdisciplinary teams. This is hardly surprising, as these objectives are nearly impossible to achieve virtually. Understandably, students in the virtual curriculum expressed disappointment over their lack of ability to participate in patient care. Virtual cases, even if educationally thorough, are a poor proxy for live patients. Meetings with clerkship leadership improved upon this weakness, by encouraging students to ask questions through role-play and case discussions.

Previous studies have explored student perspectives on virtual curricula prior to the COVID-19 pandemic. Kim, et. al. found generally low levels of student-perceived value of virtual patient cases [19]; in that study, virtual cases were meant to supplement clinical experiences, not replace them. Alexander, et. al. compared a web-based radiology course with the in-person course, also pre-pandemic, and found that it allowed students flexibility in traveling for interviews [20]. We anticipated that self-directed study time would be a welcome respite for medical students. Some students appreciated this time, citing relaxation and ability to focus on specific topics. Although students appreciated the ample time dedicated to self-directed learning, they felt that after a certain number of virtual cases, they reached a threshold of knowledge attainment.

Overall, students in the virtual curriculum rated their wellness low, with fewer than half of students reporting that their voice was heard during the clerkship. It is difficult to differentiate between lack of student wellness caused by the virtual clerkship and distress secondary to the individual experience of the global pandemic. As reflected in comments from the post-clerkship evaluations, students appreciated small group educational sessions and a more relaxed environment.

As restrictions eased, clerkship leadership developed a hybrid curriculum, in which students were able to enter the clinical arena, while continuing to supplement students’ education with virtual materials and experiences. Many of the clerkship objectives that were difficult to meet in the virtual clerkship were more successfully achieved in the hybrid curriculum. History and physical exam skills were greatly improved in this model when compared to the virtual model, as were student ratings of collaboration with interdisciplinary teams.

While the hybrid curriculum was better able to meet clerkship objectives, the student experience of this clerkship model remained inferior to the traditional model due to still-limited patient care, particularly gaps in ED and outpatient experiences. In post-clerkship evaluation comments, students expressed appreciation for the teaching by faculty and residents while inperson. Interestingly, while they did express a desire to have more outpatient and ED experience, they did not note the inability to see COVID or MIS-C patients as a deficiency.

The improvement in student perception is consistent with previous reports analyzing multimodal or blended curricula. The concept of a hybrid curriculum in medical school is not new. Prior to COVID, Chandra, et al. found success in a blended EM curriculum [21], and Dafli, et al. explored strategies for incorporating virtual patients in medical education [22]. Darras, et al. explored blended learning for radiology education affected by COVID [23]. However, optimizing this hybrid model in the time of COVID provides novel and unique challenges, and there is room for improvement. Augmenting not only the amount of clinical time, but also the breadth of clinical experiences including outpatient and emergency medicine, may improve the hybrid curriculum.

This study has a few important limitations. First, it is a singleinstitution study, which limits generalizability. Another limitation is the small sample size; although the survey was sent to all third- and fourth-year medical students, only 48 responded. As a third limitation, we were unable to craft our ideal version of a hybrid clerkship. Outpatient medicine is an integral part of pediatrics, encompassing growth and development, immunizations and anticipatory guidance, and the lack of this clinical experience is a detriment to students. A fourth weakness included the comparison of different groups at different times of the academic year. The traditional and hybrid students in the study rotated during the beginning and middle of the academic year, while the virtual students participated significantly later in the academic year.

This study also has several strengths. This is the first study to our knowledge to investigate the evolution of a single clerkship due to the COVID-19 pandemic. It also compares three distinct models, instead of simply two.

The multimodality of the study, examining both elicited (survey questions) and non-elicited (evaluation comments) student perceptions, lends it strength, as does the mix of examining both quantitative and qualitative data. This process of continued evaluation has allowed us to reconsider what was previously considered, by students and faculty, to be an excellent clerkship, thus allowing for even further development of the curriculum, beyond COVID-19-related adjustments. This study has brought to light the important balance between different integral components of pediatrics.

There are elements of the virtual and hybrid clerkships that can be carried forward post-COVID. For example, the virtual platform has allowed for students at multiple sites to participate in lectures together, allowing for better attendance and widened exposure. In addition, faculty who are often busy with clinical responsibilities, as well as retired physicians, have more flexibility to participate in virtual sessions.

Conclusion

In response to the COVID-19 pandemic, the pediatric clerkship at NYMC was challenged to evolve based on frequently changing guidelines and restrictions. With the goal of delivering a valuable clerkship, the traditional clerkship originally redesigned to fully virtual, and then subsequently to a hybrid curriculum. Although hospital restrictions were loosened, limitations in clinical experiences still existed. Medical students perceived the hybrid curriculum model to be superior to the virtual curriculum model, at levels approaching the traditional model. Feedback from students also shows that virtual time can also be restructured, with perhaps less focus on virtual cases, and smaller group discussions with faculty or among the students themselves.

Future studies should continue to monitor students’ perceptions of curriculum changes in response to COVID-19 restrictions and could benefit from focus group design to explore these perceptions more in-depth. Further studies comparing quantitative data, including NBME scores, may be useful once we are able to compare students at similar times of the academic years, for example comparing both traditional and hybrid students in July. We hope that our experiences may benefit other institutions, both in pediatrics and across clerkships.

Supplemental file (Appendix) includes all survey results aside from the results from the questions about clerkship objectives, which are included in the table and figure in the manuscript.

Contributors’ Statement: Drs Lindenbaum, Block and Noulas conceptualized and designed the study, collected data, carried out the data analysis, drafted the initial manuscript, reviewed and revised the manuscript, and approved the final manuscript as submitted and agree to be accountable for all aspects of the work.

Funding/Support: No funding or support was secured for this study.

Conflicts of interest: None.

Declarations of interest: None.

References

1. Coronavirus disease (COVID-19) pandemic (2020) World Health Organization [Internet]. Geneva; 16.
2. New York State Department of Health (2020). New York: 27.
3. Chandra S, Laoteppitaks C, Mingioni N, Papanagnou D (2020) Zooming-out COVID-19: Virtual clinical experiences in an emergency medicine clerkship. Med Educ; 54: 1182-3.
4. Durfee SM, Goldenson RP, Gill RR, Rincon SP, Flower E, et al. (2020) Medical student education roadblock due to COVID-19: Virtual radiology core clerkship to the rescue. Acad Radiol; 27: 1461-6.
5. Guadix SW, Winston GM, Chae JK, Haghdel A, Chen J, et al. (2020) Medical student concerns relating to neurosurgery education during COVID-19. World Neurosurg; 139: e836-47.
6. Kahn JM, Fields EC, Pollom E, Wairiri L, Vapiwala N, Nabavizadeh N, et al. (2020) Increasing medical student engagement through virtual rotations in radiation oncology. Adv Radiat Oncol; 6: 100538.
7. Nnamani Silva ON, Hernandez S, Kim EH, Kim AS, Gosnell J, Roman SA, et al. (2021) Surgery clerkship curriculum changes at an academic institution during the COVID-19 pandemic. J Surg Educ; 78: 327-31.
8. Pollom EL, Sandhu N, Frank J, Miller JA, Obeid JP, Kastelowitz N, et al. (2020) Continuing medical student education during the coronavirus disease 2019 (COVID-19) pandemic: Development of a virtual radiation oncology clerkship. Adv Radiat Oncol; 5: 732-6.
9. Richardson MA, Islam W, Magruder M (2020) The evolving impact of COVID-19 on medical student orthopedic education: Perspectives from medical students in different phases of the curriculum. Geriatr Orthop Surg Rehabil; 11: 2151459320951721.
10. Samueli B, Sror N, Jotkowitz A, Taragin B (2020) Remote pathology education during the COVID-19 era: Crisis converted to opportunity. Ann Diag Pathol; 49: 151612.
11. Sandhu N, Frank J, Von Eyben R, Miller J, Obeid JP, Kastelowitz N, et al. (2020) Virtual radiation oncology clerkship during the COVID-19 pandemic and beyond. Int J Radiat Oncol Biol Phys; 108: 444-51.
12. Wickemeyer JL, Yu J (2021) A model for undergraduate medical student education in otolaryngology during the Post–COVID-19 era. Otolaryngol Head and Neck Surg; 164: 562-5.
13. Armon S, Benyamini Y, Grisaru-Granovsky S, Avitan T (2021) Online Obstetrics and Gynecology Medical Students Clerkship During the Covid-19 Pandemic: a Pilot Study. Med.Sci.Educ; 31: 1-5.
14. Dedeilia A, Sotiropoulos MG, Hanrahan JG, Janga D, Dedelias P, Sideris M (2020) Medical and surgical education challenges and innovations in the COVID-19 era: A systematic review. In Vivo; 34: 1603-11.
15. Kim SM, Park SG, Jee YK Song IH (2020) Perception and attitudes of medical students on clinical clerkship in the era of the coronavirus disease 2019 pandemic. Med Educ Online; 25: 1809929.
16. Park J, Park H, Lim JE, Rhim HC, Lee YM (2020) Medical students’ perspectives on recommencing clinical rotations during coronavirus disease 2019 at one institution in South Korea. Korean J Med Educ; 32: 223-9.
17. Mehta N, Sayed C, Sharma R, Do V (2020) Medical education advances and innovations: A silver lining during the COVID-19 Can Med Educ J; 11: e141-4.
18. Friedman S, Craddock KE, Pitkowsky Z, Catallozzi M (2021) Incorporating near peers for teaching and fast feedback in a rapidly developed virtual pediatric clerkship curriculum in response to the COVID pandemic. Med Sci Educ; 31: 313-4.
19. Kim S, Willett LR, Pan WJ, Afran J, Walker JA, Shea JA (2018) Impact of required versus self-directed use of virtual patient cases on clerkship performance: A mixed-methods study. Acad Med; 93: 742-9.
20. Alexander AG, Deas D, Lyons PE (2018) An internet-based radiology course in medical school: Comparison of academic performance of students on campus versus those with absenteeism due to residency JMIR Med Educ; 4: e14.
21. Chandra S, Papanagnou D, Jenkins M (2020) Evaluation of a blended curriculum on medical student outcomes in an emergency medicine West J Emerg Med; 21: S16-7.
22. Dafli E, Fountoukidis I, Hatzisevastou-Loukidou C, Bamidis PD (2029) Curricular integration of virtual patients: A unifying perspective of medical teachers and students. BMC Med Educ; 19: 416.
23. Darras KE, Spouge RJ, de Bruin AB, Sedlic A, Hague C, Forster BB (2021) Undergraduate radiology education during the COVID-19 pandemic: A review of teaching and learning strategies. Can Assoc Radiol J; 72: 194-200.