Simulation Training and Skill Assessment in Critical Care

Article Author:
Felipe Pacheco Granda
Article Editor:
Irim Salik
Updated:
9/18/2020 6:46:31 PM
For CME on this topic:
Simulation Training and Skill Assessment in Critical Care CME
PubMed Link:
Simulation Training and Skill Assessment in Critical Care

Introduction

Medical students and residents are expected to manage several critical events following training, but they are often ill-equipped to do so because of limited clinical experiences in the diagnosis and management of many such conditions. Simulation-based training can be utilized across multiple medical specialties and is essential to every didactic curriculum to model critical events. One of the tenets of healthcare education is to develop teaching and evaluation techniques that allow instructors to gauge a trainee’s performance in settings that reflect clinical practice. Standardized patients (SPs) have often been utilized to assess the history and physical examination skills for medical students and graduate physicians.[1] Unfortunately, critical care events are not easily reproducible, employing SPs. Simulation exercises are an ideal method to recreate complex crisis situations both in the operating room and the critical care environment. Simulation provides a realistic setting in which to conduct high-fidelity scenarios that are realistic, reproducible over time and eliminate threats to patient safety. Due to the recognition of its effectiveness, and healthcare’s continued emphasis on patient safety, simulation has become more prevalent over the last two decades. The Institute of Medicine’s report “To Err is Human” underscored preventable deaths related to medical errors, leading to an increased focus on training modalities that emphasize patient safety and minimize patient harm.[2] Regrettably, the mandatory implementation of clinical duty hour reduction to minimize the fatigue-mediated risk of medical errors has resulted in fewer procedural opportunities.[3] Due to the circumstances mentioned above, simulation provides an excellent opportunity to fill gaps in residency training while minimizing patient risk. 

Function

Simulators were first introduced in the aviation industry in the 1920s so pilots could receive standardized flight training without putting travelers’ lives at risk.[4] The first documented use of healthcare simulation occurred in 1960 when anesthesiologist Peter Safar in the United States and Bjorn Lind in Norway collaborated with Norwegian toy manufacturer Asmund Laerdal to develop Resusci Anne, a full-size mannequin simulator of a pulseless dying victim.[4] Shortly thereafter, Sim One, a computer-controlled mannequin simulator, was developed and utilized primarily to teach endotracheal intubation to anesthesia residents.[5] A decade later, Harvey, another computer-controlled mannequin simulator, with the namesake of a cardiologist renowned for teaching auscultation skills, was born, to aid in the detection of heart murmurs and jugular venous pulsations.[6] The evolutionary next step in healthcare simulation was by Gaba and colleagues, who invented a high-fidelity simulator to study anesthesiologist response time to simulated critical care events in the operating room and to train anesthesia providers in crisis resource management (CRM).[7] These advancements are the foundation for much of today’s healthcare simulation resources.

Adverse clinical events in healthcare commonly occur due to interpersonal communication breakdown, systems errors, and lack of coordination between patient care teams. These realizations have largely lead to the emergence of the CRM field.[8] CRM seeks to manage perilous situations by emphasizing the importance of human factors such as hierarchy, culture, fatigue, and the anticipation of errors in stressful situations. Trainees are taught nontechnical skills, including leadership, communication, task management, and mutual respect for colleagues in a simulated environment.[9] an interprofessional training of interpersonal skills has become imperative in high-risk emergencies, including trauma resuscitation, cardiac arrest, and obstetrical emergencies. Systematic reviews reveal improved overall team effectiveness, coordination, and communication skills following team training.[10] A study of critically ill trauma patients demonstrated that team training improved the efficiency of patient care, defined by reduced times from patient hospital arrival to intubation, radiologic imaging, and operating room entry time.[11]Team training also showed improved outcomes in post-op pediatric surgical cardiac arrest, with a sustained increase in teamwork concepts up to three months following formal training.[12] Another significant result of team training has been the prompt engagement of emergency protocols such as rapid response teams due to the empowerment of clinical staff. Non-intensive care unit nurses were shown to have less anxiety and self-doubt following team training in the activation and utilization of the rapid response team.[13]

Issues of Concern

Debriefing is essential for the trainee to have a comprehensive and successful encounter in a simulation scenario. Debriefing refers to facilitated reflection in the process of experiential learning. Below are some facilitator guided debriefing tools that educators have described. “Debriefing with good judgment” refers to a three-phase conversational structure consisting of reaction, analysis, and summary. The learner is afforded an initial “reaction” to explore his/her emotional response to the simulation encounter. The analysis phase focuses on what happened during the simulation and why the learner performed the way they did. The summary phase focuses on participants describing what they learned and how this knowledge can be applied to future patient care scenarios.[14] Yet another technique is known as the "GAS" debriefing tool, referring to gather, analyze, and summarize. Gathering allows learners to establish a shared mental model through the reiteration of simulation events. Analysis utilizes self-reflection to dissect the exercise while summary enables participants to assimilate activities and knowledge.[14]

Promoting Excellence and Reflective Learning in Simulation, also known as PEARLS,[15] expands on the three phase method to include an additional description phase to identify major clinical difficulties or key events in the creation of a shared mental model. The Plus-Delta model allows learners to identify positive aspects of simulation under the plus sign and ideas for improvements or changes under the delta. The focus of this activity is not only on patient outcomes but also on the identification of systemic processes that may have contributed to these outcomes. A hybrid debriefing model, TeamGAINS, utilizes elements of team guided self-correction, advocacy-inquiry, and constructivist approaches to the debriefing process.[15] Rather than focus on individual behavior, this model centers on team dynamics and individuals within their respective system.

High fidelity medical simulation should be a close approximation to clinical practice and is most successful when feedback is provided directly during and after the learning experience.[16] With regard to medical education, simulation-based practice has a direct relationship to achieving desired outcomes; more practice yields superior results for trainees. It acts as an excellent complement to patient care grounded in clinical care settings. Simulation should be a part of a comprehensive didactic curriculum, including clinical experience, high yield lectures and question review, problem based learning discussions, journal clubs, and other learning modalities. Trainees should practice with increasing levels of difficulty and the simulator should be adaptable to multiple learning strategies. While the simulator should be able to capture a wide degree of clinical variation, it should be entrenched in a controlled environment. Trainees should be able to experience individualized acquisition of knowledge, and outcomes should be clearly defined and measured by the instructor.

Curriculum Development

Previously, in the critical care environment, training methods for skills teaching commonly utilized an apprenticeship technique. With this approach, an experienced teacher modeled the correct technique for performing a specific skill and then evaluated the trainee, also known as "see one, do one, teach one." This model is now obsolete. The constructivist learning theory suggests learners gain knowledge based on their personalized experiences. Trainees’ clinical knowledge becomes solidified, and instructors can highlight key learning points during the simulation.  Participants should be made to feel supported and respected, with the instructor encouraging nonjudgmental and open discussion. In addition to constructing and effectively executing simulation scenarios, debriefing is crucial to help learners identify gaps in knowledge, pre-conceived attitudes, and emotional reactivity that can contribute to performance deficiencies. Scaffolding refers to an effective debriefing session that guides learners to think through their actions and reactions, allowing the integration of simulation sessions with previous experiences.[17] Educators must set realistic objectives for each session based on the learners’ level of training. Active participation of trainees and comprehensive feedback is integral to solidify concepts and techniques learned during the simulation.[18] There are some limitations to the implementation of a simulation-based curriculum for trainees. It can be costly and resource-intensive in terms of the availability of staff and equipment.

The Kirkpatrick model for evaluating the effectiveness of training programs can be utilized to ascertain the utility of simulation for learner assessment and ongoing quality improvement (QI) of intensive care practice. Kirkpatrick’s hierarchy serves as a classification tool to communicate the level of learning outcome for each trainee.[19] In its original framework, there are four levels of classification of learning outcomes from educational interventions. Level one is known as reaction and refers to participants’ perception of an educational intervention. Level two is known as learning, and measures participants’ acquisition of skills, knowledge, and change of attitudes in a non-clinical setting. Evaluation during such training should demonstrate knowledge. Level three is behavior, which measures a learner’s behavioral changes, attitudes, skills, and knowledge from the training setting to a professional practice setting. Level four, referred to as results, measures the effect of learners’ skill acquisition through training on patient outcomes.[20]

Clinical Clerkships

Meaningful assessment of learners is provided by the Accreditation Council for Graduate Medical Education (ACGME) through entrustable professional activities (EPAs) and milestones.[21] Essential for all medical subspecialties, EPAs are tasks that trainees are expected to perform successfully and independently following graduation. Milestones refer to competency-based developmental outcomes that can be progressively demonstrated from the beginning of residency to graduation.[22] In 2014, milestones for critical care medicine fellowship training were published, utilizing simulation to evaluate the competence of fellows.[22] Nationally, six training programs implemented a high fidelity simulation program to assess trainee’s resuscitation of critically ill patients.[23] Through this program, fellows’ ability to work effectively and successfully with other members of the inter-professional care team was assessed and shown to be superior to fellows trained traditionally. Numerous studies have substantiated successful skill acquisition with simulation. A prospective randomized trial found that residents’ endotracheal intubation performance showed improvement with the use of simulator training as compared to traditional teaching.[24] Also, pulmonary fellows trained with virtual reality bronchoscopy performed with more agility on their first patient bronchoscopy and demonstrated a more rapid progression in clinical bronchoscopy performance compared to traditionally trained fellows.[25]

Psychologist K Anders Ericsson originated deliberate practice (DP), which refers to simulation oriented interventions that create and refine a trainee’s knowledge, skills, and attitudes.[26] The foundation of DP lies within information processing and behavioral theories of skill acquisition and preservation. Successful DP requires highly motivated learners with good concentration, engagement with a well-defined learning objective at an appropriate level of difficulty, and repetitive practice that yields informative feedback from educational sources. Trainees should monitor their learning experiences and subsequently engage in more DP until reaching a mastery standard with advancement to another more complex task. National medical foundations are increasingly turning to simulation for credentialing and continuing education purposes. The American Board of Surgery requires the ‘Fundamentals of Laparoscopic Surgery’ course to be utilized as a summative simulation-based assessment.[15] Also, the American Board of Anesthesiology allows certified healthcare simulation courses to meet its QI maintenance of certification requirements.[27] Thus far, simulation-based assessment is not a part of critical care certification.

Assessment of Procedural Skills

The three primary sources for simulation evaluation are observational ratings of trainee performance, trainee responses, and haptic sensors.[28] Observational ratings are subject to sources of potential bias unless conducted with rater training and calibration in a controlled environment. Trainee responses are either multiple-choice questions or constructed-response data that are usually more reliable than observational ratings. [29] Simulators can capture and record trainee “touch” in terms of pressure location and depth at specific anatomical sites, defining haptic sensors.[30]

The various simulation modalities available to an educator include human patient simulators, task trainers, SPs, and virtual reality/augmented reality. Task trainers represent one body part or structure and are useful for high-volume basic skills training. These are less expensive than the purchase of a full mannequin, easy to operate and allow repetition of specific skill sets. Disadvantages of task trainers include the frequent need for replacement, maintenance of parts due to high utilization, and a limited focus of training. Full human patient simulators include full-body mannequins of various sizes, race, sex, and age, and can be either high or low fidelity. While low fidelity mannequins are static, more cost-effective, and allow the practice of a specific skill, they lack realism and are unable to be utilized to gauge interpersonal skills. High-fidelity mannequins are more realistic in terms of their bodily functions, vital signs, and hemodynamic parameters, and are manipulable by an instructor. They are, unfortunately, more expensive to purchase and maintain, and require professional training and knowledge for successful operation.

SP’s are live actors that portray a role or specific medical condition. These actors can adapt or adjust to the learner’s level of expertise, and physical assessment skills are honed on a live person. The SP can also provide individualized feedback to the trainee. Unfortunately, such actors are expensive, may find it challenging to recruit and train, and may present logistical scheduling issues. Virtual reality (VR) or augmented reality simulation is a computer-generated three-dimensional interactive environment that provides an immersive effect for the learner.[31] This technology does not require simulation space, and trainees’ performance is measurable within the VR environment. VR requires a trained individual to operate the software and equipment, which can be expensive, and there can be significant variability in the quality of graphics. Most commonly, VR simulators are used to train surgeons and invasive cardiologists for very high risk, complex procedures. Simulated repetition of the task allows a three-dimensional understanding of patient anatomy, familiarity with psychomotor skills, and sequential steps needed for successful completion of the high-risk intervention.[32]

Simulation-based mastery learning is a rigorous approach to competency-based education that involves learners taking a pretest, receiving timely instructor feedback following a simulation activity, and achieving a minimum score on a posttest. Implementation of mastery learning can be logistically challenging as it requires flexibility in both educator time and simulation space and equipment. In a randomized study of training for thoracentesis, internal medicine residents had fewer clinically significant pneumothoraces and no hemothoraces if they engaged in mastery learning versus those who trained traditionally.[33] This study suggests that not only is simulation necessary for mastery learning, but it can also serve as an effective QI strategy.

Continuing Education

Healthcare simulation is commonly utilized to improve technical ICU skills, including specialty training utilizing prosthetic lungs to teach mechanical ventilator management and the use of simulators in training of point-of-care ultrasound (POCUS).[34] POCUS has become a requisite tool in the ICU environment for trainees, junior and senior faculty alike, guiding resuscitation of critically ill patients and aiding in the etiology of shock states, among other uses. A web-based learning workshop followed by critical care POCUS training was shown to improve novice learner confidence, clinical knowledge, and technical skills.[35]Placement of ultrasound-guided central venous catheters was noted to be more efficient with reduced time to insertion in first-year anesthesia, emergency medicine, and internal medicine residents trained with simulation as opposed to traditionally. [36]Another important aspect of critical care training is developing communication skills related to disclosing bad news to patients, empathic support, cultural sensitivity, and shared decision-making. A study examining a 4-day communication workshop for 115 oncology fellows from 62 different institutions found that communication skills showed vast improvement following post-workshop SP encounters compared to pre-workshop.[37] In the same vein, critical care fellows who received training with a simulation-based workshop had improved adult and pediatric patient feedback to their communication skills and lead more productive high stakes discussions based on patient reports.[38]

Clinical Significance

Simulation-based medical education reduces risk to patients and trainees, improves learners’ confidence and competence, increases patient safety, and leads to reduced future health care costs. A reliable healthcare system should have a firm commitment to a culture of safety and leadership committed to continual process improvement. Effective QI processes require interdisciplinary team preparedness, with simulation serving as an ideal tool so that members can learn to prevent adverse events, or expeditiously manage them after they occur. A common area of focus for QI relates to communication errors during the handoff between teams or during a shift change. Simulation has successfully been used to prepare for effective handoff, and to diminish hierarchy-related medical errors.[39]

Simulation is not used as a replacement for didactic or clinical learning but as an important adjunct. The aviation and aerospace industries have utilized simulation for decades, and it also frequently works in high-risk professions, including the military, nuclear power plants, business, and medicine. Experiential learning is defined as an active process whereby a trainee constructs knowledge by linking novel information with previous understanding.[20] An immersive environment is created with the use of clinical scenarios in a team environment. Scenarios can be recorded for feedback during debriefing sessions, which comprise the most important part of a successful full-scale simulation encounter with trainees. Although the benefits of simulation are numerous, the cost of this modality should not be understated. High-fidelity simulators are expensive and require maintenance. Dedicated simulation staff must also be available to schedule, organize, and run scenarios as well as aid educators. The use of SPs for communication simulations can also be costly. Faculty development in the facilitation of simulation excellence and protection of learners’ time for training are also important factors to be considered. Healthcare institutions are forced to encounter these budgetary challenges and weigh the costs of simulation with the benefits it will provide to learners and the potential enhancement of patient care.

Enhancing Healthcare Team Outcomes

Simulation allows the polishing of vital nontechnical skills such as teamwork, situational awareness, task management, and decision making in a controlled and safe environment. The ultimate goal of such tasks is to increase patient safety and improve patient outcomes. The International Network for Simulation-based Pediatric Innovation, Research, & Education (INSPIRE) has established a collaborative process to examine the effects of healthcare simulation, including studies to evaluate the effect of resuscitation training on improving the performance of a code team during cardiac arrest, among others.[40] Intensive care unit training can be augmented as medical education moves away from traditional lectures and embraces simulation to maximize learning. An effective curriculum should include attributes of simulation-oriented mastery learning for trainees before the performance of invasive procedures on patients, as well as refinement of team skills amongst physicians, nurses, and other team members in preparation for emergencies.

Simulation has lead to vast improvements in the clinical training of medical students, residents, and junior faculty. Improved patient outcomes commonly occur with higher fidelity simulation.[41] [Level I] This is especially pertinent in the critical care setting due to the high complexity and acuity of procedures performed daily. Simulation is also more effective when it involves ultrasound and other imaging modalities. [42] [Level III] The goal of high-fidelity simulation is to improve common health practices, decrease medical errors, and improve healthcare outcomes. Results are measurable via patient satisfaction scores and reduced mortality risk for high-risk procedures. In a study performed in nurses, seven 4-hour simulation sessions were offered for 3 weeks in a high-fidelity simulation center. The study analyzed code blue data after 3 months of intervention, finding that the number of code blues decreased by 14%. Although the number of code blues pre-cardiac arrest increased by 52%, the number of pulseless cardiac arrests decreased by 59%, suggesting that after receiving training, nurses were more expedient in identifying patient deterioration and activation of rapid response protocols. This study demonstrated a 16% increase in the success rate of return of spontaneous cardiac circulation after the intervention.[43] [Level III]

Simulation is now utilized for systems assessment and identifying latent safety threats in high-risk environments.[44] [Level V] Various sociological factors should be taken into account during inter-professional simulation education, including hierarchy, power, and team dynamics[45] [Level V]. Psychologist Eduardo Salas and his colleagues claim that miscommunication is the root cause of nearly 70% of sentinel events in medical practice.[46] Patients have increased nosocomial infections, increased adverse drug events, and higher risk-adjusted mortality due to lack of situational awareness, poor role clarity and leadership training, faulty coordination between medical subspecialists, and incomplete debriefing and handoff procedures.[47] The Salas team eloquently summarizes that simulation represents an opportunity to “practice both task-and team-related skills in a ‘‘consequence-free’’ environment, where errors truly are opportunities for learning and providers receive constructive feedback, focused on improvement, and nonjudgemental.”[46]


References

[1] Devitt JH,Kurrek MM,Cohen MM,Cleave-Hogg D, The validity of performance assessments using simulation. Anesthesiology. 2001 Jul     [PubMed PMID: 11465581]
[2] Richardson WC,Berwick DM,Bisgard JC,Bristow LR,Buck CR,Cassel CK,Coye MJ,Detmer DE,Grossman JH,James B,Lawrence DM,Leape L,Levin A,Robinson-Beale R,Scherger JE,Southam AM,Wakefield M,Warden GL,Corrigan JM, The Institute of Medicine Report on Medical Errors: misunderstanding can do harm. Quality of Health Care in America Committee. MedGenMed : Medscape general medicine. 2000 Sep 19     [PubMed PMID: 11104488]
[3] Durkin ET,McDonald R,Munoz A,Mahvi D, The impact of work hour restrictions on surgical resident education. Journal of surgical education. 2008 Jan-Feb     [PubMed PMID: 18308282]
[4] Grenvik A,Schaefer J, From Resusci-Anne to Sim-Man: the evolution of simulators in medicine. Critical care medicine. 2004 Feb     [PubMed PMID: 15043230]
[5] Denson JS,Abrahamson S, A computer-controlled patient simulator. JAMA. 1969 Apr 21     [PubMed PMID: 5818529]
[6] Gordon MS,Ewy GA,Felner JM,Forker AD,Gessner I,McGuire C,Mayer JW,Patterson D,Sajid A,Waugh RA, Teaching bedside cardiologic examination skills using "Harvey", the cardiology patient simulator. The Medical clinics of North America. 1980 Mar     [PubMed PMID: 6155573]
[7] Gaba DM,DeAnda A, The response of anesthesia trainees to simulated critical incidents. Anesthesia and analgesia. 1989 Apr     [PubMed PMID: 2929977]
[8] Curtis JR,Back AL,Ford DW,Downey L,Shannon SE,Doorenbos AZ,Kross EK,Reinke LF,Feemster LC,Edlund B,Arnold RW,O'Connor K,Engelberg RA, Effect of communication skills training for residents and nurse practitioners on quality of communication with patients with serious illness: a randomized trial. JAMA. 2013 Dec 4     [PubMed PMID: 24302090]
[9] Reznek M,Smith-Coggins R,Howard S,Kiran K,Harter P,Sowb Y,Gaba D,Krummel T, Emergency medicine crisis resource management (EMCRM): pilot study of a simulation-based crisis management course for emergency medicine. Academic emergency medicine : official journal of the Society for Academic Emergency Medicine. 2003 Apr     [PubMed PMID: 12670855]
[10] Härgestam M,Lindkvist M,Jacobsson M,Brulin C,Hultin M, Trauma teams and time to early management during in situ trauma team training. BMJ open. 2016 Jan 29     [PubMed PMID: 26826152]
[11] Capella J,Smith S,Philp A,Putnam T,Gilbert C,Fry W,Harvey E,Wright A,Henderson K,Baker D,Ranson S,Remine S, Teamwork training improves the clinical care of trauma patients. Journal of surgical education. 2010 Nov-Dec     [PubMed PMID: 21156305]
[12] Figueroa MI,Sepanski R,Goldberg SP,Shah S, Improving teamwork, confidence, and collaboration among members of a pediatric cardiovascular intensive care unit multidisciplinary team using simulation-based team training. Pediatric cardiology. 2013 Mar     [PubMed PMID: 22972517]
[13] Wehbe-Janek H,Pliego J,Sheather S,Villamaria F, System-based interprofessional simulation-based training program increases awareness and use of rapid response teams. Joint Commission journal on quality and patient safety. 2014 Jun     [PubMed PMID: 25016676]
[14] Murray DJ,Boulet JR,Kras JF,McAllister JD,Cox TE, A simulation-based acute skills performance assessment for anesthesia training. Anesthesia and analgesia. 2005 Oct     [PubMed PMID: 16192533]
[15] Holmboe E,Rizzolo MA,Sachdeva AK,Rosenberg M,Ziv A, Simulation-based assessment and the regulation of healthcare professionals. Simulation in healthcare : journal of the Society for Simulation in Healthcare. 2011 Aug     [PubMed PMID: 21817863]
[16] Gordon M,Darbyshire D,Baker P, Non-technical skills training to enhance patient safety: a systematic review. Medical education. 2012 Nov     [PubMed PMID: 23078681]
[17] Rudolph J,Raemer D,Shapiro J, We know what they did wrong, but not why: the case for 'frame-based' feedback. The clinical teacher. 2013 Jun     [PubMed PMID: 23656682]
[18] Rudolph JW,Simon R,Rivard P,Dufresne RL,Raemer DB, Debriefing with good judgment: combining rigorous feedback with genuine inquiry. Anesthesiology clinics. 2007 Jun     [PubMed PMID: 17574196]
[19]     [PubMed PMID: 30658946]
[20] Cannon-Bowers JA, Recent advances in scenario-based training for medical education. Current opinion in anaesthesiology. 2008 Dec     [PubMed PMID: 18997530]
[21] Nasca TJ,Philibert I,Brigham T,Flynn TC, The next GME accreditation system--rationale and benefits. The New England journal of medicine. 2012 Mar 15     [PubMed PMID: 22356262]
[22] Fessler HE,Addrizzo-Harris D,Beck JM,Buckley JD,Pastores SM,Piquette CA,Rowley JA,Spevetz A, Entrustable professional activities and curricular milestones for fellowship training in pulmonary and critical care medicine: executive summary from the Multi-Society Working Group. Critical care medicine. 2014 Oct     [PubMed PMID: 25226119]
[23] Salzman DH,Watts H,Williamson K,Sergel M,Dobiesz V,DeGarmo N,Vora S,Sharp LJ,Wang EE,Gisondi MA, A Multicenter Collaboration for Simulation-Based Assessment of ACGME Milestones in Emergency Medicine. Simulation in healthcare : journal of the Society for Simulation in Healthcare. 2018 Oct     [PubMed PMID: 29620703]
[24] Mayo PH,Hackney JE,Mueck JT,Ribaudo V,Schneider RF, Achieving house staff competence in emergency airway management: results of a teaching program using a computerized patient simulator. Critical care medicine. 2004 Dec     [PubMed PMID: 15599146]
[25] Wahidi MM,Silvestri GA,Coakley RD,Ferguson JS,Shepherd RW,Moses L,Conforti J,Que LG,Anstrom KJ,McGuire F,Colt H,Downie GH, A prospective multicenter study of competency metrics and educational interventions in the learning of bronchoscopy among new pulmonary fellows. Chest. 2010 May     [PubMed PMID: 19858234]
[26] Pion GM,Cordray DS, The Burroughs Wellcome Career Award in the Biomedical Sciences: challenges to and prospects for estimating the causal effects of career development programs. Evaluation & the health professions. 2008 Dec     [PubMed PMID: 18838396]
[27] Berkenstadt H,Ziv A,Gafni N,Sidi A, Incorporating simulation-based objective structured clinical examination into the Israeli National Board Examination in Anesthesiology. Anesthesia and analgesia. 2006 Mar     [PubMed PMID: 16492840]
[28] Williams RG,Klamen DA,McGaghie WC, Cognitive, social and environmental sources of bias in clinical performance ratings. Teaching and learning in medicine. 2003 Fall     [PubMed PMID: 14612262]
[29] Scalese RJ,Obeso VT,Issenberg SB, Simulation technology for skills training and competency assessment in medical education. Journal of general internal medicine. 2008 Jan     [PubMed PMID: 18095044]
[30] Downing SM, Reliability: on the reproducibility of assessment data. Medical education. 2004 Sep     [PubMed PMID: 15327684]
[31] Huang GC,McSparron JI,Balk EM,Richards JB,Smith CC,Whelan JS,Newman LR,Smetana GW, Procedural instruction in invasive bedside procedures: a systematic review and meta-analysis of effective teaching approaches. BMJ quality & safety. 2016 Apr     [PubMed PMID: 26543067]
[32] Lighthall GK,Barr J, The use of clinical simulation systems to train critical care physicians. Journal of intensive care medicine. 2007 Sep-Oct     [PubMed PMID: 17895484]
[33] Barsuk JH,Cohen ER,Williams MV,Scher J,Jones SF,Feinglass J,McGaghie WC,O'Hara K,Wayne DB, Simulation-Based Mastery Learning for Thoracentesis Skills Improves Patient Outcomes: A Randomized Trial. Academic medicine : journal of the Association of American Medical Colleges. 2018 May     [PubMed PMID: 29068818]
[34] Barsuk JH,Cohen ER,Mikolajczak A,Seburn S,Slade M,Wayne DB, Simulation-Based Mastery Learning Improves Central Line Maintenance Skills of ICU Nurses. The Journal of nursing administration. 2015 Oct     [PubMed PMID: 26425976]
[35]     [PubMed PMID: 22762931]
[36] Singer BD,Corbridge TC,Schroedl CJ,Wilcox JE,Cohen ER,McGaghie WC,Wayne DB, First-year residents outperform third-year residents after simulation-based education in critical care medicine. Simulation in healthcare : journal of the Society for Simulation in Healthcare. 2013 Apr     [PubMed PMID: 23222546]
[37] Back AL,Arnold RM,Baile WF,Fryer-Edwards KA,Alexander SC,Barley GE,Gooley TA,Tulsky JA, Efficacy of communication skills training for giving bad news and discussing transitions to palliative care. Archives of internal medicine. 2007 Mar 12     [PubMed PMID: 17353492]
[38] Arnold RM,Back AL,Barnato AE,Prendergast TJ,Emlet LL,Karpov I,White PH,Nelson JE, The Critical Care Communication project: improving fellows' communication skills. Journal of critical care. 2015 Apr     [PubMed PMID: 25535029]
[39] Calhoun AW,Boone MC,Porter MB,Miller KH, Using simulation to address hierarchy-related errors in medical practice. The Permanente journal. 2014 Spring     [PubMed PMID: 24867545]
[40] Calhoun AW,Nadkarni V,Venegas-Borsellino C,White ML,Kurrek M, Concepts for the Simulation Community: Development of the International Simulation Data Registry. Simulation in healthcare : journal of the Society for Simulation in Healthcare. 2018 Dec     [PubMed PMID: 29672467]
[41] Beal MD,Kinnear J,Anderson CR,Martin TD,Wamboldt R,Hooper L, The Effectiveness of Medical Simulation in Teaching Medical Students Critical Care Medicine: A Systematic Review and Meta-Analysis. Simulation in healthcare : journal of the Society for Simulation in Healthcare. 2017 Apr;     [PubMed PMID: 28704288]
[42] Wagner M,Hauser K,Cardona F,Schmölzer GM,Berger A,Olischar M,Werther T, Implementation and Evaluation of Training for Ultrasound-Guided Vascular Access to Small Vessels Using a Low-Cost Cadaver Model. Pediatric critical care medicine : a journal of the Society of Critical Care Medicine and the World Federation of Pediatric Intensive and Critical Care Societies. 2018 Nov;     [PubMed PMID: 30234738]
[43] Crowe S,Ewart L,Derman S, The impact of simulation based education on nursing confidence, knowledge and patient outcomes on general medicine units. Nurse education in practice. 2018 Mar;     [PubMed PMID: 29190590]
[44] Colman N,Doughty C,Arnold J,Stone K,Reid J,Dalpiaz A,Hebbar KB, Simulation-based clinical systems testing for healthcare spaces: from intake through implementation. Advances in simulation (London, England). 2019;     [PubMed PMID: 31388455]
[45] Houzé-Cerfon CH,Boet S,Marhar F,Saint-Jean M,Geeraerts T, [Simulation-based interprofessional education for critical care teams: Concept, implementation and assessment]. Presse medicale (Paris, France : 1983). 2019 Aug 2;     [PubMed PMID: 31383383]
[46] Salas E,DiazGranados D,Weaver SJ,King H, Does team training work? Principles for health care. Academic emergency medicine : official journal of the Society for Academic Emergency Medicine. 2008 Nov     [PubMed PMID: 18828828]
[47] Ross AJ,Anderson JE,Kodate N,Thomas L,Thompson K,Thomas B,Key S,Jensen H,Schiff R,Jaye P, Simulation training for improving the quality of care for older people: an independent evaluation of an innovative programme for inter-professional education. BMJ quality & safety. 2013 Jun     [PubMed PMID: 23220568]