From his STAT59 blog, Statistical Consultant Jeffrey Franc explains how to design a perfect simulation research study
As a statistical advisor, I am often asked to assist in the development of simulation-based research studies. Usually, the researcher is very excited about a new simulation they have developed, and wants to do a research study to show how awesome it is. However, simulation research studies have some peculiar requirements, and getting it right can take some careful planning to avoid wasting time and effort.
Let’s discuss the (fictitious) student Ariel Wang. Ariel is a PhD student working in Critical Care Medicine. She has been working for several years developing a high-fidelity simulation that her department has found very useful in teaching resident physicians about the use of various types of airway strategies. In particular, she finds the simulation is excellent for teaching the advanced skills of video-laryngoscopy in crisis situations. Ariel would like to conduct a research study on her simulation.
Ariel is unsure how to design simulation research. Her initial plan is to take a group of residents, expose them to her simulation, and then have them write a multiple-choice test about how to manage a difficult airway. She hopes to prove that her simulation is a good way to teach advanced airway skills.
In various similar forms, this is a common first idea for many researchers in simulation. However interesting this initial plan, it is not strong as a research protocol. It is vague and contains a number of biases. Ariel needs some help to develop her initial idea into the perfect simulation research study.
To start, Ariel will download the STAT59 Research Study Design Canvas and she will follow six simple steps to transform her vague idea to the perfect simulation research study.
1. Focus on a Single Validated Outcome
The best studies have a single very clear objective. Ask yourself, “what are you trying to prove?
To improve her study, Ariel should focus on a single validated outcome. Ariel consults a professional librarian to help her find a published and validated test to evaluate simulation performance. She finds the Ottawa GRS [c’è anche una versione italiana] and decides to use this as her scoring tool. Her final study outcome will be the difference between GRS scores attained by the two groups during a simulated airway crisis. She will compare groups that have received the standard residency training in intubation on patients in the operating room (control), versus those who have the standard training and additional simulation training (intervention).
2. Write for a Specific Audience
Think carefully about your audience by asking yourself: “who cares?”
Ariel feels her study will be particularly relevant to educators in Anesthesia, Critical Care, and Emergency Medicine as well as residency training programs in smaller hospitals, or in areas with a shortage of experienced teachers. Next, she chooses a target journal for publication. By looking at the journal’s Instructions for Authors section on the website, she can design her study from the outset to meet the journal’s criteria. This strategy avoids any last-minute surprises at the submission stage.
3. Study a Concept: Not Your Simulator
What is the value proposition for your study? What value does your study have for others?
To improve her study, Ariel decides that she will want to emphasize that simulation training in addition to standard training will give superior skills in crisis resource management.
4. Get a Librarian to Help
Unless you are a professional librarian, seek help. A good literature search can make-orbreak your study.
Through a careful literature search conducted by the librarian at her University, Ariel finds that other researchers have studied the use of simulation to train crisis resource skills in different scenarios. She finds good evidence that this concept is true in many areas, but has not been extensively studied in intubation skills. These are the articles she will cite when she writes the introduction and discussion of her paper.
5. Plan In Advance for a High-Quality Analysis
Never collect any data until you have set down the analysis plan. Get help if you are unsure.
Ariel’s initial study plan, a multiple choice test after exposure to the simulation, suffered from many biases. She decides, then, to perform a randomized controlled study. She will use a proven randomization procedure to create two groups of students: one group will receive a day of training on the simulator, and another will receive a day of training in the operating room. She will then test each group’s performance during a simulated crisis by evaluating their team performance using a validated assessment tool: the Ottawa GRS. She reviews her plan with the statistician who will be doing the data analysis before collecting any data.
6. Get Consent (Yes, You Need to Get Consent)
Ensure you have access to the data. Get consent from participants. Get ethical approval
Having access to data for teaching purposes, or even for teaching quality control, does not mean Ariel can use the data for research. In fact, she will need to obtain consent from the students – before they complete the simulation exercises – to be part of her research study. In addition, it requires departmental ethical approval.
