Why More Operation-based Simulation Exercises should be Unannounced

Simulation exercises are becoming more important in the face of a highly dynamic and fast-changing Global Health Security space with system resilience and multi-sectoral collaboration increasingly essential parts of what needs to be tested. When using simulation exercises in Health Security to prepare for events, the goal is to test capacity to respond — the quality of services during response, how operable the plans and Standard Operating Protocols in place are and whether there are sufficient resources to mount an effective response. This is the first in a series of lessons learned designing and participating in more than a few dozen simulation exercises in the global health security space.

In this piece I outline why operation-based Simulation exercises in public health should, as much as is feasible, happen unannounced.

Only operation-based exercises (see endnotes) can be unannounced and the majority of them conducted are indeed done with some form of advanced notice given to the participants of the exercise. No clear categorical distinction is made between announced and unannounced exercises. This is because in reality most exercises are announced ahead of time. This is not deliberate and happens because the planning for large-scale operational simulation exercises often involve so many moving parts that one way or another the participants get advance notice and are appropriately prepared. This doesn’t seem to bother most simulation exercise experts in public health today. In fact, published commentary on simulation exercises often seems preoccupied with the panic that unannounced exercises can create and warn that stakeholders and the public be appropriately informed of the exercises ahead of time. Fallah and Tantum (see endnotes), writing in The Conversation, concluded that we don’t know if a recent South African Simulation Exercise was successful because ‘lessons about emergency preparedness were overshadowed by panic’. I think this position is shortsighted. The damage public panic can cause is a real danger. However, this is a bug an exercise designer should design for (mitigate the impact of etc.) rather than embrace as a feature to design around.

When testing protocols to be deployed during emergencies, allowing the participants deploying these protocols to prepare ahead undermines the ability to test how people will react, how the protocols will perform and how both of these will interact with existing systems under real-life conditions. This is because;

• Participants, feeling they are the ones under evaluation, often overcompensate for gaps in protocols and procedures being tested. Regardless of how often they are told they are not the one under evaluation, participants often feel under pressure not to misinterpret the protocols or let down their superiors. Because they are aware of the special conditions of the exercise a lot of allowances is granted to processes that may be imperfect (and therefore require small but vital tweaks) — participants lean in more heavily than they should to ensure things work. If the participants are unaware they are under observation, they are more likely to flag errors in processes and try to address them. A group of responders in a recent (2017) simulation exercise took this to the extreme by (quite unnecessarily and comically) donning PPEs while anticipating the arrival of an ill person for an exercise. Even for the participants trying to disregard the prompt and go about their day as though nothing out of the ordinary was going to happen, there are small imperceptible ways in which they lean heavily into the protocols and overcompensate.
• Unannounced exercises provoke staff errors and atypical staff attitude that may not be observed during announced exercises. These provide key insights to the operationalization of Plans and Protocols. When designing Standard Operating Procedures and measures, particularly those requiring a multi-sectoral response, it is difficult to anticipate all the ways in which real-time operationalization may be affected by existing rules and systems and staff-related factors like workload, self-efficacy (things SOP designers often heavily discount). Exercises provide you with additional windows of insight into the way staff and systems interact with protocols and measures. A good example of this is a very recent public health event in which designated health facilities rejected a referred ill person (for reasons related more to self-efficacy than to the availability of supplies and/or infrastructure). During an announced exercise this would not have come to light.
• In my experience, a lot of what goes wrong during emergencies involve external/extenuating factors and not necessarily a gross disregard of procedures or protocols. A sudden loss/unavailability of critical staff, a slight or not so slight delay, a sudden, unanticipated depletion of supplies etc. may introduce significant inefficiencies in time and effectiveness of the response. The ability of Plans, Protocols, measures and stakeholders to anticipate and display resilience against these factors is a critical part of tests (that are often underemphasized). If participants are able to prepare and therefore mitigate these factors, then these exercises would have failed to achieve one of its key objectives — optimize protocols and plans to adapt to the existing systems in a dynamic environment.
• Unannounced exercises may test more protocols or plans than it sets out to, especially where the exercise is happening in a non-public health setting, like stadiums, seaports, and airports etc. During a recent (2018) unannounced simulation exercise in Abuja, an incident with an observer/evaluator at one of the sites of the event triggered a security protocol (involving the Police, the State Security Service and Airport Security) to deal with a breach that had implications for national security. Naturally, this was baffling for the security agents and difficult for the observer, but a debrief of the event helped optimize both Point of Entry public health emergency response protocol and aviation security protocols. This and other examples have demonstrated how easily new, unplanned injects may introduce unintended exercise of procedures. It will be a mistake to view this as a bug. If an unannounced simulation exercise leads to panic in a facility or a city, seeing this as a failure of the exercise is, in fact, a failure of imagination.

Planning complex unannounced exercises can be hard, especially if the number of participating agencies are more than a few, if aresponse from highly regimented sectors (like security and law enforcement) is required, or if responses can easily spark viral (and disruptive) public and media response. The disruption and chaos they cause can also be stressful for staff and participants. But a recent study led by Debra Freund (see endnotes) found no significant difference in perceived level of stress to staff whether the (in situ) simulation exercise was announced or unannounced.

The first step in designing simulation exercises in public health settings should be determining what will be appropriate considering the objectives — an announced or an unannounced exercise. The second step will be designing the how of executing an unannounced exercise or at the minimum introducing elements of surprise through injects into the scenarios while causing as little business disruption as possible.

ENDNOTES

1. Global Health Security is the domain of Global/Public Health work that ensure epidemics do not spread in a country or across borders. How this works is better defined or explained by the implementation and monitoring frameworks designed for it, including the USA-led Global Health Security Agenda and the WHO’s Joint External Evaluation. This piece speaks specifically to simulation exercises for outbreaks and public health emergency preparedness and response.
2. There is a general convention on the types of simulation exercises. The WHO in its Simulation Exercise Manual makes a distinction between Discussion-based Exercises (mainly Tabletops) and Operation-based exercises including Drills (limited, procedure-specific), Functional, Field and Full Scale exercise.
3. See Mosoka Fallah, Lucy Tantum, “Is it Ebola, or just a drill? How to test a public health crisis response”, in The Conversation, which concluded bizarrely that we don’t know if a recent South African Simulation Exercise was successful because ‘lessons about emergency preparedness were overshadowed by panic’. The only thing I can say here to this is that that is not at all how the evaluation of simulation exercises work. Or should work.
4. Freund, D, Andersen, PO, Svane, C, Meyhoff, CS, Sørensen, JL. Unannounced vs announced in situ simulation of emergency teams: Feasibility and staff perception of stress and learning. Acta Anaesthesiol Scand. 2019; 63: 684– 692. https://doi.org/10.1111/aas.13321