With the goal of reducing cyber risk and by supporting effective incident response, heretofore in our series on Preparedness, we have explored the different components of the Preparedness Cycle — the continuous cycle of planning, organizing, training, equipping, and exercising. In this second to last post in the series, we’ll briefly look at the last two parts: evaluating and taking corrective actions. For common understanding, let’s start with two exercise-specific definitions via the FEMA Preparedness Toolkit:
- Evaluation: “Exercise evaluation is the cornerstone of an exercise and maintains the functional link between exercise and improvement planning. Through exercise evaluation, organizations assess the capabilities needed to accomplish a mission, function, or objective. Effective exercise evaluation involves planning for exercise evaluation, observing and collecting data during exercise conduct, analyzing data, and reporting exercise outcomes.”
- Improvement Planning: “Exercises afford organizations the opportunity to evaluate capabilities and assess progress toward meeting capability targets in a controlled, low-risk setting. An effective corrective action program develops improvement plans that are dynamic documents, with corrective actions continually monitored and implemented as part of improving preparedness.”
For those desiring a “deep dive” into exercise evaluation and improvement planning, review the guidance in the 2013 Homeland Security Exercise and Evaluation Program (HSEEP). That will provide details on the process of developing and conducting evaluation and improvement planning and documentation, addressing ideas such as Exercise Evaluation Guides (EEGs), data collection, after action reporting, and developing an improvement plan and corrective action program. Below, I’d like to share a few ideas for additional consideration.
Do What Works
The HSEEP guidance above provides specific approaches that work. Using well-established standards like Core Capabilities and EEGs provide common terms and references, and help promote consistency in evaluations and documentation. All good! However, not every exercise is resourced (nor really requires) the complete HSEEP approach. HSEEP is guidance and should be treated as exactly that. If you want to irritate an exercise pro, tell them you want an “HSEEP-compliant exercise” and watch their eyes roll into the deepest parts of their skull … What is critical is that you plan for evaluations hand-in-hand with training and exercises and that you have a deliberate approach. Your organization may have some specific ways you like to capture and report information or you may need to be mindful of certain sensitivities. More often, you have to contend with being under-resourced and need to manage the best evaluation you can with what you have available in both people and time. What is most important is that you know what you have available, deliberately plan as part of the training and exercise development process, and ensure evaluation does occur and is documented. If you do that, however exactly you have to do it, you’re doing pretty well!
Get Buy-In
As noted in our mini-series on exercises, exercises tend to get the most attention. Exercises are fun! — evaluations are much more boring, and can be contentious, and frustrating… Getting buy-in early and from the right people can save planners (particularly junior personnel) a lot of grief and greatly help support an effective and value-added evaluation. We want to gain buy-in into our approach to the evaluation, as well as to the activities supporting the evaluation and improvement planning. So, who do we need buy-in from? Well, ideally, everyone. But given we can’t court every leader and participant, it is good to try and ensure that your exercise sponsor is on-board, as well as those that will help conduct the evaluation. For events like After Action Meetings (AAM; again, refer to HSEEP guidance for details), know who some of the key players and influencers are and work with them to help them understand what you’re doing, where it’s going, and to get their support for the process and your efforts. And know who you’re going to be putting some focus on and get ahead of potential tensions and flare-ups — but engage them privately before doing so publicly. If you’re about to go into an AAM and know that a certain organization or department is about to hear some things they won’t like, talk to them ahead of time (which hopefully you’ve done in developing the evaluation) and agree to how you may approach some of the more difficult areas. They may still not like your approach, but by engaging them, you may get more support, or at least less objection (and sometimes you won’t, and it might get ugly…). In both developing the evaluation process and in conducting the evaluation and after action activities, building support and getting others to invest in what you’re doing can grease the process and make it a lot more successful.
Seek Continuous Improvement
One of my favorite books is the classic Animal Farm and like Boxer, the hardworking but rather dim horse in that story, my typical approach to things is to put my head down, block out the noise and tell myself, “I will work harder.” After many years of ugly running and punishing my Achilles, I started cycling about a year ago. Applying my usual approach, I try to muscle through every challenge, which has some utility. But, when I take the time to look at my stats, assess parts of the ride and how I tackled them, compare with previous workouts, and otherwise assess and evaluate my performance, I’m able to better understand how I did and how I can improve. My goal is to keep getting better. In Animal Farm, Boxer’s valiant efforts end in the care of the “Horse Slaughterer and Glue Boiler,” and I’d prefer a smarter, more positive outcome. By properly planning and preparing for my ride evaluation, taking the appropriate amount of time to review, assess, and evaluate my performance, I am able to work towards continuous improvement and hopefully reaching the desired level of physical fitness. Hopefully… The same approach should be applied towards exercises and preparedness broadly. Develop a multi-year plan (as discussed in previous posts in this series), establish goals and milestones, plan but be flexible, and seek to continuously improve the readiness and resilience of your organization through effective evaluation, corrections, and improvement planning.
With this post, we’ve worked our way through the Preparedness Cycle! In the concluding segment to this series, I’ll talk to Jeff Peters as we conclude this series on the Preparedness Cycle, some common issues, best practices, and more.
Share this:
SurfWatch Labs, Inc. 
2 thoughts on “Preparedness & Cyber Risk Reduction Part Six: Evaluate & Improve”