Those motivational posters with the pithy quotes at the bottom are great. You know, the ones espousing the value of "Perseverance" or "Teamwork." I have an idea for one of my own that would show a monk meditating in the courtyard of a burning city. It would read, "Grace Under Pressure."
It's amazing that monks can stay calm and "in the zone" at all times under any circumstance. But they don't get there overnight; it takes practice, years and years of dedicated practice. It also takes discipline, and that is what really separates the enlightened from the pretenders.
Of course, by now you're wondering, what does this have to do with security? The reality of being a security professional is that being compromised goes with the territory. Maybe not today, maybe not tomorrow, but at some point, a compromise will happen, be it the result of a flawed Web application or a malicious insider, and most practitioners will not be ready. Instead of relying on a pre-established, rock-solid incident response plan, the security manager will panic, bungle the response and eventually start looking for a new job, because it's almost impossible to recover one's reputation internally once a major incident occurs and isn't handled properly.
But it doesn't have to be this way. It is possible to be calm in the face of a towering inferno as long as the incident response plan is well-defined and ensures that organizational data remains safe, while containing any damage. There are a number
Step 1: Write down the plan
A lot of practitioners have an incident response plan, but it resides in their heads. That is not good enough. Practicing the steps while on the treadmill? Not good enough either. The plan needs to be communicated more broadly than the family dinner table, and that means writing it down, sharing it with colleagues, revising it and then making sure all affected parties know about it.
The document should lay out exactly what will happen in the event of an incident. It needs to specify accountabilities, escalations, and if/when to bring law enforcement into the picture. If it's not written down, it might as well not exist, because not only could something happen to prevent you from being in position to respond during an incident, but a good incident response plan is also one that all parties are familiar with long before an incident occurs.
Step 2: Get buy-in
Once the plan is written down, it needs to be circulated amongst the organization's internal IT power brokers to make sure that everyone understands the document the same way and knows their responsibilities. Of course, that includes the CIO, since an incident may involve taking systems and/or networks out of service. Appropriate response tactics need to be discussed and accepted before the incident, or be ready to suffer the consequences of mismanaged expectations.
Don't forget about other members of the senior team as well, especially human resources and the general counsel. In the event the incident results in a legal prosecution, both of those individuals will be able to determine exactly what data needs to be gathered and how the response and other personnel action need to play out.
Step 3: Understand escalation
I mentioned escalation above, but it's important to get a little more specific. The first thing to determine is the notification process. Who gets called and when? Under what circumstances is the CIO, CFO, CEO, etc. brought into the situation? When should they be awakened in the middle of the night, if ever? The time to find out that the CEO doesn't care isn't at 3 a.m. when she's chewing your ear off for disturbing her beauty sleep.
It's also important to make sure that the level of acceptable damage control is specified. If a key customer-facing application may be compromised, is an out-of-band investigation conducted, or is the application taken down? Who should be making that decision? Odds are it's not the security professional, but having someone accessible at all times to make those kinds of calls is absolutely critical.
Step 4: Practice, practice and then practice some more
When was the last time your enterprise conducted an incident simulation? Most organizations answer that question in years, if ever, and that's a problem. The fact is every organization has a dynamic workforce, set of trading partners, application base and lots of other things changing at all times. That means the incident response plan needs to be a truly living document. It needs to change when the business changes and those changes need to be reinforced though practice.
I know that it may be a challenge to get the troops motivated about incident response simulations when it's far from the real thing, but it's important. Major mistakes can be made or time lost when an enterprise fails to realize there's a gaping hole in the plan until an incident occurs. Remember, optimally a lot of the responses under duress are conditioned responses, and they only get that way through practice and repetition.
Step 5: Learn from mistakes
Even the best, most field-tested security professionals mess things up, especially in the middle of an incident. That's why besides containing the damage in real time, doing a detailed post-mortem is the most important aspect of the process. Everyone makes mistakes, but hopefully not the same mistakes more than once.
Unless a concerted effort is made to understand the nature of the incident, what went wrong and what will be changed moving forward to ensure it doesn't happen again, history will repeat.
So, swallow a bit of pride, dig deep into the incident and make whatever process changes are required to ensure the same incident doesn't happen again. This type of post-mortem is also invaluable during an audit, so show the auditor how the organization recovered from an incident and learned from it.
Remember, the difference between someone that is perceived to be a hero in a tough situation and a goat looking for his or her next job is all about how the incident is handled. Follow these five steps and security professionals can live to fight another day.
About the author:
Mike Rothman is president and principal analyst of Security Incite, an industry analyst firm in Atlanta, and the author of The Pragmatic CSO: 12 Steps to Being a Security Master. Rothman is also SearchSecurity.com's expert-in-residence on information security management. Get more information about the Pragmatic CSO at http://www.pragmaticcso.com, read his blog at http://blog.securityincite.com, or reach him via e-mail at mike.rothman (at) securityincite (dot) com.
This was first published in March 2008