Stable systems leave us unprepared for incidents

Many years ago I worked on the shop floor of a national retailer. When the tills failed for one reason or another, there was a manual process that had to be quickly rolled out. Out came the pocket calculators, hand-written receipts and manual credit-card imprinters. At the time, this was not an uncommon occurrence and all the staff consequently knew what they had to do. The process took a bit longer but we were quite sleek at keeping the traffic moving through the shop, even the time it happened the Saturday before Christmas

Nearly 20 years on and I'm not sure that this would necessarily still be the case. As the IT supporting these services becomes more stable, the instances of outages happen less often and there is less working knowledge of what needs to be done when a failure occurs. Only through training and practice can businesses be sure that their staff know what to do in the event of an incident. Without this, organisations risk losing business due to not being able to sell their goods and services at the time when people want to buy them. The expectations of customers to be able to buy what they want when they want to and be processed as fast as possible are certainly far greater now than they were in the early nineties, and there are more alternative options now for them to make their purchase.

It was an article in The Register which made me consider this as a topic to cover. Although not a recent finding, the article comments on the outcome of the investigation into the crash of Air France flight 447 in 2009 which concluded that after a failure of the autopilot, the pilots did not have sufficient skills and experience to fly the plane manually. This issue resulted in the fight plunging into the Atlantic ocean with the tragic loss of all 228 people on board. The report highlights that as pilots become so dependent on the autopilot, using it for many of the tasks in the flight, that when it is suddenly and unexpectedly not available to them that skills to pilot a plane the "old fashioned" way, may be somewhat rusty.

This highlights the importance of incident training and business continuity exercising. A business continuity event or crisis is something that no business wants to think will happen to it but as I've mentioned in previous posts, there are many external and uncontrollable factors that can introduce this scenario. Don't just test IT failover or run the generators... Test and exercise the people who will be expected to take the reigns, assume "manual control" and make difficult decisions in a short time-frame that may ultimately save costs, reputation and in many cases... lives.

Image: bk images / FreeDigitalPhotos.net

2012: Thinking beyond The Olympics

The London Olympics next year is very high in the public conscious and will continue to be so. The increase in traffic and the number of people using many services including public transport means that those businesses in central London are already gearing up for potential disruption and putting plans in place to reduce the impact of the event. This is certainly a good move for companies in the middle of town but what about the rest of us? If you don't have a significant central London presence then you're probably not too concerned. However, here are some other factors to consider:

Geography: other sites and transport links

It's probably fair to say that central London is going to be worst affected in terms of traffic but there are a number of locations outside of the M25 which will also be holding events. Consideration needs to be given to where spectators will be travelling from and by what means. Spectators from other countries may not spend their entire time at the games, so tourist areas around the country, particularly in the South East will see an increase in visitors. If you're doing business in areas of interest to tourists in general, have you considered the potential impact?

Timeline: when does your plan start and end?

What timeframe do you use when planning for any disruption? Do you start and end at the opening and closing ceremonies or consider inclusion of related events such as the Olympic torch relay and the Paralympics? From the torch relay arriving in the UK on 18th May to the closing ceremony of the Paralympics on 9th September is over 100 days. If you've only planned for three weeks then maybe it's time to rethink?

Suppliers and Customers: secondary and tertiary elements

Your business may not be based in central London or even have a significant presence in and around the capital. What about your suppliers and other third parties on which you rely? Having dealt with the primary consideration of your own business, you then need to consider the secondary impact of your suppliers not being able to fulfil their obligations to you. I've discussed previously about the issues of companies relying on trains and the tube to get their staff into work. These are third parties over which you have no control and no agreed levels of service. It is important to make sure therefore, that those third parties with which you do hold such contracts have considered how they are going the continue to provide the required levels of service throughout the games. The tertiary elements are the suppliers to your suppliers. For key third-parties, it's one thing to have them guarantee a level of service to you but can you be sure that they have undergone the same level of due diligence for their suppliers? The secondary consideration is ensuring your suppliers can still provide services to you, the tertiary consideration is ensuring that they are also considering the same risks to their businesses.

Your customers are the other group with whom you have contracts and to whom you have committed an agreed level of service. As well as making appropriate plans during 2012 for your own benefit, customers will want to see that you are also considering the continued provision of service to them. Where that service relies on infrastructure that will be under increasing demand and pressure during the Olympics, the basis of this requirement is well-founded.

Staff attendance: has everyone got tickets?

How many of your employees successfully obtained tickets for an Olympic event? How many have tickets for the same event? You may not get an idea until the opportunity to book time-off for 2012 comes around, but even then you will have people who wait until closer to the time to book their holiday even though they've had their tickets booked and the dates known for over a year.

Minimum notification periods, the ability of management to reject holiday requests and the threat of disciplinary action for taking time off without approval may not seem as important to some individuals as the seemingly once in a lifetime opportunity to be at the London Olympics. You may know how many staff have requested leave on a particular day but how can you be sure until the day arrives and people turn up (or not)? Being ready for staff shortages during key events is important. You may feel comfortable in the knowledge you can discipline or even dismiss those who deliberately do not turn up for critical duties, but that doesn't help you on the day.

Other events in 2012

The olympics, in isolation, is going to be disruptive enough but don't forget all the regular and special events that happen throughout London that might just add an additional level of complexity and concern to an already busy summer. The Queen's Diamond Jubilee earns us an extra bank holiday in June and will include a number of events around the capital and Wimbledon attracts plenty of crowds and will do the same again across June and July. The Notting Hill carnival promises to once again be bigger and better in 2012 than previous years and there are plenty of festivals and other events across the capital which will help contribute to the mayhem.

Now is the time to act!

Don't leave it until the last minute to prepare yourself for next summer. Act now to make sure you're ready for possible disruptions:

- Think about which of your suppliers may be affected by disruptions
- Consider which critical suppliers you want to approach to discuss their plans to deal with how any disruptions may affect their supply chain.
- Consider how disruptions in London may affect your customers
- Find out who in your company is planning to take time off, well in advance.
- Consider the timeframe you want to plan for. How does this match up to critical times in your own business processes?

Photo: xedos4

Availability: More than technical resilience

It's easy to let the definition of information security controls become biased in terms of just the IT and not the data or wider assets. The IT, although an asset in itself, is very much a supporting tool to store, process and transmit the data assets. This includes your data and your customers' data and should be classified according to it's value and the impact of loss, unauthorised access or change. Data will include general company confidential data, intellectual property, personally identifiable information or data which if exposed may harm the competitive advantages of the business. Confidentiality is always a key concern for data and availability is typically left to the technical resilience of the IT systems in which the data is contained.

It's only when you look at assets other than the data and consider how to protect those and how they interact with the data that you need to think about more than IT. Other assets include people (staff & contractors), buildings, supporting services/utilities and the reputation of the company. Reputation is always a difficult one when you need to quantify the level of impact of any event, but you can certainly think about the type of events you want to mitigate against to keep customers happy and ensure potential customers want to do business with you. How you deal with the risks against other assets will in turn help to protect this one.

Having systems backed up or having a failover system is all well and good but how do people access that data if their normal routine is disrupted or if they cannot physically access the location where they normally connect to the systems and are not aware of alternative options.

IT failings are only an element of business continuity events. Common causes include business locations being inaccessible or staff journeys being impacted, predominantly for reasons outside of the control of the organisation, such as bad weather. Whenever it snows in the UK, the country tends to grind to a halt, either through inaccessible transport routes or through staff not wanting to risk travelling or not knowing how to handle the conditions properly. Every time we get more than an inch of snow lots of people will ask why we weren't more prepared and why we didn't learn from last time. Someone will typically relay a story about how the last time they flew to Calgary (or similar location) there were fleets of snow ploughs constantly clearing runways and ask why don't we have the same at Heathrow? For most observers, the answer is obvious, Heathrow only gets disrupted a handful of times a year by snow and typically recovers quickly. Calgary however is somewhere people go because they have so much snow and therefore they are far more likely to invest in controls and infrastructure to keep planes landing in all conditions. Where snow is something of an irregular irritation for Heathrow it is an indication of the prime business environment for Calgary, so well worth the investment in the fleet of Ploughs (plus sweepers, blowers and melters).

Industrial action is another continuity consideration. We hear how a day of strikes have caused "so-many-millions of pounds of lost business", well this shouldn't need to be the case. If the only way for staff at a company to continue working is to jump on a train or tube and go to a specific building then that's a significant requirement you are putting onto another organisation with which you have no contracted service levels. If a strike takes out this one and only access mechanism and the only backup options are full to capacity from other companies' staff impacted by the same event, then you leave yourself unprepared.

A Business Impact Analysis (BIA) of all elements of the business will help assess each activity and how the loss of it for different periods of time might impact the business as a whole. Typically the technical roles within the company will know what they need to do if the requirement comes to connect remotely and more often do this on a regular basis. It's the functions that are often considered to be back-office that are perhaps less ready for a continuity event. Functions like procurement, billing and payroll are traditionally office-based activities, working from desktop PCs with data maintained locally. Although a BIA might indicate a minimum disruption to the business if these functions were not able to work for a day, the requirements for a company to buy goods and services, bill their customers and pay their staff become more critical for longer more protracted outages. Suddenly, without these functions, other measures that organisations have in place for resilience can be affected. Automation and remote management capabilities for these functions are all well and good but if they are only going to be used in an emergency, how do you make sure that the people in question know what to do and when?

Exercising in Business Continuity is just as important as any other availability control. There's no point in implementing continuity measures if nobody knows that they are there or what they need to do with them. Any business will hope that it never needs to use its continuity measures and the chances are that any event may happen long after the measures are implemented. Keeping the measures up-to-date and making sure they technically work is one part of testing. Exercising the people is another. These tests should be defined as part of a wider crisis management plan to test against multiple scenarios.

In conclusion, technical resilience is only a small part of ensuring the availability of services and data. An understanding of the criticality of each business area and the impact the loss of any of them for varying periods of time should be understood. Once controls are implemented to reduce the risk or impact of different events, both the controls themselves and the people required to operate them in the event that they are required.

Photo: think4photop