Stable systems leave us unprepared for incidents

Many years ago I worked on the shop floor of a national retailer. When the tills failed for one reason or another, there was a manual process that had to be quickly rolled out. Out came the pocket calculators, hand-written receipts and manual credit-card imprinters. At the time, this was not an uncommon occurrence and all the staff consequently knew what they had to do. The process took a bit longer but we were quite sleek at keeping the traffic moving through the shop, even the time it happened the Saturday before Christmas

Nearly 20 years on and I'm not sure that this would necessarily still be the case. As the IT supporting these services becomes more stable, the instances of outages happen less often and there is less working knowledge of what needs to be done when a failure occurs. Only through training and practice can businesses be sure that their staff know what to do in the event of an incident. Without this, organisations risk losing business due to not being able to sell their goods and services at the time when people want to buy them. The expectations of customers to be able to buy what they want when they want to and be processed as fast as possible are certainly far greater now than they were in the early nineties, and there are more alternative options now for them to make their purchase.

It was an article in The Register which made me consider this as a topic to cover. Although not a recent finding, the article comments on the outcome of the investigation into the crash of Air France flight 447 in 2009 which concluded that after a failure of the autopilot, the pilots did not have sufficient skills and experience to fly the plane manually. This issue resulted in the fight plunging into the Atlantic ocean with the tragic loss of all 228 people on board. The report highlights that as pilots become so dependent on the autopilot, using it for many of the tasks in the flight, that when it is suddenly and unexpectedly not available to them that skills to pilot a plane the "old fashioned" way, may be somewhat rusty.

This highlights the importance of incident training and business continuity exercising. A business continuity event or crisis is something that no business wants to think will happen to it but as I've mentioned in previous posts, there are many external and uncontrollable factors that can introduce this scenario. Don't just test IT failover or run the generators... Test and exercise the people who will be expected to take the reigns, assume "manual control" and make difficult decisions in a short time-frame that may ultimately save costs, reputation and in many cases... lives.

Image: bk images / FreeDigitalPhotos.net

What is the scope of Information Security?

Trying to define the scope of the information security organisation and ensure that the appropriate elements are included is still given much consideration. My view on this is to go back to basics and look at what you are trying to achieve with this function. Information Security is concerned with the Security of Information (shocker!) and in order to achieve this you need to consider what information is; in what forms it is stored, processed and transmitted; and by whom, which mechanism or within what type of environment/container. These factors allow you to assess the vulnerabilities of information in the various formats and situations, what threats are present and therefore what the risk is to that information... this is probably sounding familiar.

When considering the risks around information in IT systems, this includes any piece of IT in your environment from the enterprise level systems management environment to the USB stick on a keyring. These systems will hold information or be capable of holding information which should be classified by the organisation according to its sensitivity. Even devices which don't have an end-user storage capability such as network devices hold device configurations, the disclosure of which to unauthorised parties might compromise the desired obscurity of the network configuration or the intellectual property of the organisation who defined the configuration in the first place.

The "wider than just IT" remit

A key considerations needs to be information in other, non-IT forms: Contracts, financial documentation and HR records on paper are key information assets to the organisation and should be included in any risk assessment. People are the other element for consideration. Who is going to access your information: Employees? contractors? customers? third-parties? Once information has been read then it is transferred to a much more unpredictable storage medium, the human brain. Information that was once in a secure environment within the corporate building becomes "you'll never guess what I've just read about!" on a mobile phone on a train. The reputation of your organisation is therefore an asset at risk. Once in the hands of people, your control over your information is reduced and other controls both IT and non-IT are required to protect the data, from encryption on portable media to awareness training.

In considering the threats and vulnerabilities of information, you need to consider how you manage the information in a number of circumstances, how you control the information assets in your company and how you ensure that your employees, contractors, customers and third-parties are trustworthy enough to have the information, are trained in the importance of information protection and have contractual or legal measures to protect the company from any unauthorised disclosure. You also need to consider the physical environment in which the information is held. It's all very well putting amazing IT controls in place if someone can walk into an office or datacentre and simply stroll away with it.

The ability to respond to adverse events will also act as a key control to reduce risk. The more prepared you are, the faster you can react to any event which threatens the security of information, the more you can limit any damage and the faster you can recover. Incident Management and Business Continuity therefore go hand in hand to address this requirement at different levels. Preparation includes training, good information flows, documented processes, awareness and exercises. Key to this is ensuring that everyone has the ability to identify a security incident and alert the appropriate contact - defining when an incident becomes a security incident is another topic for discussion.

Once all these factors are included with all the IT controls, they must be checked for compliance, which includes not only to internal policies but also to legal and regulatory standards. Many standards with which organisations need to comply will go beyond the IT systems and focus on the information itself, such as data protection legislation.

How the scope pans out

The scope of information security therefore includes elements such as IT Operations, Human Resources, procurement, service management, physical security, incident management, business continuity, legal and compliance. There will be separate departments that implement and manage the controls within the organisation and it is the remit of information security to ensure that the processes they operate take into account the controls required to mitigate the risk and that cooperation is obtained for audit and compliance work. This should be in the form of a working partnership, not a dictatorship.

When you consider particularly the integrity and availbility of information in addition to confidentiality, the scope increases further. As an example, physical security should include not just fences, locks, biometrics and CCTV, but also the physical attributes to maintain the availbility of data such as the ability of the building to resist environmental threats such as extreme weather, the stability of ultilities such as electricity and the resilience of the cooling system in server rooms.

Security can be whatever you make it, and different models will fit different organisations. However, to properly consider all the risks to the information you need to protect, you need to think beyond the IT and look at all the information that is of value to your organisation in any format.

Image: jscreationzs / FreeDigitalPhotos.net

Event Correlation: There is no such thing as BAU

BAU or Business as Usual is a term that is used to define a number of different things depending on the nature of your business. For projects going through new implementations or changes, the progress through transition, transformation and testing ultimately leads to the point where it is supported by the normal business and technical management processes that will keep it going until the next major change. More generally, BAU is used to describe the steady state of any process, service or infrastructure, the point at which there are no exceptional changes or problems and where it can easily "tick over" in the same way day after day.

BAU can however lead you into a false sense of security. Achieving a steady state of operation should make it quicker and easier to identify issues which arise. However, problems occur when you start to consider regular issues or low-level low-impact incidents which occur on a daily basis, as normal or part of the BAU operation. Once you do that, you may be ignoring the signs of a larger problem which is bubbling away under the surface.

There are a number of shows on TV now that dissect significant incidents and disasters to examine how they were caused. Typically, these incidents are things that are in the public conscious, were heavily reported in the news at the time and either threatened or took lives. Incidents such as plane crashes, train accidents, ferries sinking or industrial accidents of some kind are all subjects of these shows. The key point made by all of these programmes is that these things don't just happen without any warning signs and cannot be attributed to a single issue or failing. These types of incident are a chain of events which have come together to cause a far more significant incident or disaster. The reason that these issues are not identified in time to prevent a disaster is that they are each only visible to different people, have no correlation or visibility in a holistic fashion and more often are not considered to be issues because they are just things that happen as part of Business as Usual.

For a plane crash, the programme wil talk about a number of minor factors which could contribute to an accident: the maintenance team not following proper procedures in order to get their job done quicker, the ground crew who ignore an issue with the plane, the fuel truck driver who incorrectly tries to convert litres to gallons, the pilots who don't get enough sleep and are not fully alert, the air traffic controller working long hours with too many planes to mange, the airport with out-of-date equipment to facilitate landings, the company that transports dangerous materials on the flight without appropriate controls or the airline that pushes for faster turnarounds to make or save more money. These are all typical findings but are all either treated as normal events and not given the visibility at a level that can assess the overall risk to the flight itself. It's not until after the event does someone (typically the team investigating the crash) put all the pieces together to lead up to the event. By then it's too late.

The same applies in information security. Events don't just occur and incidents don't happen without warning, however there are often minor issues which are ignored as "acceptable failings" such as the patches that don't get applied in time, the ongoing virus detections which are quickly handled by the AV and not investigated, that one ID that always seems to log failed access attempts, the documentation not completed during changes as it holds the process up too much and demands from customers and management to respond faster and achieve more in less time. On their own, these are things that may just be treated as BAU occurrences, but may actually be symptoms of a larger problem bubbling away under the surface. The only way you're going to identify the true risk posed by the aggregation of these events is to firstly have visibility of them and secondly to understand how these individual issues might ultimately cause a larger problem. This is where event correlation is important.

There are plenty of options for Security Information and Event Management (SIEM) tool sets to correlate event data from the many technical sources within your environment. The signature of a security event can comprise information from many sources in the network which individually may not seem significant. However, SIEM tools are only part of the solution and although they can sift though potentially millions of alerts and log entries to give a concise and actionable picture of technical events, this then needs to be combined with other information to give you a correlation at a higher level. Process failings and incidents which are not detected through technical measures are also elements which can contribute to a security incident and it may be that a low-level correlated event from your SIEM system, combined with additional information gathered externally, indicates a more significant threat that you are facing. Security management standards such as ISO 27001 define the importance of measuring the effectiveness of all you security controls, not just the technical ones, as an ineffective manual or procedural control can just as easily contribute to a security incident. The human element can not only be the weakest link but is typically also performing the types of controls where failings and effectiveness shortfalls are far more difficult to detect due to no technical monitoring being in place.

The upshot is that it is important to know your operating environment and have an overall view of both minor incidents that may currently be treated as 'normal' as well as the effectiveness of all your controls, both technical and procedural. Only by being able to correlate the risk of each event and though an understanding of how even if individually the risk of each one is negligible, the combined risk is perhaps intolerable, will you be able to predict and prevent the big incidents or disasters.

Photo: David Castillo Dominici

Information Security vs Formula One Safety

Security is often considered a cost to the business, an overhead that they just have to begrudgingly accept. The pot of gold for any CSO is that magical Security Return on Investment calculation which will justify why the investment needs to be made. Security is a lot like insurance - you need to make the investment now to protect you from an event or events that may happen in the future. The benefit comes from demonstrating how the investment you make now in the controls to remove or reduce the impact of such events is less than the cost of handling the event without those controls.

If we were able to accurately predict with significant certainty, what events were going to happen over the course of a year, how often they would occur and exactly what the cost to the business was going to be be, then security budgets would be both easy to calculate and easily justified. Sadly, there is a significant amount of uncertainty which can lead to an attitude of "we'll worry about it when it happens".

One of my favourite analogies is to consider security like the brakes on a car. A number of people use this example to demonstrate why security should be integrated into any solution from the beginning and why it should not simply be an optional extra tacked on at the end. It can also be used to address attitudes to security and the importance of a well-prepared and executed risk management and security plan to justify security investment. I like to take this example and expand it to compare security investment in business to safety investment in motor racing... In this case, motor racing and more specifically, Forumla One.

The immediate reaction is that the brakes on a car are the things that slow it down and bring it to a stop, which can be the perception of security from a business. However, the brakes are the elements that allow a car to travel fast as the driver knows that the brakes are there when they need them. How fast would a Formula One driver really want to go down the straight if they knew that there were no brakes available as the corner approaches? The brakes enable the car to go fast in the knowledge that when the driver needs to slow down for a corner or to avoid a hazard, that they can do so easily. Lap times are therefore far quicker with brakes than if they were to attempt a lap without them.

That doesn't of course mean that Formula One cars won't ever crash, we know that still happens. The brakes aren't the silver bullet to remove all problems and some might see Security as needing to be just this. The brakes help to ensure that in most circumstances in the event of an incident that the impact (in more than one sense of the word) is reduced and recovery is quicker. Brakes on the car are only one of the protection mechanisms in place. The design of the car, the drivers clothing and all the many monitoring capabilities that the team has to assure the heath of the car are all controls to protect the driver.

The driver is not the only asset that needs protecting, the team's reputation also needs to be considered. Who will want to drive for or sponsor a racing team if it's cars have a really bad safety record? This isn't just about winning races and beating competitors, it's about protecting assets. A key point to make here is that the justification of spend to reduce risk, needs to be assessed on all assets, including people and reputation, not just IT and data.

This analogy could go on for quite a while and become even more multi-faceted, but I'll reign it in here in an attempt to get to the point. Formula One teams spend a lot of money on safety to ensure that their drivers are protected, the money they spend is offset against the ability the car and driver have to drive fast and win a race, collecting prize money and more sponsorship deals. Knowing which safety measures to implement will be based on a risk assessment of the likelihood and impact of events that might impact their ability to win races. By analysing previous incidents, performance of their and other teams and by continually monitoring the state and health of their cars when racing, they can make smart investments, win races and increase revenue.

This is the same for any business. To be successful, they need to look at what what the assets are they need to protect, including data, information systems, infrastructure, buildings, people and the reputation of the organisation. They then need to ensure that they understand the threats to those assets, the probability of that threat causing an incident and the impact it will have to the business - any information on past incidents within their own company or to other companies will aid this assessment and should also be used ongoing to gauge the effectiveness of any implemented controls. The required security investment must be made to clearly meet the output of this assessment in the form of defined controls, technical or procedural, that will reduce the chance of an incident occurring, reduce the impact of an incident if it occurs and allow the business to recover as quickly as possible.

Any Formula One team that makes Safety an afterthought will have the same problems as any business that makes Security an afterthought.

Photo: Pete Keen