Barts virus caused ‘major internal incident’

Tags: A Ambulance Barts and the London emergency iS London Mytob Network Pathology Patient safety Safety Security Surgery Virus

15 Jan 2009

Last November’s Mytob worm attack on the network of Barts and the London NHS Trust led to its ‘major internal incident' plan being activated, with some ambulances redirected from A&E.

The network failure was one of the most severe known to have occurred at an NHS hospital trust. To clean and restore the infected network, Barts had to draft in help from neighbouring trusts and a 40-strong team from BT.

An interim report on the incident says clinical services were affected, though effective alternative arrangements worked. However, it also says that protracted delays in getting the network back up, and in providing access to the clinical systems that run on it, created potential risks to patient care.

The incident began on Monday 17 November, but the network was still down two days later. Even with extra help, it took over a week to get all top priority areas reconnected. It took the trust until the 2 December to fully recover from the failure.

A trust spokesperson told E-Health Insider that an investigation into the incident continues: “The investigation into how the virus managed to evade our security is not yet complete. Until then it would be inappropriate to speculate on the outcome. It is expected that the results on the investigation will be presented to the trust's board in late January.”

The interim report states that sometime before 17 November, the trust’s computer network became infected by a variant of Mytob.

“The virus started to have significant effects on the performance of the trust’s network from about 12 noon on Monday 17 November, making most of the trust’s applications, including those covering patient administration, pathology and imaging, inaccessible to clinicians during the afternoon.”

Initially the IT department responded by “isolating components of the network” and putting in place scripts to prevent infected PCs from accessing the network.

Parts of the network were back up by the evening. However, these measures proved ineffective when large numbers of staff attempted to log-in the following morning. This led to the network being taken down for a second time.

On Tuesday 18 November, the trust decided it needed to put its ‘major internal incident' plan into action to ensure that key clinical systems continued while network access was being re-established.

As part of this process the trust sought “an ambulance diversion to limit the flow of emergency patients to the hospital.” The diversion held until the evening for trauma and complex surgery cases “because of pressure on blood cross-matching and access to cross-sectional imaging techniques.” The report says: “No patient safety incidents were recorded over this period.”

With the network down, only limited access to the system was available in A&E and other key areas. As a result, the trust had to revert to paper systems, including “runners” and manual requesting and communication of tests.

The report says recovery proved difficult for a number of reasons: getting the right script to counter the virus took time; with the network down it was difficult to know the extent of the problem; and disinfecting PCs had to be done locally at individual workstations rather than by remote updates. Problems with network stability arose when the trust tried to reconnect cleaned PCs to the network.

As a result, a decision was taken to draft in extra help. “The incident team requested the director of ICT to consider obtaining external support from other NHS providers and BT. All neighbouring trusts, including central London teaching hospitals, provided staff to help disinfect PCs.”

Even then, it took the weekend of 22-23 November to get through the majority of prioritised areas for reconnecting PCs to the network.

The interim report to the trust board by the medical director, Charles Gutteridge, concludes: “The systems supporting and maintaining the network have been shown to require urgent review and improvement. As more and more patient-related data is only available on IT systems, the need for resilience within the network becomes more critical.”

Finally, the report says the trust found itself wanting in the expertise required to deal with such a major network disruption. “It is clear that solving large scale network interruptions requires expertise and staff numbers which are beyond the day-to-day ICT resources of the trust.”

Related articles

Barts takes a week to recover from Mytob virus

Jon Hoeksma

Readers Comments

Simple solutions

15 Jan 09 08:04

Yet mytob could come my way and I would not have a problem and I don't use anti-virus software. Answer - use software that isn't vulnerable. Why not use a proper server OS such as UNIX/Linux or Solaris. This kind of problem is then not an issue.

I can understand why they want desktops to use windows, though I still don't think it's a good idea, but there really is no excuse for servers to be running it.

Windows updates?

15 Jan 09 23:06

I would be interested to know what the status of the windows updates was on all of the infected machines. Doesn't Mytob make use of a known vulnerability?

If the machines hadn't been patched against a known vulnerability then there has to be a certain amount of culpability laid at the door of IT

Windows Updates

16 Jan 09 17:18

None of this surprises me - I'm a contractor who works across the UK and I have NEVER come across any trust who has a current installation of windows on its PCs.

In fact I know of a number of trusts who still today have PCs running Windows-98 and are connected to the trust networks.

Time for a change?

kirky76@googlemail.com

19 Jan 09 08:35

To suggest that Unix is not vulnerable software is not entirely accurate or helpful. It is vulnerable, just to a different set of exploits. It is naive to suggest that the NHS, the 4th largest employer in the world will suddenly move to using non-Microsoft products.

The answer to this problem is far more effective use of IT security following a defence-in-depth principle. Also, senior staff in the NHS need to take note of the need to invest in IT security solutions. The network and associated IT systems are a critical part of the hospital infrastructure, just like power, water and heating and yet they are often not valued in the same way by senior NHS leaders.

One other huge issue here is the reliance on signature-based protection. I suspect that Mytob spread so easily simply because it was either a) a new variant that didn't have a signature or b) the existing AV was not effectively updated. Either way, it highlights the fact that signature based detection is fundamentally flawed. The industry has recognised these limitations for some time, maybe it's time for the NHS to reconsider their endpoint protection regime otherwise I fear this will not be the last major worm outbreak we see.

How to stop a virus in the first place.

andrew.clarke@lumension.com

22 Jan 09 20:53

A virus is an unknown application, so just control it and stop it executing.

What stops a virus causing havoc? - Application Control

The principle of Application Control is to only allow applications that are required to execute. ie Unknown applications such as a virus or malware will NOT run - so we can stop this havoc from occuring.

The technique is also referred to as "whitelisting" - this straightforward control mechanism allows the IT manager to eliminate unknown or unwanted applications in their network, improving overall managaeability but most importantly by reducing the risk of malware, viruses and spyware.

Therefore, by using application control in your environment would quite simply not allow a virus to run.

Check out: http://tinyurl.com/cmpp3y for more specifics.

Virus protection

sleepyfox@gmail.com

23 Jan 09 14:49

Having worked on the National Programme and with Acute Trusts I can vouch that very few have the resources to use up-to-date versions of Windows Desktop software or to maintain/monitor their networks adequately for security purposes.

As far as white-listing goes: if only it were that simple. Sadly many vulnerabilities exist which exploit flaws in Windows native code, drivers, DLLs or Windows Desktop applications e.g. IE and the Office suite - all of which would not be caught by white-listing.

White-listing is instead one of *many* tools in the security toolbox, and not a silver bullet. The tools need to be used in an appropriately coordinated and planned manner to minimise risk of vulnerabilities being exploited.

However, unless you are a 'burn-victim' or a security professional, it is unlikely that you are capable of making an informed estimate of the cost/benefit ratio of funding the security for your site/Trust - which is why under-informed bean counters will be the ones deciding the reality of how the Trust's IT Security Policy is implemented. This leads to poor security, and a heightened risk of patient safety incidents.

Although there were apparently no serious consequences to patient safety from this major security breach, it is surely only a matter of time until there is elsewhere, as the NHS becomes - like many other industry sectors - more 'fly-by-wire' as the years go on.