CSC failure leaves 80 trusts without IT systems

31 Jul 2006

A technical failure at an NHS Connecting for Health data centre on Sunday has left 80 NHS trusts across the North West and West Midlands, including eight acute trusts, without access to patient data on their clinical and administration computer systems.

E-Health Insider understands that trusts across NWWM were left without access to key IT systems after yesterday's failure at both a centralised data centre and a disaster recovery facility with a full back-up system.

The problems were continuing this morning with three trusts confirming off the record that they were experiencing problems with unavailable systems. In addition Birmingham Children's Hospital NHS Trust confirmed that it had been hit. "We are experiencing technical problems with Lorenzo[its computer system]," a trust spokesperson told EHI.

One trust IT manager told EHI: “The data centre in Kent had major problems at the weekend. Every application provided is off at every hospital and PCT.” The manager added: “This will really hit places like University of Birmingham, as Monday morning is always the busiest time.”

The systems affected include patient administration systems, holding millions of patient records, being managed for NHS trusts by US company Computer Sciences Corporation (CSC) as part of the NHS National Programme for IT.

It is not yet clear if the failure affects all NWWM sites that have received patient administration systems from CSC. However, if it does, up to nine acute trusts and approximately 40 community trusts may have to revert temporarily to manual systems.

CSC is delivering the software applications through two supposedly foolproof data centres, one in Maidstone in Kent and the other in a back-up centre which is meant to automatically take over in the event of any problem.

CSC and NHS Connecting for Health were preparing a joint statement as this article went to press.

++++++++++++++++++++

Update (15.00, July 31)

++++++++++++++++++++

Subsequent to the original story above being published by E-Health Insider, NHS Connecting for Health issued the following statement:

"Regrettably, NHS Connecting for Health (NHS CFH) can confirm that there has been serious interruption to computer services provided by CSC Alliance in the NHS in the North West and West Midlands of England since 10am on Sunday 30 July 2006. This incident was caused by Storage Area Network equipment failure and has affected several other organisations which also use the CSC Maidstone Data Centre.

"Technical issues following power system interruptions mean that data held on computers in the central "data centre" for the region cannot be accessed. The nature of the incident meant that service could not immediately be provided by the back-up systems, also provided by CSC Alliance. No data has been lost.

"Experts from CSC and its sub-contractor Hitachi have been working round the clock to restore access to data. NHS Connecting for Health has been kept informed about progress. A decision has been made to prioritise the restoration of key services. Whilst some services will be restored within a few hours other services will take longer to restore.

"In the meantime the affected trusts - all those which have had new administrative computer systems installed by CSC - are continuing to provide normal service by operating manual contingency systems. Some 80 trusts (72 primary care and 8 acute trusts) are affected. NHS CFH and CSC regret the inconvenience this incident is causing and are committed to resolving the issues as soon as possible."

Reader's Comments

Fallen at the first post!

31 Jul 06 16:51

Yet again another failure this time on a massive scale. No data was lost? lets count our blessings..NO... well lets focus on the real issue, the 'foolproof' system failed and more worrying is the 'back up' that will feed the nation failed too. Is this a warning shot of things to come?

Normal service

31 Jul 06 17:00

If normal service can be provided without the affected systems than I would be interested in knowing what benefit NPfIT is designed to deliver and I say that as a keen advocate of the programme!

Is the technology up to the challenge?

31 Jul 06 17:27

Without wanting to sound too much like a Luddite, I've often worried that information systems / technology are not yet advanced enough to fully support clinical practice across the numerous organisation that comprise "the NHS" and that the available expertise might be too thinly spread in the service. This latest technical hitch (primary plus backup systems) tends to strengthen these concerns.

Like the previous commentator, I've also often wondered just what benefits NPfIT/CfH was going to deliver. If a manual fallback is going to be needed - and this incident suggests that this may be the case - then the whole thing appears rather pointless.

Was this on the risk register?

31 Jul 06 18:14

From past form, the next statement from CSC and CfH is likely to be more self-congratulatory than humble - how they've now completely recovered, no data lost, how they'll now be implementing even more robust failover methods, etc. So perhaps there is only a brief opportunity to ask whether the technical experts chez CSC and CfH had specified and listed this obvious and mission-critical risk? It's far from unknown for prime and recovery sites to be lost at the same time.

All the eggs in one basket?

31 Jul 06 19:41

"Technical issues following power system interruptions" Unbelievable - what happened to the backup power supply? Presumably there was one or was it a casualty of the CfH determination to get the lowest possible price? If I were a Trust Chief Executive I would be very nervous about such a centralised system. Where does the buck stop if a patient comes to harm in this sort of situation? Whoever thought up the idea of putting "All the eggs in one basket" is presumably feeling a bit anxious.

No loss of access to clinical records?

angus.goudie@GP-A89021.nhs.uk

01 Aug 06 09:50

I was suprised at this comment on the news and wondered if it merely reflected the fact that all clinical records of these trusts were still wholly on paper, or if there was an alternative system being used for clinical data. The worry is that with increasing computerisation of the clinical data (a step that for patient care would be a major advantage) such a failure would leave no clinical fallback data. I'm sure a local mirror at each trust would be much more reliable as well as protecting against spine problems/ jcb through the cables etc.

No business continuity

01 Aug 06 09:58

As an expensive PAS replacement, Trusts can survive for a while without the system. The problem with this centralised approach is that if NCRS ever succeeds in replacing paper casenotes, the clinical capacity of 1/5 of the country could be seriously adversly affected at just the same time that the Service was dealing with a major crisis. Current Disaster Recovery plans for closing one hospital and using the neighbour would have to be modified to send busloads of patients up motorways to a different cluster, or re-employ an army of clerks to deal with re-emergent paper mountains.

Is it by design that all the main data centres I know of are in the low-lying South East of England, closest to continental thunderstorms - was the UPS failure a lightening strike ?

The interdependance on other datacentres for Smartcard authentication, spine lookup, and the lack of resilience in the N3 design all make the NCRS a nonsense to use as the foundation for a modernised resilient NHS.

Foolproof

01 Aug 06 10:12

Foolproof computer systems? That will be the day!

I read on the BBC web site that clinical info was not effected. Does that mean information could still be accessed?

Does the fact they are now working without the central computer mean that relevant patient medical info can still be passed on to those who need it? If yes, why spen £12 billion on NPfIT?

If NPfIT had used a system that allows hospitals/medical practices to simply send relevant info, would they still be capable of doing that?

If the new system means that hospitals/medical practices have to log onto the spine, would a simmilar system crash cause problems? The only way I can see to prevent that is if each person that accesses info from the spine, they make a back up copy on their own system. Is that how it will work.

One other thing that I think is important. Although the artical above does not say it, can people stop saying it will conect hospitals and 30,000 GPs to allow them access to clinical information. Can they start to point out that it will also connect clerical/admin staff, NHS mangers, Nurses, Researchers/planners ect (as far as I am aware identifiable data will be uploaded onto SUS who will be allowed to pass on information to researchers/ planners ect, although in most cases, not all, they will remove identifiers so that you can not be imedeitly identified).

Performance penalties

01 Aug 06 10:24

And the service credits for this failure will doubtless fail to trickle back to the Trusts who are grappling with paper records, overtime for data re-entry, angry patients, frazzled staff.

Doubtless it will instead be lost in legal fees, PR fees, and the other management overheads to claim that the brave new world is really wonderful.

Risks or savings?

01 Aug 06 10:44

The failure of centralised data warehouses surely brings into question the viability of their use? As some will remember the explosion at the Buncefield fuel depot effectively wiped out the data service to a number of NHS trusts including Addenbrookes in Cambridge.

Isn't it time to set the theoretical cost saving in overheads aside and work with a local system architecture? I would have more confidence in systems being managed on my own site (even my own department) than a remote data centre where one failed system kills services to almost a third of the country.

First strike only

01 Aug 06 11:45

It is far too quick off the mark to claim this is 'proof' of non-viability of any centralised solution. This incident may turn out to be a remediable operational failure with lessons fully learned.

Offsite data hosting works well given

1. fully redundant network connections to the data centre

2. suitably designed and configured hardware and software systems at the other end of the pipe

Chelsea and Westminster, I believe run on this model and have total unplanned downtime since go-live in 1998 best measured in minutes. Stay awake at the back there ;-)

Clinical Information

01 Aug 06 11:59

Clinical information will not be held on the system until late 2008 - that is why none has been lost.

Foolproof II

nhstechie@btinternet.com

01 Aug 06 15:03

" (as far as I am aware identifiable data will be uploaded onto SUS who will be allowed to pass on information to researchers/ planners ect, although in most cases, not all, they will remove identifiers so that you can not be imedeitly identified). "

That would be in breach of the DPA. Anything on SuS will need to be pseudonymised before being widely shared - the info general researchers and planners will have access to will be aggregated.

However, if a medical researcher were to discover a case where an individual patient had a medical condition which had not been recognised it would be unethical not to be able to reverse the pseudonymisation to enable that patient to be identified and treated. This would also be needed if a case-finding algorithm were run to identify "at risk" patients - e.g. those whose with a high risk of developing CHD or diabetes.

See http://www.connectingforhealth.nhs.uk/sus for more detail.

Probably too early to speculate on the cause of the outage - just glad I'm not a techie in Maidstone, it was bad enough when my PAS was down in a single Trust setting!

First strike only - II

01 Aug 06 15:43

With reference to your point 1: 1. fully redundant network connections to the data centre

The contract signed by DH with BT for the delivery of N3 connections to acute trusts provides for so-called resilience through assured diversity of routing - in practice for most trusts this amounts to primary and secondary connections being routed through the same underground conduit from the same local exchange, so that the resilience is illusory, with no real redundancy in the network connection between trust and data centre. Until the N3 contracts are updated to deliver real redundant network data paths between trust and data centre, we will all be vulnerable to loss of service in the network element.

First strike III

01 Aug 06 16:54

This is not the first failure. Late last year, all the Radiology Systems supported by the Southern Cluster datastore had 4 outages in one month, each lasting a considerable period. The backup store was not ready to use, but the Trusts had not been informed.

The N3 comments above are very important. Wonderful having a non-stop facility (if it actually works) which is completely isolated from the users by a digger.

The impact of significant failure under this programme is not just one hospital having a problem, but a whole swathe of the country being isolated.

And many of us in the service have been raising these concerns all along. Wake up CfH and listen !

RE foolproof II

01 Aug 06 17:15

If the pseudonymisation is reversed, who would get to know the patients info? For example, would SUS inform the patients GP, or would the researcher be given access to the patients identifiable details? It is things like that that need to be made clear to patients, but sadly no one seems to do that (I cant even get my PCT to say if any of my identifiable medical info is available to anyone outside my previous practices!).

It is the fact that SUS would hold identifiable data about me, that causes me the problem. The only way I can see SUS being able to do what what said is if all patient details were uploaded onto it. There ius no way I'd put up with that

Is it a case of the Health and Social Care Act 2001 and the Control of Patient information Requlations 2002 would not allow the release of identifiable medical info from SUS for things like research?

I agree it seems to early to speculate about what went wrong, but it does highlight the risk of using a central system.

First strike only - 11

04 Aug 06 07:58

With reference to N3, your comments are correct, but trusts can bring primary and secondary in at different points on the campus to ensure redundancy (if their infrastructure allows)! Also note that N3 may terminate primary and secondary circuits in the same POP (albeit electrically seperated), and of course the secondary circuit is rated at 50% of the primary. The redundancy issue is a real problem in all N3 installations, particularly in community settings, unless of course you have a COIN that is being customised and you can thereby ensure resilience and diversity in a real sense!

Remember Manchester?

mary.hawking@nhs.net

05 Aug 06 19:55

It would be an advantage if the pathways by which data left an organisation was separated into two physically different strams. Remember the fire in a tunnel in Manchester last year? My understanding is that, wheerever the data came from in the first place, it all went through the same tunnel - cutting out not only GP Enterprise systems but credit card recognition facilities - no cash points and no-one able to use credit or debit cards . Why not mirror between a local system and the central system for backup?