Spine to be shut for two day 'refresh'

01 Nov 2007

Connecting for Health (CfH) and BT have confirmed that the NHS Spine will be unavailable when it is taken down for a major upgrade from next Friday evening until Sunday morning (9-11 November).

The Department of Health agency describes the refresh of software and hardware as ‘unusual’ and ‘unlikely’ to occur again in the future. A practice ‘refresh’ of the Spine has already been carried out as a dry run in a recent technical rehearsal.

CfH is warning that the planned maintenance work will affect all NPfIT systems linked to the Spine, including: the Personal Demographic Service, Choose and Book and GP2GP.

The agency said the upgrades did not represent a complete replacement of the Spine. “The exercise is solely an uplift of the database management hardware and software. The application code and functionality of the Spine remain unchanged. The change will be transparent to users,” said a spokesperson.

BT says the upgrade, which includes a refresh of the Spine software and hardware, is required in order to “enable a more automated, faster and surer failover process and continuing levels of quality support.”

A CfH spokesperson said: “Maintenance of the Spine usually takes place with no obvious impact to system users. As this maintenance session implements a complete software and hardware refresh of the database infrastructure, NHS Connecting for Health have had to adopt a different approach.”

The period of planned maintenance is from 2200 on Friday 9 November to 1000 Sunday 11 November 2007.

CfH said: “Usual business continuity measures, which all organisations should have in place, should be adopted during the period of planned maintenance.”

In guidance to trusts, CfH says that some services may be affected: “The NHS CfH architecture is designed to allow systems to run in local mode if they are unable to contact the Spine. GP and hospital patient administration systems will still be available during the 36-hour maintenance period. However, their integration with national systems and services, such as Choose and Book and the Personal Demographics Service may be affected.”

Choose and Book will face major disruption during the planned maintenance for all users.

Over the weekend of 10-11 November, clinicians and patients will not be able to view, make, change or cancel an appointment using the Choose and Book service. There will be no access to the Directory of Services and callers to the Choose and Book appointments line will only be able to have discussions about choice.

CfH has advised clinics operating during this time to print patient referral information for the 36-hours downtime in advance.

GP practices using the GP2GP service to transfer records will not be able to send or receive information from 2200 on Thursday 8 November to 1000 Sunday 11 November 2007, allowing for the 24 hours transfer completion time.

It will not be possible to issue prescriptions with barcodes on them, so pharmacy dispensers will have to manually input prescription items into their systems.

In addition, updates to patient demographic data will not be saved on the Personal Demographic Service during the downtime.

CfH said trusts have been aware of these plans since the beginning of the month and should be well prepared for the maintenance period. “The overall impact is expected to be minimal,” the agency said.

Adding that the benefits of the proposed maintenance will include the ability to make use of new and emerging technologies, access to improved functionality, maintained stability and speedier recovery, improved system performance and on-going quality of support.

Links

BT Health

Connecting for Health

Reader's Comments

Business as usual

01 Nov 07 12:10

Spine to be shut down for two days. Will anyone notice?

Reality Check

thrust2@hotmail.co.uk

01 Nov 07 13:51

This is a sensible, industry std maintenance action. The disruption over a weekend will in reality be minimal (thats why they chose it!). Lets all be thankful they are ensuring that the spine stays at maximum integrity and performance as we all move forward!

Reality check - mate

01 Nov 07 16:34

"ensuring that the spine stays at maximum integrity and performance as we all move forward!"..... and a big thank-you to the Connecting for Health rebuttals unit.

NHS doesn't stop at the weekend

01 Nov 07 20:10

The spine is there to support patient healthcare if CfH are to be believed. Interestingly healthcare still continues at weekends, A&E departments don't close. People don't get turfed out of hospitals at the weekend. District nursing services, mental health services all continue as well. The sooner CfH and the LSP/NASP's understand this the better for all concerned. Also, we have a winner for the most prepostourous posting ever on EHI by thrust2.

Hmmmm

nhstechie@btinternet.com

01 Nov 07 20:26

Is it just me being thick, or do these two statements really contradict each other?

CfH said: “Usual business continuity measures, which all organisations should have in place, should be adopted during the period of planned maintenance.”

": “The NHS CfH architecture is designed to allow systems to run in local mode if they are unable to contact the Spine. GP and hospital patient administration systems will still be available during the 36-hour maintenance period...."

"Usual business continuity methods" is a CfH euphomism for falling back to paper records when all else fails!

So which is it - 'switch off and go to paper' or 'local business as usual'?

How often will this be needed? and communication issues..

maryhawking@tigers.demon.co.uk

01 Nov 07 20:37

I think there are a number of concerns about this. I'm a GP: at the moment my practice is not very spine dependent (and I have been told by N3 that the spine being closed does not mean that N3 is!: in fact, N3 didn't seem to have heard of the closure!). My practice received a document printed off the Spine website on actions to be taken - but this seemed to have come without expaination from the PCT IM&T department. It dealt with actions to be taken - but not with why the closure was needed or whether this was likely to be a regular occurence. Is there a communication/customer relationship issue here?

I am not clear whether this type of closure is necessary for maintenance upgrades: *does* the London stock exchange or airline booking systems have closures for maintenance/upgrades? Is this going to be a regular occurrence? that is, do we need to have business continuity plans to cover temporary closure of all Spine functions, and include this in our planned budgets? (the actions suggested look rather expensive to me!) As a GP using GP2GP, I would have appreciated having it highlighted that I would need not to register patients from 22.00 on Thursday 8th November - and whether, if I missed this, this would affect the registration process, or only the transfer of records... In summary Was this necessary and will it occur again? What is the strategy for informing the people affected? Could they re-write the document on actions needed by end user rather than appication?

9 to 5!?

02 Nov 07 00:14

"The disruption over a weekend will in reality be minimal (thats why they chose it!). "

What a good job the NHS shuts down at 5pm Friday and doesn't re-open until 9am Monday!

Contrary to what you may have read in the Daily Moan, many GP practices function at the weekend, as do A&E, Out-of-Hours services, twi-light nurses, minor injuries units, drop-in centres, Mental Health crisis teams, labs, radiology services, Ambulance services.

Healthcare is a 24/7 business. Let's hope this doesn't become a regular occurence.

I'm afraid bland reassurances from CfH fall on deaf ears around here - been there, done that, got the scars to prove it!

Spine to shut down

c.lake@ewks.co.uk

02 Nov 07 10:07

BT and CFH clearly need to provide systems that are robust, safe and as effective as possible. The maintainance they're talking about is most likely an upgrade to Oracle 10G, the database engin. My concern is that they need to turn the system off completely to do it. This suggests to me that the back-up systems are not working as they should. Lets hope that when the system is finished they can carry out maintainance without disruption to the service

Back up, schmack up!

nhstechie@btinternet.com

02 Nov 07 20:47

"This suggests to me that the back-up systems are not working as they should"

What back up systems? The LSP systems are supposed to fail-over (hence the deserved furoré when CSC's didn't last year) but I've never seen anything to indicate a similar fail-over for spine services - happy to be contradicted!

NHS doesn't stop at the weekend

05 Nov 07 11:10

"NHS doesn't stop at the weekend 01 Nov 07 20:11 The spine is there to support patient healthcare if CfH are to be believed. Interestingly healthcare still continues at weekends, A&E departments don't close. People don't get turfed out of hospitals at the weekend. District nursing services, mental health services all continue as well. The sooner CfH and the LSP/NASP's understand this the better for all concerned. Also, we have a winner for the most prepostourous posting ever on EHI by thrust2."

Believe it or not CfH, the LSPs and even the software suppliers DO UNDERSTAND that healthcare needs don't just stop on Friday 5pm!.... Its very naive to think that they don't, afterall the majority of staff in these agencies are ex NHS. Its fairly obvious that whilst its not perfect, the impact of upgrading at the weekend is far less than during the week when you have outpatient clinics running, routine ops scheduled, patients in for day surgery etc etc etc..... but the biggest point to make is "The spine is there to SUPPORT patient healthcare" - IT systems will NEVER replace clinical judgement, and lack of services should never stop clinical care. The loss of PDS for a weekend means that you have to use locally cached (in PAS) demographic information that may be out of date - but was this not the case prior to NPfIT?

Spine Backup

steve@uninet.demon.co.uk

05 Nov 07 12:45

Spine is spread across two data centers. Either site can support the Spine service (PDS, PSIS, etc ..) So the implication is that a full upgrade is required across both sites to a common system. Its either an Oracle upgrade and subsequent migration, or an upgrade to TMS (messaging bus).

NHS doesn't stop at the weekend

05 Nov 07 22:30

NHS doesn't stop at the weekend 05 Nov 07 11:11

Totally missing the point here. The fact is that when was the last time you can remember the banking system shutting down for 36hrs. The issue is here that the Contract is very poorly written for the NHS in that the LSP's and NASP's are allowed to take the system offline for 36hrs. It was never the case prior to NPfIT for two reasons, the local dept understood the impact of downtime and the wasn't billion pound contracts to "professional" IT service companies. Fact is here there should be a failover so impact is milli-second. CfH is far too lenient with the suppliers as it cannot afford another to walk away.

Banks and systems

steve@uninet.demon.co.uk

06 Nov 07 10:54

I find the repeated comments about Banks and Exchanges "never" shutting down as interesting.

In the case of Banks, before the advent of the internet they would shut down every day. Even at that most major bank systems are still shut down every day for overnight processing. For example, you don't expect that all retail branch systems are on-line 7 x 24?

Certainly in case of the LIFFE market and London Stock exchange they shut down every day for clearing. Also they don't operate at the weekend. Agreed that when they are on on-line during the working day they need to be 100%.

In terms of Medical systems the attribute that would concern me more than the system being on-line would be its ability to deliver full functionality. After the upgrade weekend I suspect the Spine will be "on-line", but will everything actually work as it did before the outage?

When is the NHS Busiest?

simon.lacey@buckspct.nhs.uk

06 Nov 07 13:46

A Monday or a Sunday?

Therefore common sense would suggest that downtime is best scheduled for a week, when disruption can be kept to a minimum.

When do the railways schedule engineering who? Same applies.

Really people - its not complicated!

Smart Card Log Ins

06 Nov 07 18:01

Nowhere is log in via Smartcard to CfH applications mentioned. I hope this is not a spine component. If so, many trusts will be affected by the down time.

Missing The Point

06 Nov 07 21:09

Too many people missing the point here. Yes, of course disruption should be kept to a minimum and that obviously is a weekend - that bit is not rocket science and should go without saying. The point here is that the NHS is trying to modernise and move to electronic health records. It cannot do this with confidence if the suppliers of software have to take software off-line for an extended to perform an upgrade. When the NHS is on electronic records, in say 5 or 6 years time - it just will not be possible to do this as the disruption to care will be huge. I personally do not think CfH or NASP's and LSP's understand this.

in a perfect world...

07 Nov 07 10:16

Technical staff in the LSPs and CfH do understand the problems with taking live systems down for maintenance, and would love to be able to do so by using failover systems while the main system is upgraded, and then upgrading the failover when the live is back up and running. However, due to management decisions, this isn't always possible as there is no failover system in place which could handle the load. A good example being Accenture's decision to use the failover servers for some of their initial iPm deployments as the servers as the main servers for the second phase - tech and support staff howled about this and tried in vain to point out this was a Bad Idea, but were ignored in the quest to claw back some of that £300 million loss. Please don't mistake high level bean counting for technical incompetence. (Or as a CfH rep in a meeting once said to me, "high level decisions made with low levels of thought")

Smart Cards

steve@uninet.demon.co.uk

07 Nov 07 12:29

To the poster that was asking if smart card login functionality was part of Spine. Yes it sure is. If the whole of Spine is down then this will certainly not work.

I would hope that if you have been told that LSP systems will not be affected that this functionality will be available and hence not affected by the upgrade.

Whole versus Partial downtime

08 Nov 07 12:20

The current dependancies on the Spine are far less than is anticipated within a couple of years, but this sends all the wrong messages. Many clinical users will be left ignorant of the maintenance, and the few who get frustrated by no access will have lower confidence in any NPfIT antics as a result.

The communications are difficult (partly because CfH have failed to learn from and use established routes, eg use of front screens etc), and the message 'we can shut down for 36 hours at a weekend' is preposterous, and damaging to trust. Not only are there many clinical users working at those times, but with CfH/LSPs/NASPS spending all the money, the NHS has not been able to staff 24x7 local IT cover, so these users may not get swift support.

But the design that has so much depending on a single element is extremely poor, and some systems will not work without the Spine for the duration (C&B, ETP). And as a post above implies, look out for a bumpy week following the upgrade.

Design

steve@uninet.demon.co.uk

09 Nov 07 13:07

"But the design that has so much depending on a single element is extremely poor, and some systems will not work without the Spine for the duration (C&B, ETP). And as a post above implies, look out for a bumpy week following the upgrade."

As someone who had a bit to do with the design of Spine I would like to point out that this is not necessarily a design issue. I suspect that this has a lot more to do with a lack of foresight and technical capability for a vendor I would prefer not to name.

A bumpy week indeed!

13 Nov 07 20:38

Tuesday 13th 2pm - now received update no 6 on a Sev 1 National PDS problem.