Cost-effective Business Continuity Solutions – So much more than Data Back-up

Most Radiology PACS currently in use have some sort of data back-up in place. At the very least, the Directory database and the Data database are backed up daily to digital tape. In my opinion, digital tape is not reliable and the problem is you don’t know what data you have lost until you try and retrieve it. My low opinion of digital tape is supported by a number of reports from the field. I suspect the vendors that continue to insert digital tape back-up solutions in their early round quotes, do so in order to keep the price of the system down, but a much better solution is worth a few dollars more.

The “tape-less” back-up is a much better back-up solution. Instead of digital tape on a shelf or in a mechanical jukebox, a far more reliable and performance-oriented solution is to store the back-up copy of the Directory and the Data on spinning disk. Thanks to today’s pricing, a multi-processor, multi-core server coupled with a disk-based storage solution is only slightly more expensive than a digital tape library. I think the reliability is worth the additional investment.

Why stop there?

Instead of just writing a copy of the Directory on the back-up storage solution, why not install a second instance of the Directory application (Oracle, Sybase, DB2, SQL, etc.) on the back-up server? Now you have a reasonably cost-effective Disaster Recovery solution, depending on where you have physically placed that back-up system.

Why stop there?

Why not add a second instance of the PACS application to the back-up server? Now you have a reasonably cost-effective Business Continuity solution. Of course this complicates the PACS application considerably. The optimal software configuration would have the two Servers (Primary and Secondary) functioning in an “Active-Active”mode, and that would mean that the Directories are being automatically synchronized in near-real-time, and the study data is being copied from Primary to Secondary on a fairly regular basis.

Only the newest generation of PACS can support this configuration. Most of the PACS being sold today can support a “tape-less” back-up server, but they do not support a second instance of the Directory application on that back-up server. The few that do support a second Directory do not support a second instance of the PACS application. Fewer still that support a second instance of the Directory and the PACS application have the back-up system operating in a standby mode. The Back-up takes over only when the Primary is off-line for scheduled or unscheduled maintenance. While this version of back-up may not sound so bad, the fact is that the failover and eventual reconstitution processes are often manual and labor intensive.

The point in all of this is, with today’s cost of hardware it doesn’t make sense to settle for a back-up solution with questionable reliability, when a much more reliable Business Continuity solution is affordable. The problem is most PACS currently being sold are “old” generations of system architecture wrapped in pretty GUI and flashy 3D applications. While GUI and display applications are important, I believe that the system architecture that supports a solid Business Continuity solution is more important, and sooner or later those old generation PACS are going to be upgraded. You can tell a lot about the longevity of a PACS, by investigating the various back-up solutions that it can support. Why start a five year contract with an old PACS? Do you have room for a forklift in your data center?

The Problem with Proprietary Data/Object Formats – their Impact long after Data Migration

This is another take on a long-standing problem with most of today’s Radiology PACS: proprietary Data/Object Formats. It has been at least four years since Presentation States and Key Image Notes were included in the DICOM standard, yet the majority of PACS vendors continue to treat these key work products as proprietary objects. The most consistent excuse is “There are many more features on our engineering schedule considered to be more important to our users.”

I can almost believe that story, since I have found that most users are not aware of the implications of proprietary data objects. Since almost every PACS supports the creation and display of Presentation States and Key Image Notes, the fact that most PACS treat these as proprietary objects is lost on most buyers and eventual users. Provided that these objects are kept within a given PACS, there is no apparent negative to their being proprietary. The user may not experience a situation where the proprietary nature of these objects presents a problem.

The problem arises when the user of one of these proprietary PACS tries to forward study data to another Facility or Health System that is using a different PACS. Whether that other PACS is DICOM conformant or not, unless it is the same PACS, those presentation States and Key Image Notes cannot be transferred, accessed, or displayed. Physicians using the other PACS will not have the benefit of seeing exactly what the radiologist interpreting the study saw in the images or what he may have typed as a text message. The benefit of these “work products” is lost.

The problem also arises when a user of one of these proprietary PACS tries to copy study data to a CD/DVD. The proprietary work products either cannot be copied, or they cannot be accessed and displayed by another PACS. This is one of the reasons why there is so much consternation over the current CD/DVD copying solutions on the market. The vendors of these proprietary PACS typically have to place a copy of their own viewing software on these CD/DVDs, because their proprietary viewer is the only way to view their proprietary study data.

The real problem will manifest itself only after the user has decided to replace the proprietary PACS with the next PACS. Data migration services will typically migrate the study pixel data to the next PACS, but few of these services currently migrate any proprietary study-related data objects. To do so would require knowing where these objects were stored in the PACS, how to extract them and how to convert them to their DICOM counterparts. This extraction, conversion, migration is not being performed and as a result, those proprietary data objects are lost forever. The images are available for historical comparison in the next PACS, but none of the proprietary work products are available. Now imagine the implication of having to window and level all of these priors again, when they are recalled for viewing with the new images. Imagine not having the spine labels, and not having any other annotation or overlay graphics created when the prior was first interpreted. That’s working without benefit of prior information, or a possible expenditure of time redoing all that work.

A PACS should treat Presentation States, Key Image Notes, .wav files, Technologist Notes, Scanned Documents, even the Radiology Report as DICOM Objects, not only so they can be shared with other systems today, but also so they can easily be migrated and used in the next PACS. DICOM-conformance is always in the user’s best interest.

Now if a prospective buyer knew the negatives associated with proprietary data objects, would they choose a proprietary PACS anyway? Logic suggests that they should think twice. At the very least, if an organization goes ahead with the purchase of a PACS that still creates any proprietary data/object formats, that organization should negotiate a “no-cost” data migration clause in their contract that pins the cost of moving these proprietary objects to the next PACS on the vendor who has continued to choose NOT to conform to the standard.

Lack of DICOM conformance is a type of vendor lock. I believe that the PACS vendors still believe that anything that complicates moving to another vendor’s PACS may persuade the organization to stay with the incumbent. It’s time to make them pay for that strategy.

Coming Up For Air

Here it is February. Where did January go? For that matter, where did December go? My bad! I’ve been so negligent with keeping my site up to date. It’s not that I was chasing snow or anything. I’ve been very busy with a number of projects; two separate RFPs for Radiology PACS, a Project Plan for a large multi-site health system, and a PACS-neutral Archive project.

I really did want to publish a commentary or two on my RSNA ’07 observations. There certainly was plenty of inspirational Material.

  • GE finally has a decent Radiology PACS (through acquisition). Now what?
  • A number of vendors start talking about PACS-neutral archives, but you really have to look hard for the vendors, and then look even harder for anyone who can speak to the subject.
  • It’s truly amazing how few vendors consider themselves DICOM-conformant, yet they do not support DICOM Presentation States.
  • Apparently 3 MP color display panels are now the norm in Diagnostic display stations.

Let’s just say I have some new things to say about some old subjects, and a few things to say about some new subjects. Sorry for the tease, but I need a little time to collect my thoughts. Check back in a few days and you’ll find some posts on such subjects as:

  • The problem with proprietary data/object formats, their impact long after data migration
  • The beauty of media-neutral storage solutions
  • Cost-effective Business Continuity Solutions, so much more than data back-up
  • Double-check that quote configuration, because there are lots of hidden costs that sneak into the picture at contract time.

Take the Archive Out of PACS

Those of you that have been following my recent posts on the subject of PACS-Neutral Archive might find it useful to visit the HIMSS or Emageon web sites to access a webinar delivered today to an audience of 70+ members of HIMSS.

slide1

The seminar covered the subject of Tag Morphing and explained how some very common problems faced by Health Systems today can be resolved by deploying a PACS-Neutral Archive; problems such as the sharing of a single archive among multiple dissimilar PACS, and the elimination of future data migrations. The Emageon web site offers the visitor the option of downloading a collection of white papers that describe the concept of PACS-Neutral Archive and Tag Morphing in more detail.

Check it out.

Is new Stark Exemption an Opportunity?

I came across an article in Imaging Economics titled “Surveys Show Paper Legacy Tough to Shake”

What caught my eye was the second paragraph statement “A new Stark exception allows hospitals to donate health information technology in the form of an EMR to private physicians.”

I was wondering if the definition of “EMR” could be extended to radiology web viewer? Is this possibly a mechanism for providing the necessary hardware (PC), software and connectivity services to the referring physician office to get them to stop requesting paper and film?

The article is worth reading as it explains why “more than 50% (hospitals) continue to print and distribute paper lab and imaging reports.” This does not come as a surprise to me, but it occurs to me that if so many hospitals are still printing paper radiology reports, a similarly large number must also be distributing hardcopy images.

Clearly the success of a Radiology PACS depends on turning off a large percentage of hardcopy and getting the referring physicians to access images and reports from their offices electronically. I have long argued that the cost of providing a suitable PC and basic connectivity services is more than paid for by the value of the hardcopy. Many clients were concerned about the Stark implication. Is this exception an opportunity?

The article goes on to explain that 62% of hospital executives surveyed in February said their organization had no plans to donate technology. “They’re waiting to see how the government changes the landscape. How will it affect their nonprofit standing, that kind of thing.” Once again, I think this is a shortsighted point of view. The continued printing of hardcopy films is certainly affecting their bottom line. Why not take advantage of this opportunity to legally equip their referring physicians with a much less expensive method to access images and reports?

Separating Storage from the PACS is Good First Step

I was browsing today when I came upon an all too brief article that appeared in HealthCare Informatics in August, 2007. The title of Stacey Kramer’s article is A Two-Tier Solution, and the first sentence states that “Memorial Hospital found it takes two vendors to handle imaging properly – one for PACS and one for storage.”

According to the article, Memorial Hospital decided to combine a McKesson PACS with “IBM’s tiered storage solution”. Unfortunately the article provides no real information on the actual configuration, or any explanation as to why this was the ideal combination.

Without any detail, I am left to speculate that this is merely an example of the customer requiring the PACS vendor to substitute the customer’s favorite storage solution for the storage solution originally proposed by the PACS vendor. If this is the case, this is hardly a breakthrough.

If in fact, “IBM’s tiered storage solution” was their GMAS configuration featuring Bycast’s potent Information Lifecycle Management software, that would be a significant upgrade over the typical PACS configuration that features direct attached storage, but once again, hardly a breakthrough.

There is no evidence to suggest that there is a separate data Directory on this separate storage solution, and that the data is in any way being managed independently of the PACS application. Bottom line, the McKesson PACS still controls the study data, and years from now, when Memorial Hospital decides to replace this PACS with another PACS, they will have to migrate all of that study data through the McKesson PACS and through the new PACS, even if this migration is right back to the same IBM storage solution.

Choosing a separate Storage Solution was a good First Step, but the next step would have been to interface the McKesson PACS to a PACS-neutral Archive. There are a number of PACS-neutral Archive software applications that could utilize the same IBM storage solution, but in this case, the study data would be controlled by the PACS-neutral Archive and not the McKesson PACS application. The study data would not have to be migrated downstream, when the McKesson PACS is replaced.

The good news is that it is never to late to build a PACS-neutral Archive, and pro-actively migrate the study data to this archive long before the data migration task gets that much bigger and much more expensive.

I have written several White Papers on the subject of Pro-active Data Migration and PACS-neutral Archives. The papers are too lengthy to publish on this web site, but they are freely available to anyone forwarding an email request.

PACS for the Smaller Organization

Over the last several months a number of posts have shown up on auntminnie.com and pacs-admin@yahoogroups.com asking readers opinions on some of the smaller PACS solutions in the market. I assume that most of these questions are being posed by members of small imaging operations performing less than 40,000 procedures per year who assume that they can only afford the relatively inexpensive PACS solutions offered by the small vendors. In the past, this was probably the case, but that is no longer the case.

Today, several of the biggest vendors in the PACS market, creators of the really big and fully featured PACS, have achieved a scaling feature that allows them to offer effectively the same fully featured PACS at a price point within reach of even the smaller imaging operation.

These vendors have achieved this scaling by reducing the number of servers in the cluster, without eliminating robustness or reliability. They have retained the basic display features, including hanging protocols, but made many of the more advanced display features (like 3D) line item extra-cost options, so they can be added for a modest license fee if needed. They have made many of the professional services that were automatically included in the big system, line item options in the scaled down package. The site that can follow directions and set up their own modality interfaces, complete their RIS interface on their own, and perform their own network testing can save some money.

Perhaps the best feature of this new generation of scaled down PACS is that their upside potential is not artificially limited. If the study volume suddenly jumps by 100%, the small system can be expanded to accommodate growth, without a wholesale exchange of hardware or a whole new tier of software licensing. The user truly pays for only what they need, and only as they need it.

In this scenario, there is no reason for the smaller imaging organization to risk an investment in a fragile company and purchase a bargain-basement PACS with limited features and limited support. It is now possible to afford the economical version of the same PACS being used by the big boys.

GC’s Major Guidelines for Picking the Best PACS

Several interesting posts popped up on Auntminnie’s PACS Forum today. Two were related to display software for referring physicians, and two others were related to the log-term archive. In my first response , I spoke about the importance of picking a PACS that featured a single display package that allowed system managers to create individual user profiles by assigning display features through user privileges. I also suggested that the Health System should consider providing the display hardware and IT support for their high volume users, because it would be cheaper in the long run than producing and managing film. In a follow-up response, I flat out stated that as much or more attention should be paid to the display software that is going to be used by the referring physicians as is paid to the display software being used by the radiologists. Failure to win over the referring physicians, especially the surgeons, will surely doom a PACS project.

The first of my responses to the archive issue focuses on using a spinning disk solution for the Disaster Recovery subsystem, and the need for some sophisticated Information Lifecycle Management software in the archive that would make it possible to migrate data from media to media and delete data based on information about the study contained in the DICOM header. In the same article, I couldn’t help but ask the question why anyone would create an exact duplicate of the original image data, if the PACS utilized any proprietary formats. It seems to me that if you are going to invest good money in a DR solution, the second copy should be 100% DICOM and 100% inter-changeable with another PACS. This would eliminate future data migration costs. In a second response, I suggested once again that the time has come to separate the Archive from the PACS. The PACS vendors insistence on using Private Tags and proprietary encoding is blatant vendor lock. It is expensive (data migrations) and it should be stopped.

So here is my simple Guideline for picking the best PACS

1) Distributed server architecture. Each facility gets its own Directory and Data database servers and there is one shared long-term archive. Each facility is self-sufficient, yet there is one consolidated patient folder. The central shared server “aggregates” all of the information from the facility servers. The user doesn’t have to know where to look for any study on any patient in the system.

2) Single master copy of display software, one common GUI, fat client for performance, web-delivered for zero administration. Each user can be granted access to whatever display features and tools they think they need.

3) Software license fee is based on the number of studies under management, NOT the number of users, or the mix of features/tools being assigned.

4) PACS-neutral Archive: guaranteed universal connectivity, ability to morph DICOM Header Tags in order to copy any meta data in Private Tags to Public Tags, no future data migration necessary. If the PACS vendor that ranks the highest in every other category cannot provide this kind of archive, buy the archive from the vendor who can and configure the PACS with a small working cache.

5) Make sure the archive supports a sophisticated ILM strategy, one that migrates data from media to media or deletes data based on information in the DICOM Tags, data transfers have zero impact on the PACS or Archive application server.

There are other important issues and features to be sure, but they pale in significance to these five.

What was the Department of Veteran Affairs Thinking?

I came across a news article today announcing the VA’s plan to establish a Disaster Recovery program for all of their Radiology Departments that had already installed the Philips iSite PACS.

“Royal Philips Electronics has announced an agreement with the Department of Veterans Affairs (VA) to provide disaster recovery services that are dedicated to VA users with Philips iSite. Managed and hosted by Philips, the VA disaster recovery services will provide automated backup of all Philips iSite Radiology image data.”

It is well known that the Philips iSite PACS stores the image data in a proprietary (iSyntax) format. The Philips PACS is not the first PACS to be deployed by the VA and it probably will not be the last. When the time comes to replace the iSite PACS with something else, all of that study data accumulated over the years will have to be migrated to that next system. That is going to cost both time and money.

A shared Disaster Recovery program is a great idea, but why deploy a DR solution that stores another copy of the study data in a proprietary format? It seems to me that the deployment of a Disaster Recovery solution is an excellent opportunity to create a second copy of the data in a PACS-neutral format. Start copying the historical data already stored in the iSite PACS to a Vendor-neutral Enterprise DR (archive) solution. Call it a “pro-active” data migration. Then continue to store all new study data accumulated by iSite to this Vendor-neutral DR solution.

When the time comes for any of the sites to move on to their next PACS, there would be no need to migrate that site’s study data over to the new PACS. A Vendor-neutral archive (server and storage) would be built and loaded with that site’s historical data (in a Vendor-neutral format) and then shipped to the site. This local facility server would interface to whatever new PACS is being deployed. The new PACS would not have to be configured with a long-term archive. There would be no need for the time-consuming and expensive data migration.

A Vendor-neutral Enterprise DR solution could also be shared with all those other VA facilities that do not have Philips iSite PACS. What are those sites suppose to do for their DR solution? How many different DR solutions does the VA want to support? Could it be that all VA facilities will be encouraged to upgrade to the iSite PACS? No doubt that’s the Philips plan.

Don’t misunderstand, I think that iSite is one of the better PACS in the market, but data migration is an inherent problem with changing PACS, in some cases with the next generation PACS of the same vendor (Siemens Magic to Siemens Syngo). It simply doesn’t make sense to build a DR strategy that doesn’t take into account the high probability that some other PACS will be deployed somewhere downstream, and thus require a sizeable data migration project. A sensible plan would take reasonable steps to avoid that problem.

It should not be a matter of money. Hardware is hardware. Granted, the Philips software license for that second copy of the data is probably less than what the Vendor-neutral Enterprise DR software will cost. But the cost of all those future data migration projects would more than likely cover the premium charged for a Vendor-neutral Enterprise DR solution that could be shared by every VA site today.

I’ve written a few other posts on this subject that you might find interesting.

PACS-neutral Enterprise Archive – Who will build it?
Looking for a PACS-neutral DICOM Archive?
An Enhanced DICOM Archive would be the ticket!
PACS Vendors think PACS-neutral Archive is crazy idea
SCAR ’06 Update

If you would like to have a tool to help you estimate the cost and time associated with your future data migration projects, just email me at graycons@well.com and ask for the Migration Prognosticator.

Pay Close Attention to DICOM Conformance

The typical PACS includes its own long-term archive subsystem. While the all-inclusive package will present few if any data compatibility issues with its own components, there may be serious problems when the time comes to exchange data with other systems. The archive that it totally owned by the PACS application is usually only marginally DICOM-conformant. The assumption of the self-contained PACS is that there is very little sharing of study data with other systems. The PACS may respond to a remote DICOM query with minimal data, i.e. the original image pixel data and little else. Presentation States, Key Image Notes, and other key meta data objects associated with the images and created by the radiologist during interpretation may not be forwarded, because the PACS doesn’t treat these as DICOM objects, or it places them in Private Tags, or it uses a proprietary Value Representation (in the tag) to encode the information. Self-contained Radiology PACS are typically very stingy when it comes time to give up their data in a data migration process. The vendors really think of it as their data, and now that the radiology department has decided to leave them for another PACS vendor, the jilted vendor may be reluctant to help in the retrieval and migration of all of the data that really belongs to the health system. This translates to expensive and time consuming data migrations. Study the Archive’s DICOM Conformance statement very carefully. Anything less than full conformance for all data objects and SOP Classes should be addressed in the Contract. Build a technically sound, workable exit strategy in advance.