PACS-Neutral Archive

What is Enterprise Imaging WorkFlow?

The current generation of department PACS was designed over ten years ago. That statement also applies to the workflow and worklist components of the PACS solution as well as the system architecture. The workflow application was designed to assemble images for interpretation by a single physician group working in a single imaging department, working with a single department PACS. The concept of prioritization was placing a STAT icon next to a study ordered by an Emergency Room physician, and perhaps applying a background color to that line item in the list. Today, even mid-sized healthcare organizations are commonly made up of several amalgamated radiology groups (some owned, some affiliated). They have multiple EMRs and PACS solutions. These organizations have to manage complex cross-site credentialing issues while trying to deliver a common standard of care across the new integrated enterprise. In addition to this, the organization has to hold each physician group to similar performance goals. The worklist of each individual physician absolutely has to consider such input as: physician availability (schedule, locations, etc.), turn around time, physician RVU loading, sub-specialty reading, credentialing, critical results reporting, and peer review. None of the current generation department PACS have a workflow application that addresses today’s issues much less future issues we can barely imagine. A new generation of workflow application that is applicable to the enterprise is clearly needed.

Slide4

Enterprise Workflow Launching Most Suitable Display Application as determined by Study Descriptors

There have been improvements over the last ten years in the features and functions of the diagnostic display application, and yet most imaging departments have had to augment their core PACS application with a number of third-party specialty display applications. The physicians have to work through a pull down list of these applications in order to find the one that is the most suitable to use in the interpretation of the study they have pulled off their worklist. Similar to the way the enterprise workflow solution needs to provide a federated view of available studies to read, the enterprise workflow needs to provide federated access to all the available diagnostic display resources within a site or across a multi-site enterprise.

An enterprise workflow/worklist application is also one of the key components of the next-generation PACS. The PACS 3.0 configuration that I describe in the white paper recently published as a three-part series on AuntMinnie.com. is based on a Vendor Neutral Archive. The various diagnostic display applications that might be used in one or more imaging departments to interpret the images are simply plug-ins to the VNA. As the brain of the PACS 3.0 configuration, the enterprise workflow/worklist application is the entry point of all of the interpreting physicians in all of the imaging departments. The individual physician worklist presents the highly specific list of studies to be read and the underlying workflow launches the most appropriate diagnostic display application based on the pre-defined choices of the physician and the specific type of study selected from the list.

The new diagnostic imaging paradigm, PACS 3.0, is relevant to more than the radiology department. Medical imaging has always been an enterprise operation. Radiology and cardiology are the most obvious medical imaging operations, but many more clinically relevant medical images are generated in other departments. The images captured during an endoscopic procedure do in fact comprise an imaging study that is interpreted. Images captured during an office visit (dermatology) or throughout the course of treatment (wound care) are perhaps not considered an imaging study, but they nonetheless are clinically significant and should be retained as part of the patient’s medical image record.

The PACS 3.0 paradigm should be extended to all of the departments in the enterprise that utilize imaging. The endoscopy study can be ingested by the VNA and the enterprise workflow application can create the worklist for the Endoscopist, and that workflow application would most likely launch a basic clinical display application to review the study, whether the images are DICOM or JPEG.

A similar scenario can be applied to each of the other imaging operations, whether those images are DICOM or non-DICOM. The later may require a front-end application to create the study from a collection of individual images and associate the proper patient and study metadata to the study, but the VNA is the data repository and the enterprise workflow application is the entry-point for the physicians responsible for interpreting the images.

Now is the time for a single workflow/worklist application that can be used across the entire enterprise. Not only does a single workflow application simplify physician access, a single workflow application consolidates software applications, simplifies IT support, and makes economic sense.

The white paper I recently wrote on Enterprise Workflow would be useful in your strategic planning. The paper describes a key attribute of Enterprise Workflow being [1] its ability to auto-route the new image study and its relevant priors to the server hosting the display application that is most suited to the interpretation of the study, and [2] its ability to launch that display application in patient context when the study is selected from the worklist. This functionality is based on the workflow application’s ability to recognize and use the study descriptors and the physician reading preferences to determine the most suited display application. In this sense, the enterprise workflow application is really the brain of the enterprise medical image management solution.

Upgrading Department PACS with Entry-Level Vendor Neutral Archive Components

Organizations that appreciate the benefits that the Vendor Neutral Archive (VNA) will bring to the organization, sometimes find it difficult to finance the complete VNA solution in a single budget cycle. It is most unfortunate if the decision comes down to deploying none of the solution, due to a lack of funding for all of the solution. In my opinion, continued investment in a proprietary Picture Archiving and Communications System (PACS) archive and the continued accumulation of proprietary data is a flawed strategy.  In a paper sponsored by an unrestricted grant from IBM Systems and Technology I propose an alternative to the full VNA deployment, and discuss an affordable entry-level phase to a multi-phase VNA deployment strategy.

Any IT initiative that is focused on an upgrade to a department PACS storage solution, a refresh of the existing storage solution or a replacement of the disaster recovery (DR) storage solution initiated by an end of life letter, is an opportunity to carefully consider the value to the organization in continuing the proprietary PACS archive paradigm. Migrating the image data from the PACS to a properly configured entry-level VNA is the organization’s opportunity to once and for all normalize its image data and end the cycle of expensive and time-consuming data migrations that are required with every PACS replacement project. The hardware agnostic, entry-level VNA gives the organization the opportunity to apply advanced storage technology to all of its data management applications. All of these benefits can be realized at a fraction of the cost of a fully configured VNA, and in many instances for a price very close to that being quoted for the proprietary PACS archive solution.

Image-Enabling the EMR – a Hybrid Approach

Market adoption of the Electronic Medical Record system in the United States has soared over the last three years, and penetration is expected to continue at a brisk pace for the next few years.  Markets and Markets.com estimated in June 2011 that the compound annual growth rate of EMR systems in the US market would be 18% for the period 2010 to 2015.  Healthcare organizations have recognized that taking clinical records digital is a more efficient methodology for managing and communicating the patient’s longitudinal medical record.  EMR sales have no doubt also been stimulated by the carrot and stick approach that the government is taking with the Meaningful Use initiatives spread over this same period.

The EMR alone is not designed to be the single central repository of all of the patient’s clinical records.  Most healthcare organizations in the US quickly discover that after investing many millions of dollars, burning untold man-hours in the associated training programs, and imploring the medical staff to actually use the system…after the first 30 days of experience with the system, that frustrating question from the medical staff is

“Where are the images?”

Clearly, a key part of the EMR project must be to determine how to deliver the complete longitudinal medical record to the caregivers…including ALL of the medical images.

The long-term plan must include not just the images from radiology and cardiology, but also those from pathology, ophthalmology, endoscopy, dental, and those captured by digital cameras in surgery, burn care, dermatology, etc.  Anything less is simply an incomplete plan.

One strategy for image-enabling the EMR is to interface it to the department PACS.  Lots of EMRs are interfaced to the radiology PACS.  Of course that doesn’t deliver access to the cardiology images, unless the radiology PACS manages those images as well.  The major flaw in this strategy should be fairly obvious…it would be fairly expensive and very inefficient to push all of the images form the various imaging departments to the radiology PACS.  Another flaw with this strategy is using a radiology-oriented viewing application to visualize images from each of those other departments, not to mention the non-DICOM formatted images.

An alternative is to build an interface between the EMR and each of those department PACS and free-standing imaging systems, but that strategy is flawed as well.  Not only would it be expensive to build and maintain all of those individual interfaces, but the physician user would be forced to learn and use multiple viewing applications, and it would be virtually impossible to combine images from multiple disciplines on the same viewing screen.

The long-term plan should also include a methodology for delivering all of the image types to platforms far more ubiquitous than desktop computers located on desks in nursing units and offices.  It means delivering them to laptops, and tablets and even smart phones.  Currently the majority of department PACS systems feature clinical viewing applications that are thin clients that run on Wintel platforms. The few that are web-based are also limited to Wintel platforms and a specific browser.  In this sense, the typical clinical viewing application incorporated into the department PACS is largely unsuited to image-enabling the EMR.

Standalone UniViewerThe superior strategy for image-enabling the EMR is to use a zero client application that features server-side rendering that can be hosted in a virtual server environment.  The HTML download of the rendered image by the rendering server easily overcomes the traditional physical limitations of low bandwidth connections.  The zero client means that the user can pretty much use whatever platform they choose: Windows, Mac OS, Linux, iOS, Android, etc. and any browser they prefer.

This class of standalone zero client viewing application is generally referred to as a Universal Viewer, or “UniViewer” as I have referred to it in many of my writings.

Some of the earliest Universal Viewers were add-ons to the Vendor Neutral Archive.  The viewer could then draw upon the VNA for the image data.  More importantly, the nature of that integration meant that the vendor could use a web services interface like WADO-RS to transfer the images between VNA and rendering server, so performance was not burdened by the relatively slower DICOM transfer protocol.  Unfortunately, image-enabling the EMR in this manner means deploying a combined VNA and Universal Viewer…an expensive and time-consuming IT project.

Image-enabling the EMR should not require deployment of a full VNA.

Many of the better Universal Viewers have the ability to aggregate image data across multiple image data repositories.  They can be directly interfaced to multiple department PACS as well as free-standing workstations or imaging devices that archive either DICOM or non-DICOM image data.  They can take a URL call from the EMR interface and query/retrieve a specific image, a specific study, or a specific patient’s entire longitudinal medical image record scattered across multiple repositories.

Unfortunately the Universal Viewers are currently limited to the use of DICOM communication transfer protocols with the department PACS, largely because that’s the way the PACS vendors want it…standardized but slow.  Consequently the EMR user’s ad hoc query through the Universal Viewer of a study being managed by a department PACS would be painfully slow.  Not because the Universal Viewer is slow to render and deliver the HTML version of the images, but because the Universal Viewer is stuck with DICOM Q/R as the data retrieval mechanism.  The present work-around to the DICOM bottleneck is to give the Universal Viewer’s rendering server a large image cache.

The purpose of the rendering server’s cache is to pre-stage those new and relevant prior studies that the EMR portal users are most likely to request for viewing.  The larger this cache, the larger the volume of new studies that will be available for access and the larger the percentage of relevant priors that will also be available for access.  An organization should be able to monitor its existing PACS in order to determine the average number and age of relevant priors that are being reviewed along with new studies.  A cache large enough to manage the most recent year of all new studies would probably be holding 90+% of the relevant priors.  Of course the advantage of the Cache is defeated unless the interface between the cache and the rendering server is something like WADO-RS.

There are several workflow solutions to getting the images from the PACS to the Cache.  One approach is to have the Rendering Server query each PACS for new image studies on a regular basis.  This approach requires that the rendering server have a mechanism for remaining synchronized with each PACS to make certain that it always has the latest version of the study.  Using this approach, image availability to the portal user is highly dependent on the specific PACS, as some PACS will respond to an external query and release the images as soon as they have been acquired, while other PACS will not release the images until after the study has been read.  A second approach is to equip the Universal Viewer with an HL7 Orders interface and a relevant prior algorithm, so it can use the HL7 Order feed to trigger a pre-fetch of the relevant priors from the appropriate PACS and a Q/R of the new study.  There are a few other options that can be applied to very stubborn PACS solutions that are beyond the scope of this article.

Regardless how the images are retrieved from the PACS, the issue once again is the scalability of the cache as well as the directory database and file system that is managing the cache.  Reducing unwanted delays in image retrieval means extending the cache to include as many probable priors as possible.  There is also the issue of data purging.  The Universal Viewer application needs to be able to use several variables such as study age and time-last-accessed to determine when it purges study data from the cache.  Many Universal Viewers are not designed to manage a large scalable Directory or Data database, nor or they very smart about the purge strategy.  At some point, the cache starts to become a somewhat sophisticated data management system.  At what point is this Universal Viewer cache another island of data that becomes a burden to IT?  On the other hand, a Universal Viewer that can only support a minimal cache and a simple FIFO purge policy is probably not going to keep up with performance expectations.  An organization should not be surprised to see the cache continue to grow in size to meet the expectations of the users.

So here is my suggestion…start off with a Universal Viewer application interfaced to a single instance of a Vendor Neutral Archive licensed to simply act as the cache for the Universal Viewer’s rendering server.  Most combinations could and would use web services like WADO-RS (not DICOM) to transfer the image data between the VNA and the rendering server. VNAs support all types of image acquisition and workflow schemes, so they could easily handle any of the existing PACS in the field.  Unlike most of the Universal Viewers, VNAs can be scaled all the way up to a full VNA configuration that is managing all of the enterprise image data.  Unlike the simple Universal Viewer cache, the VNA could also be performing a normalization of the incoming data, so population of the cache over an extended period of time combined with cancellation of the purge mechanism would effectively become a pro-active DICOM data migration…something that every organization will eventually have to do when those PACS are replaced.  This is an ideal upgrade path for a Universal Viewer that (by itself) is limited by its cache scalability and really has no upgrade path beyond image-enabling the EMR.  This is also an excellent strategy for leveraging current demands for image-enabling the EMR into the VNA that is the better solution for enterprise data management.

Multi-Phase VNA Deployment Strategy

I have been preaching the merits of a multi-phase deployment strategy for a Vendor Neutral Archive for some time now.  This approach is very reminiscent of the deployment strategy for Radiology PACS adopted in the mid-1990’s.  A properly configured VNA is an even larger system than an organization’s largest PACS, so it too represents a large investment.

Nevertheless, a lack of funding to cover the deployment of the complete VNA should not deter the organization from making a start.  Every day going forward, the organization is forwarding a significant volume of new studies to their department PACS, which in the course of ingestion routinely makes proprietary modifications to the DICOM header.  Those PACS-specific idiosyncrasies in the headers are the primary reason behind the expensive and time-consuming data migration that is required when a PACS is replaced or simply upgraded to a next generation.  In effect, an organization is adding to its data migration liability every day that it continues to forward new study data to its department PACS.  At the very least, the organization should consider a pro-active data migration project…whereby all of the image data being managed by its department PACS would be converted from the proprietary format to a Neutral format and simply parked in a separate storage solution.

Most of the Vendor Neutral Archive vendors can configure entry-level versions of their products that will perform this pro-active image data migration and simply store the result for future use.  This approach makes an excellent Disaster Recovery solution that can be shared among multiple PACS.

I recently spoke at length about this subject with David Yeager,   who is a freelance writer and editor.  David spoke with several other individuals knowledgeable in this space and combined all of that information in a very comprehensive article on this subject.  The article appears in the October issue of Radiology Today.  David’s article is an easy read of a complex subject, so I encourage you to check it out here.  The online article also contains some very simple graphics to reinforce the concepts.

There are several entry points to a multi-phase VNA deployment strategy.  Replacing a PACS vendor’s DR solution with a Vendor Neutral DR solution is one such example that makes both strategic and financial sense.

The Real Reason for Deploying a Vendor Neutral Archive

I came across a PowerPoint presentation that I created in July 2006 that referenced a DICOM Migration Server, a term at that time referring to an “open” DICOM Part 10 Storage Solution.  I vaguely remembered the subject, so I opened the file and reviewed the slides.  I felt as though I had traveled back in time to the very earliest days of the paradigm shift that would one day be referred to as the Vendor Neutral Archive.  That’s six years ago last month.  Slide after slide contained bulleted descriptions of the numerous problems facing an organization that had managed to accumulate no fewer than five different department PACS.  Five separate silos of data that could not be exchanged between the PACS. Five different image viewers that the referring physicians had to toggle between.  The last few slides in the deck laid out a rather optimistic (at the time) plan for a strategic solution to the mess.  A grin spread across my face.

I closed the slide deck, assigned a loud red label to the file so I could easily find it again, and fast-forwarded to the present thinking, “You’ve come a long way baby!”

I have been intensely focused on both the concept and the reality of Vendor Neutral Archive for those last six years.  Perhaps that is why it seems so obvious to me why a healthcare organization should make the switch.  They should “take the A out of PACS”, move the data to a VNA, associate a universal viewer to the VNA and use this combo system to manage the distribution of that data to every other system and caregiver throughout the organization.  These are things that even the best of today’s department PACS are simply incapable of effectively doing in a multi-vendor environment.

Based on the questions I continue to see quoted in the print and electronic publications, posted on-line in the focus groups, and raised at the end of many of my webinars, there still appear to be a large percentage of both PACS admins and IT systems analysts that don’t “get it”.  They seem hung up on the technical features of the VNA and all of the potential snags that they fear are bound to occur when two systems, more importantly two vendors, are forced to work together.  The litany of both identified and suspected complications goes on and on.  No doubt the incumbent PACS vendors skillfully placed many of the items on these lists.

OK, it’s time to step back from the techy stuff for a minute.

It’s true.  Many currently installed department PACS are incapable of efficiently inter-operating with a foreign archive without help, simply because they were not designed to work with a foreign archive.  The installed base of VNA solutions would be a pitiful fraction of the real number, if the VNA guys had not developed some very clever workarounds to the inadequacies of many PACS.  Without help, most PACS could not be paired with a VNA. They lack the ability to store images to a foreign archive and then remember where they stored those images.  They are incapable of propagating ADT updates or Merge and Splits to an outside archive.  They have no concept of a deletion policy and therefore have no mechanism for reacting to an externally executed Purge Strategy.  Some PACS have no concept of a relevant prior coming from another PACS, and if the VNA suddenly delivers the study to its doorstep, the PACS thinks it’s a new study and puts it in a reading list.  As I have said, the litany of both identified and suspected complications goes on and on.  The naysayers apparently have not taken the time to read up and learn how all of these problems have been resolved.  As a consequence of those workarounds, the actual installed base of VNA solutions is actually a pretty impressive number.

My advice to those that still don’t get it is don’t get hung up on the technology.  The real argument for deploying the VNA is CONTROL.  It’s time for the organization to take control of its data.  Every day that goes by, another “x” gigabytes is forwarded to the department PACS where it is converted to an effectively proprietary DICOM format that the organization will eventually have to pay in time and money to move to yet another PACS with its proprietary format.  Regardless how soon the organization can afford to replace the incumbent PACS, it’s time to start migrating the data to a VNA…in effect it’s time to mitigate the cost of that future data migration.

What about future VNA migrations, when the first VNA has to be replaced with another VNA?  That’s a really good question.

The answer is actually quite simple.  The real objective in negotiating the contract for the VNA is to gain access outside of any confidentiality stipulation in the contract to the VNA database and all of the tools required to allow the organization to move its data from VNA 1 to VNA 2, at no cost.  Without that arrangement, you’ve missed the point.

Bottom line…initiate a pro-active DICOM migration of the PACS data to a entry-level VNA.  Take control of your data.  As soon as possible, replace the uncooperative PACS with a real PACS, one that fully interoperates with a VNA.

PACS / VNA Compatibility Issues

While much has been written and stated on behalf of the Vendor Neutral Archive being the ideal strategy for managing medical image data across the enterprise, little has been said about PACS Compatibility with the VNA Solution.

There’s a good deal more to this compatibility issue than the PACS being able to communicate with and exchange data with the VNA using DICOM.

Most department PACS, including Radiology and Cardiology solutions, were not designed to inter-operate with a foreign archive.  This is not to say that PACS systems were not designed to occasionally share study files with an external DICOM conformant system.  Most PACS can accomplish this using the DICOM communications protocol.  What I mean is that most PACS system designs are predicated on the assumption that the PACS will be the sole manager of the study data for the lifetime of the system.  And because of this design assumption, many of the current generation of department PACS are ill suited to the tasks required to fully inter-operate with a VNA.

Since most organizations probably did not include this compatibility issue in their PACS selection process, it may come as some surprise to learn that interfacing their existing PACS to a VNA is going to require solving a number of significant in-compatibility issues.

I thought it would be useful to present a summary of the more significant interoperability issues, because organizations need to be aware of the potential problems when they begin the process of planning for a VNA deployment.  The right VNA solution will have to be able to address these incompatibility issues.  It might also be prudent to consider these issues when planning for the next department PACS purchase, because sooner or later, odds are the organization is going to see value in data consolidation, and system interoperability.

Here is a list of the more critical PACS / VNA compatibility issues.

DICOM and IHE…The PACS should support a full suite of DICOM SOP Classes covering the full array of image objects that belong in the patient’s longitudinal record, not just those objects created in the imaging department.  This would include most of the DICOM Structured Report objects, image objects from Ophthalmology, Endoscopy, Pathology, and some of the non-image Cardiology objects like Waveforms and Hemodynamic data.  The PACS should also support image-related objects like Presentation States and Key Image Notes.  The system should also support a few key IHE profiles including Consistent Presentation of Images Profile, Presentation of Grouped Procedures Profile, Key Object Notes Profile, Simple Image and Numeric Reports Profile, and Transparent Query/Retrieve.

Foreign Study Support…The PACS should support the import and representation of Foreign Studies.  Ideally the PACS would directly accept from the VNA, studies originally acquired/processed by another (disparate) PACS or Image Source that are being managed by the VNA.  At the very least, the PACS would be able to receive from the VNA a non-billable order and use that order to aid in the acceptance of studies originally acquired/processed by another (disparate) PACS.

Store and Remember…The PACS should be able to “Store” (archive) DICOM objects originally acquired by itself to a foreign archive (VNA), and then “Remember” that the objects are stored on the VNA when the time comes to retrieve them.

Study Aggregation…The PACS should have the ability to automatically and pro-actively search for studies in the VNA that were originally acquired by another PACS and stored in the VNA (i.e. search the VNA for relevant priors originally acquired by another PACS).

HL7 Updates…the PACS should not only be able to accept HL7 updates from the local RIS or HIS and apply those updates to the metadata in the PACS that is associated with the image data the PACS is managing, but it should also be able to forward the same metadata updates to the VNA.

Object Versioning…The PACS should have the ability to forward to the VNA any updates or changes made to the study data (both pixel and meta data) after the initial “archiving” of the study data, effectively “re-archiving” the image or study.

Retention Messaging…The PACS should have the ability to accept and utilize the messaging from the VNA that is designed to communicate what image and study files have successfully been purged by the VNA’s Information Lifecycle Management (ILM) application.

The subject of PACS / VNA Messaging is actually the most critical of the PACS compatibility issues.  Perhaps one of the more challenging aspects of PACS / VNA interoperability is keeping the two disparate systems synchronized with each other.  Most but not all PACS accept and utilize HL7 updates from the HIS or RIS.  Many PACS do not have a reciprocal mechanism for updating a foreign archive (VNA) with changes that were made to metadata or pixel data in the PACS.

An even more challenging issue is presented by the advent of the purging mechanism that is supported by many VNA solutions.  This issue is referred to in the above list as Retention Messaging.  If a PACS is configured with a small local cache, and it is programmed to allow the oldest studies to fall off of that cache when the watermark is reached, how does it communicate to the VNA that it no longer has that study?  Correspondingly, if the VNA purges a study that has reached its retention limit, how does the VNA communicate to the PACS that the study no longer exists?

A number of the more advanced VNA solutions are attempting to resolve this “synchronization issue” through the use of a number of standard and not-so standard messaging techniques, because the VNA vendors recognize that few PACS vendors have considered VNA compatibility in their PACS designs and fewer still have implemented the appropriate IHE profiles.  Some of those solutions include:

  1. Private DICOM messaging
  2. Custom HL7 messaging
  3. The new IHE Profile called Imaging Object Change Management (IOCM), which is still in development

The Imaging Object Change Management Integration Profile will specify how one actor communicates local changes applied on existing imaging objects to other actors that manage copies of the modified imaging objects in their own local systems. The supported changes will include (1) object rejection due to quality or patient safety reasons, (2) correction of incorrect modality work list entry selection, and (3) expiration of objects due to data retention requirements. It will define how changes are to be captured and how to communicate these changes.

The successful assimilation of disparate PACS into an enterprise Vendor Neutral Archive configuration will have its challenges.  I think it is better to fully understand these challenges in order to better prepare for them, and I suggest that this knowledge play a key role in the VNA selection process.   It also makes sense to include the knowledge of these issues in the next PACS selection process, and thereby eliminate as many future interoperability issues as possible.

Pro-Active Image Data Migration to a VNA-enabled Storage Solution is a Strategic First Step in a Multi-Phase VNA Deployment Strategy

The concept of Vendor Neutral Archive (VNA) seems to be gaining traction in the medical Imaging market.  Why wouldn’t it?  It makes a good deal of sense…addressing problems directly attributed to the way department PACS manage image data.  Health Imaging & IT published a list of the top 10 PACS Problems in its July 2011 issue.  In my opinion, the VNA directly addresses 7 of the 10 problems listed in that article. Categories include Integration, Downtime, Hanging Protocols, Interoperability, Out with the Old (data migration), Whose PACS? (Radiology or IT), and Disaster Recovery.

Just to be clear on the concept, I offer the following succinct definition.  The Vendor Neutral Archive is an Enterprise-class data management system that consolidates primarily medical image data from multiple imaging departments into a Master Directory and associated consolidated Storage Solution, thus replacing the individual archives associated with departmental PACS…systems with unfortunate proprietary characteristics that limit their interoperability.   By virtue of it consolidating all of the enterprise image data, the VNA effectively becomes the unified image data repository for the Electronic Medical Record system.

The problem with VNA is not the concept, the problem is the expense.

The properly configured VNA is a BIG system.  It’s often nearly as expensive as the organization’s biggest PACS, since it will be managing all of the data from all of the department PACS, and the proper architecture is a mirrored, dual-sited configuration.

The point I want to make here, is that the properly configured VNA is big enough and expensive enough to require multiple years and multiple budgets…and this is very reminiscent of those first generation PACS deployments in the early 90’s.

Even as late as the mid 90’s, those first generation Radiology PACS required large investments.  And there was admittedly some doubt as to whether that generation of PACS would successfully replace film.  As a consequence, early PACS deployment strategies focused on solving more manageable, smaller problems with entry-level products that were modestly priced.  The mini-PACS for CT, MR, and Ultrasound, and Teleradiology systems were considered PACS precursor products that could ultimately be stitched together to create the full blown department PACS.  The typical PACS deployments in the mid-90’s were phased in over several years and several budgets.

Is there a multi-phase strategy for deploying a VNA?  I think there is.

As long as the organization’s image data is being managed in individual department PACS, the data format is (to some degree) proprietary to the PACS.  Consequently the organization faces the liability of future data migrations, whenever a department PACS is replaced with another vendor’s PACS, or even upgrade to a next generation PACS from the same vendor.  Therefore the first step in a multi-phase VNA deployment strategy should be to migrate all of the organization’s image data out of the individual PACS and into a VNA-enabled Storage Solution…converting the data into a “neutral” format in the process.  In this strategy, the organization is simply “parking” a copy of its image data for future use.  The hardware and software configuration does not have to support any department PACS operations.  This entry-level VNA is simply managing a copy of the image data “at rest”.

I refer to this first step as the Pro-Active Data Migration, and the ideal time to initiate this migration is at the mid-life of the organization’s largest PACS, which is frequently its Radiology PACS.  By the time all of the historical data has been successfully migrated and cleansed, its time to replace that largest PACS.   Now that the organization is in control of its image data, both the old and the new PACS vendors have much less leverage in the selection process.

Here are a few tips.  (1) Look for a VNA vendor that understands that the value of the VNA software license used to manage image data “at rest” (data that is not being accessed by a PACS or a Viewing application) should be a fraction of the value of the fully-activated VNA license.  (2) Consider forwarding the migrated image data to a vendor-managed Cloud Infrastructure, whose fee-for-study costs are probably going to be considerably less than those for an on-site, self-managed solution.

Once the organization has established a repository of one neutral copy of its image data, there are numerous second and third steps that can be chosen to complete the VNA build out.

Best-of-Breed VNA Looks Good on Paper, but can it be Built and Supported?

The initial focus of the Vendor-Neutral Archive concept was two-fold [1] to consolidate individual department PACS archives and [2] facilitate data exchange between those PACS.  As the concept evolved, so did the complexity of the configuration.  Today’s VNA configuration will probably include a pre-fetch and routing application, an HL7 interface, one or more methodologies for acquiring non-DICOM image objects, a QA/QC application suite, a universal viewing application…the list goes on.  It is highly likely that a second party is providing one or more of these numerous applications.  Different vendors will most likely provide the hardware infrastructure.  The system solution package will be comprised of multiple professional services components, which may very well be provided by different vendors.

The composition of the ideal VNA and the specific configuration that best meets the initial and long-term requirements of the healthcare organization will likely be an assembly of best-of-breed components and subsystems from multiple vendors. Therein lies a well-known problem.  Multiple vendors means multiple Service Level Agreements and multiple 800 numbers to call for support.  Is it possible for a single vendor to offer technology choices, multiple configurations, then actually build and support that custom solution?  Can a site-specific, best-of-breed VNA succeed?

Sponsored by an unrestricted grant from Dell Healthcare, I recently completed a white paper that addresses this very subject.   The paper was published/released November 27, 2011, on the opening morning of RSNA.  You can download a copy of the paper from this web site, or visit the Dell website through this link to find my paper and other related information from Dell.

The paper explores the complexity of the VNA and presents some of the options available to organizations looking for a best-of-breed VNA implementation.  It also looks at the profile of an ideal partner that has the software, hardware and services expertise to integrate, deploy and support multiple best-of-breed solutions.

A webinar also sponsored by Dell Healthcare and based on this paper was given on Thursday, November 17, 2011.  Follow this link to the Dell web site to retrieve the replay of that webinar.

 

True Business Continuity Requires Two Separated Instances of all Key Applications

I sat in on a webcast Wednesday, July 20, 2011, and noticed how the concepts of Disaster Recovery and Business Continuity were frequently paired…in the bullets and in the presentation…with the inference that if you had a DR solution, you had a Business Continuity solution.  That’s not the case.  A Disaster Recovery solution is basically a second copy of the data, usually stored remotely, so it can be accessed whenever it becomes necessary to replace the first copy of the data that might be lost or temporarily unavailable.  A DR solution could be as simple as an off-line digital tape cartridge or a DVD disk stored in a safe room.  It could be a near-line solution such as a tape library system or an on-line solution like a separate spinning disk system, either of which is directly attached to the PACS.  A second SAN or NAS storage solution paired with its front end gateway server could be located in a remote data center in order to increase its chances of avoiding whatever disaster might take down the PACS and/or its primary copy of the data.  The DR solution could be as elaborate as a Cloud-based storage solution, which often feature multiple data centers located in distant states.

In all of the above examples however, the original PACS application or a standalone display application like a web server is required to access and display that back-up image data. If the PACS application or any of the display applications are down, or in any way unavailable, the second copy of the data cannot be accesses and it cannot be displayed.  In this case, there is no Business Continuity.

A significant number of installed Radiology PACS are based on a single instance of the data management and display application, but they are configured with both a primary and a secondary storage solution, each located in separate data centers.  Depending on how geographically separate those data centers are, the secondary storage solution represents a solid Disaster Recovery solution.

On the other hand, very few Radiology PACS are configured with two instances of the data management application or the display applications.  And because there is only one instance of these applications, such a configuration does not have a Business Continuity solution.  With few exceptions, if the PACS application is unavailable, neither the primary nor secondary copy of the data is accessible.  If the display applications are unavailable, neither the primary nor the secondary copy of the data can be displayed.

A true Business Continuity solution requires two instances of the data management application, which can access either the primary or secondary storage solutions, and two instances of whatever display application the user prefers for accessing and displaying the data.  These two paired applications…data management and data display…should be geographically separated, so either of them can survive the disaster that might befall the other.

Since few PACS can be configured with multiple instances of its data management and display application, the current strategy for building a Business Continuity solution has shifted to deploying a dual-sited Vendor Neutral Archive and a dual-sited UniViewer.  The two separate instances of the VNA and the two separate instances of the UniViewer back each other up in the event of a disaster.  In the event that the only instance of the PACS and/or its only instance of the display application becomes unavailable, the new studies are probably interpreted at the modalities, and the priors are retrieved from whichever VNA is available and displayed using whichever UniViewer is available.  That is a true Business Continuity solution.  Take a look at my previous post regarding Failover Strategies, if you want to get a better idea of how primary and secondary subsystems back each other up.

Failover Strategies in Mirrored Configurations of Medical Image Management Systems

The subject of Failover and Resynchronization is near and dear, as I’ve been configuring mirrored systems for years.  I have become quite familiar with how various vendors address this requirement.  The principal reason for building a mirrored Picture Archiving and Communication System (PACS) or a mirrored Vendor Neutral Archive (VNA) solution is Business Continuity.  Most healthcare organizations realize that they cannot afford to lose the functionality of a mission critical system like a PACS and an Enterprise Archive, so they need more than a Disaster Recovery strategy, they need a functional Business Continuity strategy.   Unfortunately, it’s really tough to build a dual-sited, mirrored PACS that actually works.  The sync and re-sync process drives most PACS vendors nuts.  There are very few PACS that can support multiple Directory databases.  I think this shortcoming of most PACS systems is why we have been configuring mirrored VNA solutions from the beginning…if you can’t configure the PACS with a BC solution, then you should at least configure the enterprise archive with a BC solution.

In the dual-sited, mirrored image management system, there are two nearly identical subsystems, often referred to as a Primary and a Secondary.  The two subsystems are comprised of an instance of all of the application software components, the required servers, load balancers, and the storage solutions.  Ideally these two subsystems are deployed in geographically separate data centers.  While it is possible to make both subsystems Active, so half of the organization directs its image data to the Primary subsystem and the other half directs its data to the secondary subsystem, the more common configuration is Active/Passive.  In the Active/Passive configuration, the organization directs all of its data to the Primary subsystem and the Primary backs that data up on the Passive Secondary subsystem.

When the Primary subsystem fails or is off-line for any reason, there should be a largely automated “failover” process that shifts all operations from the Primary subsystem to the Secondary subsystem, effectively making it the Active subsystem, until the primary subsystem is brought back on-line.  When the Primary subsystem comes back on-line, there should be a largely automated “resynchronization” process that copies all of the data transactions and operational events that occurred during the outage from the Secondary back to the Primary.

Business Continuity operations can be even more complicated in an environment where there is a single instance of the PACS and a dual-sited, mirrored VNA configuration. In this environment, the failover and resynchronization processes can be somewhat complicated, giving rise to numerous questions that should be asked when evaluating either a PACS or a VNA.  I thought it would be beneficial to pose a few of those questions and my associated answers.

Q-1: If the hospital-based PACS and Primary VNA are down, how does the administrator access the offsite Secondary VNA and subsequently the data from the offsite VNA? Is the failover automated, or manual?  If manual, what exactly does the admin do to initiate the failover?”

A: The response depends very much on the VNA vendor and exactly how that VNA is configured/implemented.  Some VNA solutions have poor failover/resynchronization processes.  Some look good on paper, but don’t work very well in practice.  With some VNA vendors, system failover and resynchronization in a mirrored environment is a real strong suit, as they support many options (VMWare, Load Balanced-automatic, Load Balanced-manual, and Clustering).  Some VNA vendors have limited options, which are costly and actually create down time.  The better approach is a Load Balanced configuration with automatic failover (which requires certain capabilities existing on the customers network-VLAN/Subnet/Addressing), with manual failover being the second option (and more common).  VMWare is becoming much more common among the True VNA vendors, but many of these vendors will still implement the VMWare clients in a load balanced configuration until customers are able to span VMWare across data centers and use VMotion technology to handle the automatic failover.  There is also the option of using DNS tricks.  For example, IT publishes a hostname for the VNA which translates to an IP in Data Center (DC) A, the DNS has a short Time to Live (TTL), such that if DC A fails, IT can flip the hostname in the DNS and the TTL expires in 1-5 seconds, then all sending devices automatically begin sending/accessing DC B.

There is also a somewhat unique model that implements the mirrored VNA configuration in an Active/Active mode across both Data Centers – whereby the VNA replication technology takes care of sync’ing both DC’s, the application is stateless so it doesn’t matter where the data arrives, because the VNA makes sure both sides get sync’d.

The point in all of this is simply that the better and obviously preferred approach to failover is a near fully automated approach, ONCE THE SYSTEM IS SET-UP.  Resynchronization of the data should be automated as well.  Only updates/changes to the user preferences might require manual synchronization after a recovery.

Q-2: What do the UniViewer (zero client, server-side rendering display application) users have to do to access the secondary instance of the UniViewer? Do the users have to know the separate URL to login to that second UniViewer?

A: If implemented correctly, the UniViewer should leverage the same technology as described above for the VNA.  The user’s URL call goes to a load balancer, which selects the Active UniViewer rendering server.   If the Primary UniViewer (Active) has a failure, another node, or another data center takes over transparent to the end user. The rendering server in turn points to a load balanced VNA such that the users need to do nothing differently if the UniViewer servers or the VNA servers switch.

Q-3: Where do modalities send new studies if the onsite PACS and/or the Primary VNA are down?

A: Once again, this is highly variable, and there are several options.  [1] If the designated workflow sends new data to the PACS first and that PACS goes down, then I’d argue that the new data should be sent to the onsite VNA.  That means changing the destination IP addresses in the modalities.  [2] Vice-versa if the designated workflow sends the new data to the VNA first.   Most of the better VNA solutions can configure a small instance of their VNA application in what I refer to as a Facility Image Cache (small server with direct-attached storage).  One of these FIC units is placed in each of the major imaging departments/facilities to act as a buffer between the Data Center instance of the VNA and the PACS.  [3] In this case, the FIC is the Business Continuity back-up to the PACS.

If both the PACS and the local instance of the VNA are down, the new study data should probably be held in the modality’s on-line storage, for as long as that is possible.  The modalities could also forward the data across the WAN to the Secondary VNA in the second data center, but the radiologists would probably find it easier to access and review the new study data from the modality workstations.

Of course all of these back-up scenarios are highly dependent on the UniViewer.  In the case of those PACS with thin client workstations, if the PACS system goes down, the workstations are useless.  In the case of fat client workstations, most are capable of only limited interactions with a foreign archive.  See the next question and answer for additional detail.

Q-4: Do the radiologists read new studies at the modalities and look at priors using the UniViewer whose rendering server is located in the offsite data center?

While that is possible, my recommendation would be to use the UniViewer for both new and relevant priors.  Some of the UniViewer technology is already pretty close to full diagnostic functionality, some of the very advanced 3D apps being absent. There are already examples of this use of the UniViewer at a number of VNA sites…not only for teleradiology applications, but also diagnostic review if the PACS system goes down.  My prediction is that the better zero client server-side rendering UniViewer solutions are going to be full function diagnostic within a year.   This is a critical tipping point in the VNA movement…a real game changer.  Once the UniViewer gets to that level of functionality, the only piece of the department PACS that is missing will be the work list manager.   As soon as it’s possible to replace a department PACS with a solid [1] VNA, [2] UniViewer, and [3] Work List Manager, the PACS vendors will have a very difficult time arguing that their PACS (less the Archive and Enterprise Viewer) is still worth 90 cents on the dollar, as they are doing today.

Q-5: Does the EMR, if linked to the onsite UniViewer, have a failover process to be redirected to the offsite UniViewer so that clinicians using the EMR still have access to images through the EMR, or do the users need to have the EMR open in one browser and another browser open that points at the offsite UniViewer which they login to separately?

A: Failover from Primary to Secondary UniViewer should be and can be automated (see 1 and 2 above), if implemented correctly and support by the UniViewer technology.

In conclusion, most healthcare organizations are highly vulnerable to the loss of their PACS, because most PACS cannot be configured with a Business Continuity solution.  That problem can be remedied with a dual-sited, mirrored Vendor Neutral Archive paired with a dual-sited UniViewer.  While most VNA vendors can talk about Business Continuity configurations, their failover and resynchronization processes leave something to be desired.  The reader is encouraged to build a set of real-world scenarios, such as those presented here, and use them to discover which VNA will meet their Business Continuity requirements.  The Request For Proposal (RFP) document that I have created for VNA evaluations has an entire section on Business Continuity and the underlying functionality.