Optical Libraries

Archive Storage: Evolving into Mainstream Storage Solutions

Article by Roy Slicker and Jim Wheeler Pegasus Disk Technologies

Historically and for reasons of technology, primary and secondary storage solutions have been separated due to the types of media being used. Primary storage vendors focused on magnetic redundant array of independent disk (RAID) for quick access to massive on-line databases while providing some level of fault tolerance. Those same storage vendors use magnetic streaming solutions to back-up or off-line those same massive databases.  Secondary storage vendors take a different approach, those supplying application providers in the financial, medical, insurance, securities, government and other critical long term storage applications, focused on media that supplies longer shelf life in near-line storage solutions, such as optical, CD, DVD and in some cases even tape libraries.

The need for these fault tolerant near-line storage solutions is because digital records are prone to becoming lost or damaged just like paper. It is all too easy to misplace a single bit, even erase or damage it unknowingly during the life of the stored data. If this happens there is the chance that the software's ability to retrieve and read that data accurately is lost, and thus the record could be lost… Forever! Archiving is not solely about saving space on your RAID system. Hard disk storage is becoming very affordable. The point of archiving is to retain your data for future use. It is pointless to archive if you cannot retrieve any piece of your data quickly and efficiently. Without the technology within the system to be backwards compatible and portable over a number of file formats to include future file formats, the lifespan of a record or a given database can be shortened to just a few years. Many of the applications mentioned above require storage specification of tens of years or even hundreds of years. Magnetic media has not been proven to have that kind of a shelf life, thus the need for near-line optical based solutions with shelf life specifications to meet the needs of long-term archive applications.

Now consider for a moment when talking about databases and database activity, how much data in the repository is stagnant? As a rule of thumb, the 80/20 rule applies. Only 20 percent of the database will ever need to be accessed after the first 90 days. This leaves 80 percent of the database stagnant. Yet it is not purged or discarded, because this stagnant data may have some future value, or it may have a regulatory requirement to be kept for a period of time. Now consider what is paid by a large company in primary storage (RAID), and consider that up to 80 percent of that data could be moved to a near-line solution that is cost efficient to expand yet is still accessible as an on-line application. This is secondary storage.

This issue comes to play in that these two storage segments, primary and secondary, have been addressed as separate issues. This has lead to software management tools that are stove-piped based solutions and have limited cross communications: leading to separate software, separate hardware, and a separate infrastructure to manage the archive data separately from primary data.

Imagine if a storage solution provider could integrate a total storage solution that addresses all the primary storage needs of an ERP solution; AND improve the efficiency of that solution with the integration of tools that address all the needs of a near-line storage solution to migrate and address archive data in a manner that the data is accessible on-line; AND this same solution could manage the back-up processes as well from a single console. How would such a solution impact the IT manager's day-to-day activity? How would such a solution affect the costs of ERP implementation? How would such a solution improve system performance? These are the kinds of questions that Pegasus Disk Technologies is addressing.

What do storage vendors need to address to meet the evolving needs of users?

• Single point management—the ability to manage all storage devices from a single access point.

• Migration— the ability to access, import and read legacy file formats.

• Portability and technology obsolescence protection — the ability of the archive file system to easily integrate with future hardware and software systems.

• Conversion— the ability to migrate proprietary legacy file formats to current archive file formats without losing data integrity or wasting valuable storage space and/or system performance.

Single Point Management: The InveStore interface provides a virtualized view of all hardware attached to the storage enterprise. This interface allows for a single share point to all supported storage hardware, making interaction between devices as easy as drag and drop or copying between directories. This interface results in compatibility with many storage technologies from multiple vendors, allowing the user to migrate archive data from older legacy hardware to any supported archive storage product. Furthermore, the virtualized interface has another very important benefit - ease of integration with current applications. The process of adding archive application software to existing applications running other storage devices is much easier for the solution providers, because the interface is the same. Solution providers continue to be concerned about the time required to add new hardware and software to their system. The

InveStore interface would shorten the integration cycle. Supported hardware becomes plug and play with the integration of InveStore, allowing integrators to focus more time on selling and less time on integration.

Portability: Pegasus has maintained operating system independence through the use of standard programming languages and the isolation of system specific function calls into separate low-level modules. At the core of all Pegasus storage management software is the Pegasus Archive File System (Pegasus-AFS). The Pegasus-AFS supports multiple industry standard, 3^rd party and Pegasus high performance file formats on various storage media. Its interface to individual operating systems, such as InveStore for Microsoft Windows, provides transportability across operating system platforms. Many archive applications out-live operating system versions. Transportability/migration of data across platforms has become a key issue for many of the major solution providers and their users.

Migration: InveStore can support multiple physical formats interactively on various different storage media through the use of the Pegasus-AFS. This provides OEMs or ISVs that have written proprietary file formats the ability to support those formats through the development of a simple DLL and linking it to the Pegasus-AFS, rather than developing an entire file system and drivers for new archive hardware. Additionally, the default file system can simply be used moving forward, while legacy data is still accessible on the older formats. The Pegasus DLL approach allows developers to spend more time to develop feature rich front-end applications, which will allow for more feature rich product.

InveStore features include:

• Volume spanning

• Transaction based logging

• Multiple data streams

• 64-bit file size

• Roll back (ability to undo any change to the file system)

• 256 byte file names

• Complete historical preservation of all changes to the file system

• Mixed media type support

• Fault tolerance

• Deep directory paths

• Very low overhead

• Elimination of allocation tables

• Sparse files

• File branching

• File links (hard or soft)

The file system was designed to be easily extendable in order to support new features.

Conversion: The InveStore interface provides a virtualized view of all hardware attached to the storage enterprise. This interface results in compatibility with many storage technologies from multiple vendors, allowing the user to migrate archive data from older unsupported hardware to any supported archive or primary storage product. Furthermore, the virtualized interface has another very important benefit - ease of integration with current applications, therefore reducing the sales efforts. The process of adding archive application software to existing applications running other storage devices is much easier for the solution providers, because the interface is the same. Using the InveStore interface shortens the integration cycle. Supported hardware becomes plug and play with InveStore. Overall, this results in more pull for the OEM or ISV rather than a sales force trying to push a product that requires integration or system level development.

Pegasus sees the value to blend the various types of hardware and virtually managing those devices as a single repository. Giving the functionality to the IT manager while relieving the IT manger of the headache of having to manage the devices on concurrent systems. Giving the power to IT managers to move stagnant data off their high access systems, cleaning up their network and migrating archive data to a suitable archive solution, while retaining continual access to all the data on the system.

...Pegasus Disk Technologies profile