• Backup SW
• NAS
• Solid state disks
• iSCSI
• Serial ATA
• Manufacturers
• US VARs
• UK VARs

Nibble Re: The Storage Software Industry is Still Using 1970s Style Control Theory

Why isn't the solid state disk accelerator market already a 5 to 10 billion dollar market?

I was asking myself this question the other day, prompted by Sun's disclosure at the SunNetwork Conference in September that its next SPARC processor, due to be shipping in systems by January, will once again be only a little bit faster, and not a LOT faster as most of their server customers would really like.

The fact is that if you've already got your main servers stuffed up with the fastest processors from Intel, Sun, HP, IBM or whoever, then installing a solid state disk accelerator in the right place can often speed up the entire storage network by 20 to 30 percent at a cost which may only be 10% or so of the server itself. So by any measure that seems like good value for money, even in today's cash strapped times. Of course solid state disk acceleration is happening here and there, but not yet on a wide scale, and it got me thinking about why.

Part of the problem is that it takes knowledge about where the bottlenecks are in your system and that can change with every new release of your application software. Although it's economic to buy the hardware side of the solid state disk accelerator solution, the soft side still relies heavily on human experts to make the speedup work. Without expert tuning you won't get the full benefit of the expensive hardware, and in a worst case scenario might not get any benefit at all.

That's a big risk for users to take. Users are cautious and will want to trial the new technology or have performance guarantees as part of their contracts.

Solid state disk manufacturers are keenly aware that the shape of their sales ramp is limited mainly by the number of their applications engineers until they can find ways to automate the tuning process of discovering what's in the users' network, modelling it and then optimising it. That got me thinking about how primitive the whole storage software market really is in comparion to where it would like users to think it is. Most storage software nowadays is designed around a control paradigm which resembles heavy industry in the 1970's.

In the 1970's petrochemical manufacturers used thousands of individual electronic controllers to automatically control the opening and shutting of valves which controlled the flow of liquids, gases, heat and other things flowing through the hundreds of miles of pipework in a typical plant. The electronic controllers were a great improvement on their pneumatic predecessors, because they could operate faster and were less liable to jam due to dust or drift due to leaks. But a typical plant still had to employ hundreds of instrument engineers and mechanical fitters to keep the control infrastructure humming, and dozens of "board men" whose job was solely to watch the thousands of meters and dials and alarms and keep an eye on fluctuations before things got too much out of hand.

In the hierarchy the process people and control engineers were the gods who, in their design of the plant and occasional tweaking, determined what type of control strategy would be used, and the quality and repeatability of the end product.

By the mid 1980's most of these heavy industries were on the second or third generation of digital controllers which not only simplified the collection and display of all this critical data, but also facilitated a new type of control strategy based on SCADA (Supervisory Control And Data Acquisition) systems. Eventually software would be trusted to keep an eye on things. And as the systems got more reliable, process designers realised that adaptive control systems could be entrusted to learn what was out there in the plant, measure how it responded and optimise the control over a the entire manufacturing process, not just for a single control loop.

Despite the hype put out by companies like EMC about the abilities of their storage management software we are still very far away from the ideal data storage factory control model. Virtualization is a part of the jigsaw which will eventually lead to an enterprise wide solution. But we're still very much at the level of individual loop control which the oil business was using way back in the 1970s. Your storage devices may be connected by networks but making sure that everything operates together seamlessly to deliver data quickly when people need it still requires a lot of manual intervention.

Until the storage management software you run in your orgazination is intelligent enough to learn by itself what kinds of applications you're running, and the characteristics of your different types of storage devices, your ability to make the best use out of new storage technologies will be limited by your own technical skills and the amount of work and effort you are prepared to put into solving your own performance and resource utilization problems. And having solved them once you may have to still solve them again whenever you buy some new software or hardware which changes the mix.

Adding a solid state disk accelerator or tape library to your system is easy now. You connect it to the network and click a mouse a few times. Telling it to find out what else is out there and optimise its operation chosen from some sensible defaults for that kind of a device should also be a one click process. Telling it to keep itself up to date automatically when new hardware or applications are added should be just as simple.

The challenge for the storage industry is to get out of the age of manually adjusting individual control loops and processes and use tools which are designed for operating a storage network like a data factory. When that happens the solid state disk accelerator market is going to be a 10 billion dollar market, and a lot of other things are going to be affected as well. The end point is to make all your computer assets work better for you, and to make sure that in the meantime you avoid buying products which won't fit in with the hands off storage management strategy.

See also:- SAN - software, Solid state disks