the SSD software event horizon - how better SSD utilization will change the enterprise and markets too


leading the way to the new storage frontier	.....


SSD news	..


SSD software	....


top SSD companies	..

PCIe SSDs

.....

meet Ken - and the enterprise SSD software event horizon

by Zsolt Kerekes, editor - October 8, 2013

In this article I'm going to introduce a new and important marketing concept related to enterprise SSDs which - for the sake of simplicity - I'm calling the "SSD software event horizon."

If you're familiar with the Hitchhikers Guide to the Galaxy - you may be wondering - does this have anything to do with the shoe event horizon?

Yes it inspired the name. At the heart of both these "event horizons" is the idea of cataclysmic economic consequences following a boom market.

No - inasmuch as the SSD software event horizon doesn't spell the permanent end of SSD business growth as we know it - although it may feel bad enough at the time.

I'm also going to introduce you to a reader called Ken - who - by asking the right questions - forced me to put a name to an important structural change in the SSD market that I have discussed with other readers before and have hinted at in earlier articles but haven't written about before so explicitly.

As for Ken... what can I say? In many ways he's similar to a lot of readers I talk to. He takes the SSD market paradigm really seriously. It's important to his career to get it right.

What do we mean by enterprise SSD?

Oh - before I go on - and in case there's any doubt about the context here - when Ken and I are talking about "enterprise SSDs" it means all the SSDs which you'll find in a rack or shelf in a rack or cabinets inside an organization. Whether the SSDs sit in servers (like PCIe or SAS) or on the SAN (or other fabric) makes no difference. "Enterprise SSDS" specifically excludes:- desktops, notebooks and portable devices - which - for the purposes of market sizing (TAM) are best analyzed using different user value propositions and market adoption models.

Anyway - let's get back to Ken.

Unlike many other readers whose introductory email often begins with something like - "I've been a great fan of your site StorageSearch.com - and have been reading your articles about SSDs for many years..."

And unlike readers who ask me questions with the promise of rewards if I can help them couched in phrases like this - "I'm working on an important strategic project which involves the SSD market and I'd like to understand it better. My organization would be happy to pay you $MCMXC / hour (an extravagantly ridiculous rate) for a few hours / days of your time etc..."

Ken just sent me this...

"I work at a financial institution which has started to cover the storage market - and in particular enterprise SSDs. I'm trying to estimate:-

how big will the SSD market will be when SSDs replace hard drives?

when will it happen? and

what will be the revenue of the SSD market at that time?

"I saw your article about distrusting SSD market projections - and I agree with that premise. So my starting point is the knowable now. I've started with market reports about the size of each market now - and done my own projections from there. But however I do it - and whatever assumptions I use - I don't see how SSDs can replace HDDs in the timescale you've talked about in some of your articles. For example you said in one article that in the enterprise - SSDs would replace / displace about 50% of the revenue of the HDD market from around 2016 and then all of it by 2020.

"I can't see how that can happen based on what's known now - and flash capacity and the timescales and investments needed. It looks to me like Seagate may be right - there isn't enough SSD manufacturing capacity. Or am I missing something?. Please can you help.

BTW - I'm new to the storage market - and only started this job last month - but I want to get it right. Am I going wrong? Or are you in fact the one who is wrong? Cheers."

I get many emails like this. And although it's flattering to my ego if people say nice things about the site, or offer me lots of money - my time is limited - and I never accept monetary or other inducements - my decision about investing time depends on how busy I am and whether the way that the question is posed reveals a hole in my content which I think deserves to be filled - and whether - by doing so - other like-minded readers would benefit.

I always tell readers who ask these types of questions - that I'm interested in knowing what they understand about the SSD market - and seeing if there are any structural holes or weaknesses in their understanding that I can help them with - by sending them links to articles which patch their understanding - rather than just telling them what I know.

And another thing - which I tell readers like Ken is - you're often better off not having had any previous experience or knowledge of enterprise storage or the enterprise server markets - because that means there's less to unlearn when examining things from an SSD perspective.

It's unusual that someone who is new to the market thinks I may have got some of these market growth ideas fundamentally wrong.

Anyway - I agreed to help Ken - on condition that I could share what was learned from that dialog with readers in a future blog. (This one.)

I know that Ken is in a business where what you know means money. So I agree in advance not to publish any data or numbers or insights which he supplies to me - but only to publish the broad outline of any insights which I have helped him with. (Also he doesn't want me to name his company. But a lot of you would recognize the name.) He agreed - so here's a summary.

I asked Ken to take me through how he got to his conclusion that SSDs couldn't replace 50% of hard drives at the date he was referring to. (That 2016 projection was actually published back in 2010 - which was before some of the products needed to do this were designed - and before some of the companies who are going to lead this change were founded. So there is some fluidity in the date - seen from the perspective of 2013 - but it's still as good a date as any other for examining the principle.)

Editor:- How did you calculate there wouldn't be enough SSDs?

Ken:- I started with known shipments for enterprise hard drives in the most recent year. Then I factored in some growth assumptions for storage capacity for the enterprise and cloud etc That gives me a figure of XYX petabytes of storage. Then I went back to the most recent numbers for flash memory and SSD production. Even if I assume wild growth rates for enterprise SSD - and assume that nearly all flash memory will be used for SSDs - then after I take out known demand for consumer SSDs, phones etc - I end up with an estimate of enterprise SSD capacity - based on current wafer fabs - which falls well short of what is needed. I can't replace all that hard drive capacity with SSDs. Not even close.

Editor:- It's easy to see where you've gone wrong then. And it's the same mistake Seagate and others have made when they analyzed the enterprise SSD market from a hard drive perspective. It's because when you're looking at storage on the SAN (or the equivalent) and when you get to 2016 (plus or minus a bit) then you don't need a petabyte of SSD to replace a petabyte of HDD.

The exact ratio for operational replacement - depends in detail on where the SSD software market is - with its developments - and where the controller market and rackmount SSD markets are with their state of technology enhancements. But even without 3D flash - there will be enough SSDs - because I estimate the replacement ratios could be somewhere between 10x and 50x.

Because what you're looking at is the virtual storage which is needed to make things work in an SSD world - which is very different to the capacity needed in the HDD world.

Many factors are involved - and it's the sum total of them all - in a systems context which mean that you need less SSD (capacity) to replace enterprise HDDs.

And the ROI etc cost comparisons mean that whatever the theoretical cost of HDD capacity declines to - and even if it declines to zero dollars per terabyte (as I said in 2010 in my article - this way to the Petabyte SSD) users in the enterprise will choose to make the switch at a date well before the cost per terabyte of flash and magnetic storage graphs begin to converge.

The main factors which close the gap you're looking at have mostly already been described in previous blogs.

In particular SSD design efficiency - at the SSD controller and system level. This includes the contributions from adaptive R/W flash management, big controller architecture and the impact of systems software.

Within the scope of the above are also calculated the comparative changes in RAID architectures. These advances aren't unique to SSD - but the necessity for innovation in SSDs has meant that newer ways of offering RAID like (or better) fault protection within SSD drives and at the array of drives level - means that these techniques will become the norm in large SSD arrays whereas the less efficient, older ways of doing RAID are still likely to hang on longer within legacy HDD installations.

Dedupe and compression are some of the best known software efficiency contributions which are more viable within an SSD array context than an HDD array. But it's only the beginning rather than the end of the effect which SSD software will have on the market. The bigger effect is the sum effect of many long term changes which for simplicity I call the SSD software event horizon.

Ken:- before you get onto a new idea - let me take you back to the question of calculating how big the enterprise SSD market will be - measured by revenue. And also - how do we calculate the effect of SSDs in servers?

Editor:- Servers is the easy part. I published the results 10 years ago and the principle hasn't changed since then.

You start from a server market in which there was zero or nearly zero percentage installation of servers in the world. (Depends on the publication date of your server market shipments report.) That's your baseline for the revenue you can replace. In an SSD accelerated world - in legacy software environments (traditional VDI, Oracle, email, webserver etc) you can aim to satisfy more users and run faster while using only 1/3 to maybe 1/2 the original number of servers.

With newer software - a mix of legacy software with some new APIs - that ratio gets better. And with newer SSD software - which bypasses or replaces a lot of that old hard drive interface emulation and queuing management etc - it will be realistic to extend that ratio - which means that 1 new SSD enhanced server will be able to replace 5 or 10 legacy HDD based servers.

That step in improved utilization from 2x to 5x - due to next generation SSD software - also means that when users make the switch to the newer software - not only do they need less servers - but they also don't need as many SSDs as they did in an earlier phase of SSD market adoption (to get the same user workloads done).

That's a painful symptom (for vendors) of the "SSD software event horizon." It means that customers who make those transitions in their software and SSD infrastructure - don't need to buy so many SSDs (for a while).

Some vendors have already seen that revenue crash happen - the most notable being Fusion-io. However it will affect all enterprise SSD vendors at some time or other.

Fortunately for the SSD market - the switch from generation X software to generation Y software - will be spread across the user base over a period likely to be 3 to 5 years. And the effect is a temporary dip in demand from some customers.

Eventually that demand itself will grow again - because SSD enabled servers are the oxygen of business and they will create new markets. But those new SSD markets are currently unquantifiable.

Ken:- so what can you quantify about the market size for server based SSDs?

Editor:- I'm just saying (as I have been saying since 2003 in my article on SSD-CPU equivalence) why I think the TAM for server based SSDs is a percentage of the server market - and almost entirely decoupled from the cost of (fast-enough) storage capacity on the SAN.

Even if you don't factor in the savings in software licenses and floor space and other running costs which you get from switching to a 100% SSD populated server infrastructure - the savings you get from buying 1/2, or 1/3 or 1/5 the number of servers to do the same jobs faster - is the starting point for calculating the value of the SSDs which make this possible.

The value of the server-side SSDs depends on how those cost efficiencies are valued in the market. You can either scale up from what's happening currently in the market to calculate a figure - or another way is to assume a percentage discount across the whole server market. What you're really calculating is how much a user will pay for a new server which runs the same workload at 2x to 5x previous servers. In the long term - in another 2 server replacement cycles - then you'll simply be replacing one type of SSD accelerated server with another. Or even repurposing and re-energizing 1st or 2nd generation SSD accelerated servers with new software.

Ken:- going back to estimating SSD revenue on the SAN etc....

Editor:- For the reasons I've indicated - there will be enough SSD capacity to do the job.

The use cases for all the different types of SSDs in the enterprise are described in my SSD silos article. And depending on how detailed you want to make your market revenue projection - you could include "fast" and "ultrafast" SSD racks into the same pot as server-based SSDs - because they effectively perform the same job - apps acceleration and reducing the number of servers required. - Whether they sit in the CPU box or on the SAN - their main benefit from the user value proposition viewpoint is the same.

Ken:- OK that was helpful. So what more can you tell me about the SSD software event horizon? Sounds like a big deal for the market. Why aren't more people talking about it?

Editor:- It's not really a single event or a point in time - but a it will be a transition in the market - which only becomes possible when the SSD market has grown to a critical mass which makes the investments in writing and testing new software economically viable.

What is the SSD Software Event Horizon?

I've called it the SSD Software Event Horizon - because the root cause is software. It's an inflection point which occurs in the growth of SSD hardware sales caused by improvements in the apps utilization of SSDs - which is enabled by a more efficient newer generation of software.

The new software doesn't have to come from the same place as the SSD - and the software may have already existed at an earlier date than when it has the effect (on shrinking SSD revenue growth).

The inflection point - needing less SSDs of the same type to get the same job done - is triggered by when the user base - customers start to adopt the new software.

In the real world of the enterprise there are always impacts from new systems software in the utilization of hardware resources. In the case of the event horizon - it's probably when the cumulative effects of SSD utilization from using the new mix of software impact the SSD utilization in the region of 30% to over 100% improvement which is the danger zone for vendors - because at that point the "SSD system" while becoming more competitive in the market sense (and offering better value than mythical competitors) also starts to compete with itself and cannibalize sales.

The inevitable result is that a customer - which may have been on a buying roll and ramping up its adoption of SSDs - suddenly discovers that it can get by without needing any new hardware for a while - because the new software regime has improved its SSD utilization (from the apps perspective). So they shut off the orders for new SSDs until they get to the point where they need more. The result is a crash in sales for the SSD vendor - because the improvement in SSD utilization caused by the new software is bigger than the previous growth rate in customer hardware purchases.

This event horizon can impact SSDs on the SAN just as much as server-side SSDs - although the exact software mechanisms may be different.

The root cause is shifting from apps software and systems software models which were originally HDD-centric - and emulating the HDD world in SSDs - towards doing the same (or equivalent) virtualized transactions for a new dynasty SSD environment.

Since 2009 - when the first really useful enterprise SSD software started to become available - SSD makers have come to regard software as their ally - because it was the availability of easy to use caching, tiering and virtualization software which made it much easier to sell enterprise SSDs into the hard drive world. Software remains a critical enabler of the SSD market - but the evolution and adoption of better and more SSD-centric software will have dramatic consequences on the market.

On the one hand - when users who are already heavy SSD users adopt newer SSD software - it means they need less SSDs than they thought they needed before.

On the other hand - the cumulative effect of hardware and software efficiencies still emerging within the SSD industry improves the competitiveness of SSD based systems to the point where - apart from mental laziness and inertia - there will soon be no good economic justifications for buying new storage or server systems for the enterprise which don't include SSDs.

Ken:- so the long term effect of the SSD event horizon will be to make enterprise SSDs even more attractive. I get that. Bad news for the hard drive systems guys. But along that road - all SSD vendors will hit painful bumps in their growth right?

Editor:- depends how reliant their sales are on single large customers. Because all end users are at different starting points and have different speeds and needs of adoption - the revenue crashes don't have to be painful. But startups and pure play SSD companies are more heavily exposed to the downsides in their customers SSD needs because they have less customers.

Ken:- OK I've got it. It's back to the powerpoints for me with some new assumptions. Thanks for your help. Cheers.

Zsolt:- Glad I could help. Good luck. (end of call)

So where does that leave us now?

This article (and conversations with Ken) started with a need to explain where the missing SSD production capacity was going to come from. And it's a no-brainer to see that if and when flash supply becomes constrained from traditional memory sources - that output can be switched to change the balance from less profitable consumer markets to the higher ASP enterprise - which is the main reason that you've seen memory makers rushing to set up their enterprise assets and credibility recently. There are other sources too. Easy to figure out from the powerpoints and semiconductor market data.

As you've probably guessed by now - Ken is a semi-fictional character who I invented for the purposes of this blog. Before Ken came along I was struggling with 3 different ways to start this article and carry it along - all of which were less satisfactory than what you've got now.

But the content of the conversation I've described with Ken is very similar to real conversations I have nearly every day.

The main difference is that (unlike Ken) the readers who approach me from the investment community - have for the most part - already done a lot of research - and often know much more financial detail about the SSD companies they're tracking than I do - and they spend more hours tracking each company than I could ever do. But what they sometimes lack - despite reading lots of market reports - seeing the conference webinars and often writing their own SSD reports too - is a conceptual framework - rooted in technology - of how the different details fit together and also a credible roadmap for projecting SSD market trends forward.

I've been living in the SSD village since before it appeared on most maps - and like the old women in the villages where my parents were born - I've seen people come and go - and have been asked so many times for directions - that from time to time I scribble something down and tack it to the wall.

But the effects of sun, wind and rain, and all those developers who come in and dig roads up, build new houses and then knock them down again - mean that from time to time the old maps don't show half of what is really there anymore. Never mind - ask me where you're going and I can easily draw a new one. (But you have to catch me first.)

About the publisher - 21 years guiding the enterprise market

....

In the early 2000s when HSM was the fashion in storage software - I asked some of these new companies - where's the SSD layer in your tiering model?

It seemed to be missing from all their pyramid diagrams.

It was missing from their thinking too.

Where are we now with SSD software? (2012)

"In the most recent quarter (ending January 31, 2017) we had more than one customer running large scale simulations and analytics replace over 20 racks (think 20 refrigerators of equipment) with a single FlashBlade (at 4U about the size of a microwave oven).

Such dramatic consolidation depends on storage software that has been designed for silicon rather than mechanical disk."

Scott Dietzen, CEO - Pure Storage - in his blog Delivering the data platform for the cloud era (March 1, 2017)

" ...One of the main rationales for FTLs, which is to emulate hard drives, is becoming an obsolete requirement."

Mike Jadon, CEO of Radian - interviewed in what's the role for a Radian Memory SSD? (November 2015)

The winners in SSD software could be as important for infrastructure as Microsoft was for PCs, or Oracle was for databases, or Google was for online search.

all enterprise data will touch SSDs

In March 2014 - a research paper by Baidu showed that by modifying standard SSDs to be compatible with its workload optimized SDF (which has access into the controller and changes some of the management assumptions) the result is 2x the usable flash capacity and 3x the I/O bandwidth.

All the marketing noise coming from the DIMM wars market (flash as RAM and Optane etc) obscures some important underlying strategic and philosophical questions about the future of SSD.

where are we heading with memory intensive systems?

"I had the benefit of the brick wall of ignorance. Not knowing what couldn't be done."

Skyera's CEO, in the article - scary Skyera

"A critical test of whether you really understand the dynamics of a complex market like enterprise SSDs - is whether you can predict what rational buyers might do when offered new product options at the extreme limits of - for example - price."

Boundaries Analysis in SSD Market Forecasting

after AFAs - what's next?

Throughout the history of the data storage market we've always expected the capacity of enterprise user memory systems to be much smaller than the capacity of all the other attached storage in the same data processing environment.

A new blog on the home page of StorageSearch.com - cloud adapted memory systems - asks (among other things) if this will always be true.

Like many of you - I've been thinking a lot about the evolution of memory technologies and data architectures in the past year. I wasn't sure when would be the best time to share my thoughts about this one. But the timing seems right now. ...read the article

"before long - nearly all flash SSDs will have to use adaptive R/W controller schemes ...."

adaptive R/W & DSP ECC for SSDs - What is it? Who does it? Why will it disrupt business in the SSD market?

hard drives

"Enterprise Flash" - how the market changed from 2004 to 2017

Sugaring flash for the enterprise

Heck no! - whatever gave you that silly idea?

Can you trust SSD market data?

Raw speed is no longer the same guarantee to market success for SSDs as it once used to be. But since you asked...

the Fastest SSDs

"I think Kaminario's K2 v5 product launch (in May 2014) will be remembered for ushering in a new era in enterprise SSD marketing - in which the increased utilization from software - stops being a wishy washy averaged (sometimes you might get it, but other times you won't) concept - and instead becomes a guaranteed figure - like endurance - which users can learn to trust."

Kaminario guarantees amplified usable capacity (May 20, 2014)

Exiting the Astrological Age of Enterprise SSD Pricing

"A new generation of enterprise SSD rackmounts is breaking all the rules which previously constrained price, performance and reliability."

exciting new directions in rackmount SSDs

Leading SSDites don't agree on fundamental tenets of SSD philosophy nor the prophecies of what comes next

the SSD Heresies

"You won't be surprised to learn that almost no-one has ever been able to guess the correct answer to the enterprise flash wizard's 3rd question."

playing the enterprise SSD box riddle game

Usable versus Raw capacity - isn't the only flash size consideration in SSDs. A lot of flash in SSDs is just plain Invisible.

SSD capacity - the iceberg syndrome

"SSD is going down! - We're going down!"

Surviving SSD sudden power loss

"...the SSD market will be bigger in revenue than the hard drive market ever was."

How will hard drives fare in an SSD world?

Memory Channel Storage combines the cost advantages of flash with the low latency legacy motherboard connections of RAM - to create a new type of fast storage device - claimed to be an order of magnitude faster than good old PCIe SSDs.

Memory Channel SSDs - will the new concept fly?

"You'd think... someone should know all the answers by now. "

what do enterprise SSD users want?

Reader comments on SSD event horizon

Endurance is a factor too

Editor:- February 3, 2014 - I had not long ftped up the first draft of my new Top SSD Companies in Q4 2013article - when I got an email from a reader who is an insider in an enterprise SSD company (and prefers to remain anonymous here) who commented on an article I'd written a few months before - the SSD event horizon.

He said - "I had not seen that article before.... really liked it and forwarded it to other people here. One other cause to consider for the event horizon is this.... A real problem is that when it comes time for (system) refresh, the customer looks at the wear life of the flash and observes that it has plenty of available life and decides to keep it instead of refreshing."

Editor's comments:- In case you haven't seen it yet either - the SSD event horizon article is hard to summarize - but - among the things discussed are the predictable revenue crashes which can hit enterprise SSD vendors from time to time - triggered when they ratchet up technical improvements in their products which increases SSD utilization for a homogeneous user base - at a faster rate than the sales ramp was growing before such changes.

Improvements in SSD systems endurance wasn't one of the examples mentioned in my original article - but when viewed from this angle - it is.

The event horizon impact on revenue is something I had also been discussing with another company last week - this time seen from the perspective of dedupe.

One of the ironies of legacy systems software running in flash systems is the way that the data weaves through layers of fossilized unreality where emulation is stacked on emulation - and hardwired into the software and data flow logic are the remembered once-deemed-to-be-efficient solutions to data flow control problems whose origins are now almost forgotten.

So the SSD emulates a hard drive.

And the hard drive emulates memory.

And then it gets worse.

how new SSD software gets things done faster

And here's a follow up on the recursive emulation thing...

Now that flash memory can easily emulate RAM (for example in NVDIMM products like Diablo's Memory1) - is there a case for running RAMdisk emulation software on the emulated RAM too?

"why are so many companies diving into the SSD market - when even the leading enterprise companies haven't demonstrated sustainable business models yet?"

Hostage to the fortunes of SSD