trust SSD market data?
SSD story... why's the plot so complicated?
|new Digital Storage for
Media and Entertainment Report|
Editor:- August 27, 2018 - Coughlin Associates
today announced the availability of its new (14th annual)
Storage for Media and Entertainment Report - 2018 - (254 pages, $7,500).
(author and President Coughlin Associates) says...
benefited from input from many experts in the industry including end users and
storage suppliers, which along with economic analysis and industry publications
and announcements, was used to create the data including in the report. We have
made modifications to earlier reports to better model current market conditions.
As a result of changes in the economics of storage devices higher performance
solid-state storage will play a bigger role in the future. The cloud and
hybrid storage including the cloud will be a bigger part of the media and
entertainment storage market going forward."
Among other things the press release about the new report includes these
- By our estimates, professional media and entertainment storage capacity
represents about 4.5% of total shipped storage capacity in 2017
- In 2017 we estimate that 71% of the total storage media capacity shipped
for all the digital entertainment content segments was in HDDs, with digital
tape at 22.7%, 4.3% optical discs and flash at 2.0%. Flash memory dominates
cameras and is finding wider use in post production and content distribution
also:- other storage
market research companies
- Overall cloud storage capacity for media and entertainment is expected to
grow about 13.3X between 2017 and 2023 (5.1 EB to 68.2 EB)
Netflix - the latencies behind flash caching
July 24, 2018 - A recent article -
of Application Data Caching : From RAM to SSD - on
Netflix Technology blog -
discusses their experience of using SSDs instead of pure RAM for caching data
(which in their case is mostly streaming videos).
This trend of
flash replacing RAM in enterprise caches was a hot topic with my readers
in 2007 when
flash SSDs were approaching the tipping point of replacing
we know - flash won.
PCIe SSD market and
demands for server based flash caching were the key enablers in that victory
and for the formation of an independent
SSD software market.
Notable ISV pioneers in the flash caching applications arena were
IO Turbine - both
founded in 2009.
The technique of using flash to replace RAM is
now standard practice and thrives in many SSD form factors.
that the low latency portion of this market could become plug compatible with
DRAM (and replace most of
the DRAM market) - were a factor in the flare up of
SCM DIMM wars in 2015.
The business lessons learned from the
fizzling out of "SCM DIMM wars 1" - despite the
shortages of 2016/17 which should have assisted the flash tiered as RAM
concept - were that were that although "flash tiered as RAM" does
indeed provide useful outcomes it is in fact usable across a much
broader range of latencies than just the ultra-low latencies which the
NVDIMM-focused marketers had hoped would dominate these design-win
That's another reason I find the recent Netflix
technology blog interesting - because (although our industry talks much about
single digit microsecond memory fabric latencies) the Netflix case study
shows that high double digit microsecond latencies can be good
enough to replace most RAM.
observed during experimentation that RAM random read latencies were rarely
higher than 1 microsecond whereas typical SSD random read speeds are between
100500 microseconds. For EVCache our typical SLA is around 1 millisecond with a
default timeout of 20 milliseconds while serving around 100K RPS. During our
testing using the storage optimized EC2 instances (I3.2xlarge) we noticed that
we were able to perform over 200K IOPS of 1K byte items thus meeting our
throughput goals with latency rarely exceeding 1 millisecond. This meant that by
using SSD (NVMe) we were able to meet our SLA and throughput requirements at a
significantly lower cost."
Starting from that base - Netflix then
went a stage further in their software implementation and were able to achieve
typical latencies (with SSD rich caches) under 100 microseconds. ...read
Tegile's flash goes to Hollywood
Editor:- March 10,
2017 - A new blog -
Goes to Hollywood by Brandon Farris,
Director of Marketing Tegile
Systems says among other things...
MLB Network, for example, uses Tegile
flash storage in their post-production environment. During the regular season,
they need to record all of the games and produce content for shows like MLB
Tonight, The Rundown, Intentional Talk, MLB Now, and Quick Pitch, which focus on
the day's activities and give a snapshot of what's going on around the league.
In the off-season, they produce several countdown shows, individual player
profiles, and other programming that goes behind the daily game and into more of
the storytelling about baseball. That's over 500,000 hours of digital content!"
memory intensive data streaming / rematerialization architecture
- unstealthed by Symbolic IO
Editor:- May 25, 2016 - 1 petabyte
usable storage in 2U along with a
flash backed RAM
rich server family which uses patented CPU level aware cache-centric data
reduction to deliver high compute performance are among the new offerings
today by Symbolic IO
which has emerged from stealth mode.
Giving a clue to
performance - Founder & CEO, Symbolic IO - Brian Ignomirello said - "One
of our early tests, allows us to run a full cable class content delivery network
over 80+ nodes, while streaming 80+ full-featured movies simultaneously on one
channel and requires less than 8% of the CPU capacity and we had plenty of
headroom to run more. IRIS (Intensified RAM Intelligent Server) is 10,000
times faster than today's flash."
Editor's comments:- I
hadn't spoken with Symbolic IO (when I wrote this) but my first impression was
that the company is in line with at least 3 strategic trends that you've been
reading about on StorageSearch.com in recent years:-
company profile summarizes their capability like this...
IO is the first computational defined storage solution solely focused on
advanced computational algorithmic compute engine, which materializes and
dematerializes data effectively becoming the fastest, most dense, portable and
secure, media and hardware agnostic storage solution."
about the company's background see this article -
IO Rewrites Rules For Storage on Information
From the marketing point of view it's interesting to see that
in its launch press release Symbolic IO positions itself in the
context in this way "IRIS... is 10 times faster than 3D XPoint."
IO says the new systems will be start to become generally available in late Q4
segmentation viewpoint the IRIS systems will be proprietary. There is space
for such approaches in the future market consolidation roadmap because not
everyone needs the fastest performance. But many
webscale SSD companies
are already using data reduction techniques for their own utilizations and
The new thing - if there is a new thing - is
that Symbolic IO will make available boxes which incorporate modern data
architectures from a single source.
Although like all new systems
companies they'll have to wade their way through the apps accreditation and
compatibility lists before their revenues create any ripples - an adoption
dampening factor I wrote about in my 2013 article
in March 2018 - Symbolic IO exited the hardware supply business and changed its
name to to Formulus Black.
what were the new ideas in the SSD market in 2015?
December 15, 2015 -
SSDs were one of the new big SSD ideas of
They're now part of the establishment. (Even judges know what they are and why
year-end review on StorageSearch.com
...read the article
- new ideas to assimilate
- big ideas to unlearn and forget
- the trends which will dominate our strategic SSD thinking in 2016
Video footage accounts for 100 Exabytes per year of new storage
September 17, 2015 - "Video footage accounts for 7% of the total
storage sold worldwide for any reason" - that factoid is from a paper -
the firehose of media files (pdf) by a media management company called
Efficiency is important for web scale users - says Coho
October 9, 2014 -
as a file system - a web scale case study - a new blog by Andy Warfield , cofounder
and CTO - Coho Data
- made very interesting reading for me - as much for revealing the
authoritative approach taken in Andy's systematic analysis - as for the object
of his discussion (Facebook's storage architecture).
It reveals useful
insights into the architectural thinking and value judgments of Coho's
technology - and is not simply another retelling of the Facebook infrastructure
it you may get different things out of it - because it's rich in raw
enterprise ideas related to
dark matter users.
All of which makes it hard to pick out any single quote. But here are 2.
- re -
match between enterprise products and user needs
says - "In the past, enterprise hardware has had a pretty hands-off
relationship with the vendor that sells it and the development team that builds
it once it's been sold. The result is that systems evolve slowly, and must be
built for the general case, with little understanding of the actual workloads
that run on them."
There are many more I
could have chosen. ...
read the article
- re efficiency
Warfield says - "Efficiency is important. As a rough approximation, a
server in your datacenter costs as much to power and cool over 3 years as it
does to buy up front. It is important to get every ounce of utility that you
can out of it while it is in production."
new article on enterprise SSD pricing
18, 2014 - in a new article on StorageSearch.com
the Astrological Age of Enterprise SSD Pricing - I explain why I think that
2014 will be seen as the start of a new phase of creativity in the enterprise
SSD market on the subject of pricing and affordability and I name 3 companies
leading this charge.
It shouldn't come as any surprise that the
dominant form factor leading this new market trend is
because as I told you in this article last year -
directions in rackmount SSDs - that's the most productive form factor for
doing something efficiently
different with technology.
And the simplest way for vendors to signal
to the world that they are masters and commanders of the solid state storage
high seas - rather than merely floating barges of nand flash which can be
swept along in any direction by the latest technology gust - is to hoist new
colors of SSD pricing. ...read the
say hello to CacheIO
Editor:- June 10, 2014 - CacheIO today
announced results of a
benchmark which is
described by their collaborator Orange
Silicon Valley (a telco) as - "One of the top tpm benchmark results
accelerating low cost iSCSI SATA storage."
CacheIO says that the 2
million tpm benchmark on CacheIO accelerated commodity servers and storage
shows that users can deploy its flash cache to accelerate their database
performance without replacing or disrupting their existing servers and storage.
Editor's comments:- The only reason I mention this otherwise
me-too sounding benchmark is because although I've known about CacheIO and
what they've been doing with various organizations in the broadcast and telco
markets for over a year - I didn't list them on StorageSearch.com before.
was partly because they didn't want me to name the customers they were working
with at that time - but also because with
SSD caching companies
becoming almost as numerous as tv stations on a satellite dish - I wanted to
wait and see if they would be worth a repeat viewing. (And now I think they
SanDisk ships 350MB/s write CFast for cameras
September 13, 2013 -SanDisk
it is shipping one of the first SSDs based on the
2.0 standard. Aimed at applications such as cameras - SanDisk's Extreme Pro
CFast 2.0 card has upto 120GB capacity (MSRP $1,809), R/W speeds upto 450MB/s
and 350MB/s respectively.
SSDs in TV
Editor:- November 21, 2011 - An
article in TVTechnology.com
discusses the state of play for SSDs in the tv industry.
one of the statements in the article - about the readiness of SSDs for the
enterprise market is
incorrect in suggesting that consumer SSDs came first. I'd like to know which
consumers were spending $40K for an entry level SSD back in the 1990s?
Nevertheless the article shows what some people in the tv equipment market are
AMP launches CineDrive
Editor:- June 29, 2011 - AMP launched the
a low power, rugged 2.5" SATA SSD optimized for the video recording
the future of data storage
Editor:- January 23, 2011
future of data storage is the lofty sounding but aptly chosen title of a
new article published online today in Broadcast Engineering -
written by Zsolt
Kerekes editor of StorageSearch.com
It's a completely new article which synthesizes and
integrates concepts from several futuristic articles which have already
appeared here on the mouse site and wraps them into a cohesive whole. Anyone
who reads it will get a clear idea of where the incremental changes they read
about in storage news pages (like this one) are likely to end up. ...read
Who Needs Smart TV?
Editor:- January 13, 2011 -
Needs Smart TV? is the subject of a new blog by Michell Prunty at
SSDs cheaper than HDDs in some new DVRs
September 9, 2010 - SanDisk
that NDS (a tv set top box designer
with with over 30 million DVR units deployed) has successfully has designed
SanDisk SSDs into a new range of lower cost set-top DVRs.
The new solution allows for the deployment of energy-efficient
STBs with decreased power consumption, small form factors and virtually silent
SanDisk says its SanDisk P4 is substantially less expensive in 4GB,
8GB and 16GB capacities than
hard disk drives, which
are often only available in higher capacities. Editor's note - This floor price
SSD penetration model was 1st proposed in the 2005 article by Jim Handy -
Flash vs. Hard Disks -
Which Will Win?
"For the first time, SSDs are now a
cost-effective alternative to hard drives in the STB market," said Doron
Myersdorf, senior director, SSDs, SanDisk.
flash storage has been used in STBs for a long time.
Samsung shipped 117
million units of its 0.5GB OneNAND devices in
2006 into various
markets including STB.
Low acoustic noise was mentioned as a
desirable factor in the world's 1st
SSD user preferences
survey (2004). In those days - because of the
cost of SSDs - we
were talking about silent running in submarines.
For DVRs with
terabytes of storage - HDDs will remain a lower cost option for a while. And
you will probably continue seeing HDDs in new consumer products long after HDDs
have disappeared from the datacenter - because the cost of ownership and
performance envelopes will diverge.
Fusion-io accelerates 3d movie releases
28, 2010 - Fusion-io
how it's been helping Hollywood's 3D movie makers.
have been used in the production of:- Alice in Wonderland, Avatar, How to
Train a Dragon and Clash of the Titans.
Utilizing Fusion-io, the S3D conversion of Clash of the Titans took
under 8 weeks instead of typically 6 months using conventional systems.
"Fusion's ioDrives really helped us meet a tough deadline"
said Sean Konrad, Project Manager for Stereoscopic Conversion on this project at
Focus. "With typical VFX projects, we complete about 20-30 shots per
day. The ioDrives (from
Fusion-io) scaled our data load capabilities by about 20x. Using
Fusion-io drives, each review session covered far more shots and we could
continue cycling work in between sessions. We don't have to worry anymore about
defragmenting disks, rebuilding
RAIDs, or swapping out
disks. We are buying many more ioDrives to implement throughout our different
Nexenta streams online tv
Editor:- May 20, 2010 -
that its products (which include
SSD ASAP features)
are being used by the Dutch Public
Broadcasting Agency NPO for storing and delivering online tv in a
configuration which includes 192TB of
hard disk drives and a
1.9TB SSD read cache.
broadcaster's website has approximately 80TB of video available to online
users who want to watch previously broadcasted television programs. During an
average evening, between 10 and 20,000 people stream data, adding up to 25GB in
capacity. The customer (who evaluated multiple vendors ) says that important
selection criteria were:- performance, price, support and power
|older articles and whitepapers
- also worth reading
|After another hard day working
the web Megabyte
liked to sit back
and view someone else's efforts.
|SSD Pricing -
where does all the money go?|
|SSDs are among the most
expensive computer hardware products you will ever buy. |
the factors which determine SSD costs is often a confusing and irritating
|| ...not made any easier when
market prices for identical capacity SSDs can vary more than 100x to 1!
Why is that? ...read
|this way to the Petabyte
|In 2016 there will be
just 3 types of
SSD in the datacenter.|
of them doesn't exist yet - the bulk storage SSD.
It will replace the
last remaining strongholds of
hard drives in the
datacenter due to its unique combination of characteristics, low running costs
and operational advantages.
||The new model of the
datacenter - how we get from here to there - and the technical problems which
will need to be solved - are just some of the ideas explored in this