High end solo enterprise PCIe SSDs are under threat of being squeezed in
the gap between DIMM wars, tiered memory architecture, M.2 SSDs and even SATA
Blame cloud architects (the biggest early adopters of PCIe
SSDs) who measure the value of every percentile of incremental apps
performance and every contribution to watts in the box.
Fabrics - market experiences|
|Editor:- March 31, 2017 - The state of the NVMe
SSD and fabric market and its growth expectations are conveniently summarized in
a new presentation -
with NVMe over Fabrics (pdf) - by Mellanox. Among other
- 40% of AFAs will be NVMe based by 2020
of having a
based SSD fabric which can be accessed by many servers and which combines
the latency advantages of local
PCIe SSDs with the
essential hooks from past low latency server interconnects - specifically -
- has been many years in the telling.
- shipments of NVMe SSDs will grow to 25+ million by 2020
There have been 3 main
ingredients to this market brew:-
- something worthwhile sharing as a resource (low latency SSD pools)
- a convenient way of connecting to them (a large installed base of server
PCIe interface chips were
the essential starting point - but it took many years for industry standards to
paper captures current expectations for how the market is expected to grow.
the article (pdf)
- software support
- which ranges from the storage stack to multi-vendor fabric support.
the changing shape of the enterprise PCIe SSD marketby
editor - StorageSearch.com
- July 12, 2016
|If you knew the
enterprise PCIe SSD market really well 4 years ago (in
2012) and if
your attention had been distracted elsewhere in the intervening years -
you'd hardly recognize it today as the same market you once knew. |
not just the names of the key suppliers which have changed but the PCIe SSD
market - which itself was a disruptive influence on the server and storage
markets from the earliest product shipments in 2007 - has not been immune
itself from further disruptive forces from within the SSD market.
think it's not too strong to say that the enterprise PCIe SSD market (as we once
knew it) has exploded and fragmented into many different directions. (And some
parts of the PCIe SSD visionary outlook have shrunk too.)
Here are the
main factors which have contributed to these extra layers of complexity.
- commoditization... Although the enterprise PCIe SSD market started life as
a high value, proprietary technology market it was inevitable that once the PCIe
flash market got big enough (and especially as PCIe SSDs started to be designed
for consumer SSD
roles) many of those technology and cost factors would come back to taunt the
parent enterprise market with cost comparisons which would undermine the
original foundations of many pioneer PCIe SSD business plans. The predictable
incumbent assassin tools included:- simpler (single chip
frameworks (like NVMe) and smaller form factors (like
M.2 - which we see being
used singly and also in arrays on PCIe carrier boards).
smaller, cheaper solutions became available and proven in the market - they
changed the cost structure and roadmap assumptions of the enterprise PCIe SSD
market and also created new application roles which included physically
smaller spaces into which the first generation products couldn't go. The
traditional way for such older product designs to go would up market - but
upmarket was becoming a more competitive space too for similar and other
- new ways of server acceleration... As the enterprise SSD market became
more confident in using SSDs - it became clear that the value of SSD
acceleration - even within a single type of nominated CPU server - could be
far more nuanced and
finely grained and adaptable to different application roles (from a small
SDS cluster upto a cloud
scale data center) by using standard SSDs with different interfaces, latency
and cost characteristics.
Traditional high performance (big form
factor) PCIe SSDs still had useful work they could do in this wider framework
of acceleration - but they were no longer the only solution, or the easiest -
but instead one of many possible solutions - in which electrical power
consumption, capacity, scalability and affordability of the server were
increasingly being seen as more desirable attributes than the peak performance
of any single SSD.
At the high end of the performance range for PCIe
SSDs we're now seeing the market space being squeezed by
SCM DIMM wars
In the current market therefore the traditional big
bang ultrafast PCIe SSD doesn't seem as impressive as it used to be and instead
is part of a continuum of possible performance options in the market rather
than the best game in town.
The notes above show some of
the reasons why the traditional PCIe slot form factor enterprise SSD market
has not followed a single and predictable pattern of market growth.
- New ways of storage box acceleration... When it comes to providing easy to
integrate low latency flash based cache storage in arrays of storage drives -
there have been 3 factors which have between them reduced the market demand for
traditional PCIe SSDs in the kind of roles we saw in hybrids in 2010, 2011 etc.
enterprise has entered or passed the early majority adoption phase of SSD
storage. This means when all the bulk storage drives in a big box are already
cheap SSDs (SATA or SAS) rather than HDDs - the optimum technology for local
caching in such boxes is more likely to look like a small amount of memory (whatever memory may be) rather
than a big fat PCIe SSD front ending the array.
For general purpose
applications some big SSD users have reported that using arrays of SATA SSDs
with their servers gives them similar application performance (and lower cost)
to having PCIe SSD rich servers (SSDserver rank
2) operating in
tandem with earlier generation HDD arrays. The business consequence has been
that follow up deployments of infrastructure across a wide range of cloud
scale uses have tended to use lower quantities of traditional PCIe SSDs than
would have been forecast from the earlier market adoption models.
significant growth is beeing seen in smaller form factors within the PCIe SSD
In new high performance storage arrays the drives in the
array are likely to be 2.5"
PCIe SSDs. This removes the need or benefit of having big form factor PCIe
SSDs in such storage boxes.
In value based hybrid arrays - if there is
still a need for some kind of low latency flash cache (in the traditional PCIe
SSD role) the need is likely to satisfied by
M.2 SSDs rather than a
larger form factor product.
In SDS applications - everything which has
been said above for servers applies too.
complicating factors have been the growing sophistication of the
SSD aware software market
and its ability to encompass many different SSD types (by interface and form
factor) and use the varied SSD latency and R/W characteristics within pools of
abstracted virtual storage. Again this kind of software feature was initially
pioneered in PCIe SSDs and associated with such products. The decoupling of
server application agility from SSD form factor and specific product lines has
increased competitive choice for SSD integrators to the extent that they can now
get PCIe-SSD -like capabilities without having to have PCIe SSDs physically
PCIe SSDs will still be very useful components in the
enterprise market. And the advantages of latency and ease of integration have
migrated into other form factors as already noted above (2.5", M.2 and also
BTW SSDs on a chip too).
The SSD, storage and server market has learned
many valuable lessons from the early phases of the PCIe SSD market (2007 to
2012) about data architecture, how to use SSDs as storage and memory and how to
improve infrastructure utilization. And the system shaping techniques so
learned now form part of the conventional wisdom of the industry. But continuing
adaptation of those new tricks is not tied or bound to the original past
physical form factors or the companies which first experimented with them.
Instead the PCIe SSD market and its associated ideas have exploded into many
different shapes and sizes and segments.
It's still an interesting
market to keep your eyes on - but is no longer at the vortex of architectural
disruption. Instead you could say it provides a sanity check and places cost
zone limits on the future direction of the DIMM wars market (with which it
overlaps in capabilities at the lower latency end of the PCIe SSD product
spectrum) and the PCIe SSD market is itself prevented from getting too greedy by
competing with the wide range of alternative technologies which (from the user
point of view) provide similar and competing system capabitites.
|how fast can your
SSD run backwards?|
|SSDs are complex devices and there's a
lot of mysterious behavior which isn't fully revealed by benchmarks, datasheets
and whitepapers. Underlying all the important aspects of SSD behavior are
which arise from the intrinsic technologies and architecture inside the SSD.
|You don't have to
understand the internal details of how these individual techniques work. And
with hundreds of patents already pending in this topic there's a high
probability that the SSD vendor won't give you the details anyway (not even
under NDA). It's enough to get the general idea.|
care management & DSP ECC IP in SSDs|
|the Problem with
Write IOPS in flash SSDs |
|Random "write IOPS"
in many of the fastest flash SSDs are now similar to "read IOPS"
- implying a performance symmetry which was once believed to be impossible.|
why are flash SSD IOPS such a poor predictor of application performance?
why are users still buying
RAM SSDs which cost an
order of magnitude more than SLC? (let alone
MLC) - even
when the IOPS specs look superficially similar?
tells you why the specs got faster - but the applications didn't.
||And why competing SSDs with
apparently identical benchmark results can perform completely differently.
|many years ago -
|BiTMICRO & CENATEK collaborate on PCI SSDs|
2002 - BiTMICRO.
announced today, a technology and marketing partnership to investigate
developing a hybrid solid state disk storage solution that brings together the
best in flashdisk storage and PCI bus-attached SSD technology.
Prices are likely to
range between $1,000 to $2,000 per gigabyte.
Editor's later comments:- the fast PCI SSDs which later
emerged from CENATEK - can be seen as immediate ancestors of the modern PCIe
But the concept of using fast bus based SSDs as storage
accelerators wasn't a new idea. It went back 3 to 4 decades.
earlier market experiments with solid state storage were always short lived -
because the expensive SSD storage in each product generation was always
competing with fast changing improvements in CPU clock speeds, bus memory
throughput or faster external magnetic storage media.
The changes in
the modern era of
SSDs - which started about 2003 - was that due to those other computer
technologies stagnating and not getting any faster - the only competitor which
killed an SSD from about that time - was another SSD.
By that time -
the new computer bus was known to be PCI express. But it was the failure of the
alternatives to solid state storage to get faster - which made the big
difference to the business viability of SSDs - rather than any inate
characteristic of PCIe SSDs.
the fastest SSDs
2.5" NVMe PCIe SSDs
after AFAs -
- added new notes to the music of memory tiering
consequences of the 2017 memory shortages
where are we
heading with memory intensive systems and software?
|new industrial single chip
PCIe NVMe SSDs from Silicon Motion|
Editor:- February 27, 2018 -
production of 2 new industrial grade single chip PCIe NVMe SSDs.
- SM689 supports PCIe Gen 3x4 interface in 16mm x 20mm
products can support multiple capacity configurations ranging from 16GB to 256GB
and include enterprise-grade advanced data integrity and reliability
capabilities using Silicon Motion's proprietary end-to-end data protection, ECC
and data caching technologies.
- SM681 supports PCI Gen 3x2 interface in 11.5mm x 13mm
Data integrity features include
end-to-end data path protection, which applies ECC to the SSD's SRAM and DRAM
buffers as well as to the primary NAND Flash memory array.
who makes single chip SSDs?
farewell to reassuringly boring industrial SSDs
PCIe SSDs (enterprise and notebook M.2) did well in Q3 2017
November 15, 2017 - TrendFocus
market shipment data for Q3 2017. Only one segment, enterprise PCIe, saw
unit growth where every other segment client drive format factor, client
modules, enterprise SATA and enterprise SAS, all declined from the prior
enterprise SSDs market declined 7% Q-Q, which includes
PCIe. The bright spot
within this overall decline was the healthy 15.6% increase in PCIe units. As
hyperscale companies continue to migrate away from SATA, PCIe should continue to
grow in both units and exabytes. SATA, still the highest volume of all
enterprise categories, managed to stay just above 4 million units shipped but
did decline sharply in CQ3. However, exabytes shipped in the SATA SSD market
grew due to the transition to higher capacity units. SAS SSDs now represent the
lowest unit volume of the enterprise SSD segments, but still maintain a large
lead in average capacity shipped at over 2.1 terabytes.
shipments fell 4.5% sequentially but exabytes shipped was flat. Client modules
now represent almost 2/3 of all client SSDs shipped. Even more impressive within
this segment is that M.2
PCIe is now 50% of this segment illustrating the continued migration for
OEMs to integrate with this interface.
3D NAND accounted for more than
50% of all bits shipped for the first time in CQ3, as all of the NAND suppliers
are well into the transition.
Toshiba says it will sell memory business to Bain led consortium
September 20, 2017 - Toshiba
its long awaited decision about who it has chosen to sell its memory business
to. It's a consortium led by Bain Capital using an acquisition cutout filter
created for this purpose called K. K. Pangea. The transaction (worth about $18
billion) is "expected to close by the end of March 2018."
reports which had speculated about the identity of members of the
consortium named at various times Dell, Apple and Seagate.
said in the above announcement "Western Digital has sought to prevent the
sale of the interests of the joint parties (meaning Toshiba and WDC) to any 3rd
party and Toshiba and WDC are currently engaged in litigation and arbitration."
Longsys showcases tiny NVMe BGA SSDs at FMS
August 7, 2017 - Longsys
that it will be sampling the industry's first 11.5x13mm NVMe BGA SSDs which
support Boot Partitions and the Host Memory Buffer features of NVMe rev 1.3
in Q4 2017.
P900 series -
(DRAM-less) PCIe 3.0x2 SSDs - aimed at the
and being shown in the next few days at the
Flash Memory Summit - will be
available in capacities from 60GB to 480GB (64-layer 3D NAND ) and use the
88NV1160 controller from Marvell.
Editor's comments:- On a historical note the world's first PCIe SSDs
in a BGA form factor were shown in
That was made by Toshiba.
But the big difference today is NVMe - which has become the new urgent low
latency lingua franca of SSDs (although SCSI will still be around for decades
too). I predict that NVMe is what will make it feasible to introduce
enterprise array architecture concepts into consumer and IoT markets using BGA
JMR enters the HPC NVMe PCIe SSD market
June 16, 2017 - Persistence can be a virtue for
PR folk and not
Although it's been 15 years since I last wrote about JMR I was still reading
This morning I noticed they're offering NVMe PCIe
SSDs (HHHL and bigger) aimed at the HPC and other high throughput integrator
markets. MSRP of the JMR
SiloStor starts at $795 for 512GB. Capacities available upto 8TB.
you need any boxes to put the new SSDs in - they might have some suggestions
too as they've been in the special enclosure business since 1982.
what's 5 microseconds latency worth in PCIe SSD fabric?
May 24, 2017 - 10 microseconds is the latency advantage of Excelero's
proprietary NVMesh compared to simple NVMeoF when managing fabrics of
dispersed NVMe SSDs in a PCIe connected network. More details like this appear
in a new blog on InfoWorld -
storage architecture for the enterprise - by Yaniv Romem CTO and
Tom Leyden VP of
corporate marketing at Excelero. ...read
Editor's comments:- when Excelero emerged from
stealth in March
2017 the low latency overhead of their software was a big deal - at just 5µS
compared to accessing a similar SSD in the same rack.
This is an
industry magic ballpark number which has been quoted as a worst case response
in earlier years by several pioneers in big memory architectures - including
details may have changed or been refined since.)
SCM DIMM wars
watchers know that if you can get inside that curve for most of your worst case
latencies then (with enough memory and cache) you can run popular memory
hogging applications with better performance than using traditional DRAM in
traditionally networked servers. PCIe fabrics compete and collaborate in the
memory systems market. Latency lessons learned from one of these contexts
can be used to guide initial expectations (subject to verification) in the
the SSD Bookmarks
now Cinderella industrial systems with "no-CPU"
budgets and light wattage footprints can go to the NVMe speed-dating ball
April 19, 2017 - A dilemma for designers of embedded systems which require high
SSD performance is how can you get the benefits of enterprise class NVMe SSDs
for simple applications - which integrate video for example - without at the
same time escalating the wattage footprint of the entire attached micro server?
new paper published today by IP-Maker -
server-class storage in embedded applications (pdf) discusses the problem
and how their new FPGA based IP enables any NVMe PCIe SSD to be used in
embedded systems to provide sub-microsecond latency using "20x better power
efficiency, and 20x lower cost compared to a CPU-based system."
company says the NVMe host IP - which is now available - can be used in an FPGA
connected between the PCIe root port and the cache memory, internal SRAM or
external DRAM. It fully controls the NVMe protocol by setting and managing the
NVMe commands. No CPU is required. It supports PCIe gen 3 x 8 interface.
Michael Guyard, Marketing
Director said that - among other things - applications include:-
- military recorders
- portable medical imaging
- mobile vision products - in robots and drones
Editor's comments:- Now Cinderella
embedded systems with low cost budgets and low wattage footprints can go to the
enterprise NVMe performance ball. The new magic - in the form of the FPGA IP
released today by IP Maker - has the potential to transform the demographics
and class of SSDs seen in future industrial systems.
SSD glue chips
CPUs for use with SSDs
not your grandfather's industrial SSD
Intel is sampling 3DXpoint PCIe SSDs
19, 2017 - Intel
that it is sampling its long awaited first enterprise SSD which uses 3DXpoint
(Optane) memory and which is aimed at the HHHL
PCIe SSD market. The
Series (pdf) has a PCIe 3.0 x 4 NVMe interface and provides upto 375GB
capacity, 500K mixed
block level R/W latency 150/200µS (queue depth 16), and endurance of 30
DWPD for 3 years
(equivalent to 18 on a 5 year adjusted basis).
The new drives are
supported by caching / tiering software (Intel Memory Drive Technology) which
collaborates with motherboard DRAM resources to transparently provide an
emulated 3DX as RAM memory pool. This is similar in concept to earlier products
in the market from various vendors which supported flash as RAM.
widely expected the new SSDs have worse performance and higher pricing than
Intel had indicated at the first public unveiling in the summer of 2015.
A rounded perspective can be seen in a new blog
Announces Optane SSDs for the Enterprise - by Jim Handy - founder Objective Analysis.
other things Jim says "Intel has announced an SSD whose performance is
close to that of NAND flash at a price that is close to that of DRAM. How did
that happen?" ...read
Editor's comments:- As the new P4800X is
not hot pluggable and as its main difference to previous flash SSDs from Intel
is its support as a tiered memory - the most obvious role for a competitive
comparison is memory channel based NVDIMM solutions - in particular the Memory1
product from Diablo
which provides 128GB of flash as RAM per DIMM socket - and upto 2TB in a 2
Density comparison - Optane PCIe and Flash DDR-4
a density comparison the current technology appears to be 1x HHHL Optane PCIe
SSD gives the equivalent emulated memory as 3x DDR-4 Memory1 DIMMs. Although
this doesn't take into account how much DRAM is needed to support each type of
configuration it's a good enough comparison point to start with. But there are
some areas of doubt about the roadmaps for each of these memory solutions. See
more analysis in SSD news.
Everspin enters NVMe PCIe SSD market
8, 2017 - Everspin
it is sampling its first SSD product an HHHL NVMe
PCIe SSD with upto 4GB
ST-MRAM based on the company's own 256Mb DDR-3 memory.
The new nvNITRO
ES2GB has end to end latency of 6µS and supports 2 access modes:- NVMe
SSD and memory mapped IO (MMIO).
Everspin says that products for the
U.2 markets will
become available later this year. And so too will be higher capacity models
using the company's next generation Gb DDR-4 ST-MRAM.
comments:- Yes - you read the capacity right. That's 4GB not
4TB and certainly not 24TB.
would you want a PCIe SSD which offers similar capacity to a backed
RAM SSD from
DDRdrive in 2009?
And the new ST-MRAM SSD card also offers worse latency, performance and
capacity than an typical
using flash backed DRAM today.
What's the application gap?
answer I came up with is fast boot time.
If you want a small amount of
low latency, randomly accessible persistent memory then ST-MRAM has the
advantage (over flash backed DRAM such as you can get from
Netlist etc) that the
data which was saved on power down doesn't have to be restored from flash
into the DRAM - because it's always there.
The boot time advantage of
ST-MRAM grows with capacity. And depending on the memory architecture can be on
the order of tens of seconds.
So - if you have a system whose
reliability and accessibility and performance depends on healing and recovery
processes which take into account the boot times of its persistent memory
subsystems - then you either have the choice of battery backup (which occupies a
large space and maintenance footprint) or a native NVRAM.
cards will make it easier for software developers to test persistent RAM
tradeoffs in new equipment designs. And also will provide an easy way to
evaluate the data integrity of the new memories.
Kingston ships HHHL NVMe PCIe SSD using Liqid controller
March 7, 2017 - Kingston
shipments of a another new NVMe
PCIe SSD based on its
partnership with Liqid.
The DCP1000 has a Gen. 3.0 x8 interface and delivers upto 6.8GB/s and 6GB/s
sequential R/W throughput respectively. The HHHL form factor SSD has upto
raw 3.2TB capacity and is rated at under 0.5
DWPD for 5 years.
NxGn Data is now called NGD Systems
22, 2017 - NGD Systems
(formerly called NxGn Data) today
availability of a new product aimed at the
PCIe NVMe SSD market.
The Catalina SSD has 24TB of 3D TLC flash which the company says uses less
than 0.65 watts per terabyte.
Corsair enters M.2 NVMe PCIe SSD market
December 13, 2016 - Corsair
its entry into the M.2 SSD
market with a new PCIe Gen. 3 x4 product aimed at the
market. The Force MP500 is Corsair's fastest SSD and uses a
WDC samples its fastest ever NVMe PCIe SSDs
December 6, 2016 - Western
it is sampling faster new models in its range of enterprise SSDs.
WDC says that these
are the fastest NVMe PCIe SSDs which they have offered to date.
Foremay ships aerospace capable 8TB 2.5" U.2 NVMe SSD
September 26, 2016 -
volume production of 8TB models in its rugged secure 2.5" U.2 NVMe SSD
product range - which with PCIe x4 lanes has R/W speeds up to 1.2GB/s with
latency as little as 25 microseconds. Optional features of the SC199 hi rel
- Military secure erase and fast erase features.
- Rugged designs with anti-shock and anti-vibration, meeting MIL-STD-810G/F
also:- military SSDs
- Anti-radiation and anti-emission, both electrical and magnetic, for
aerospace applications subject to the customer's specifications.
Radian will sample PCIe based competitor to NVDIMMs in October
August 2, 2016 -
it will be sampling in October a new byte addessable,
(NVRAM-flash cache ratio), 12TB
PCIe SSD which has
on-board host controlled tiering between its flash and NVRAM.
comments:- Radian is positioning the new product as a cost effective alternative
competitor to hybrid
NVDIMMs and similar emerging products in the
SCM DIMM wars
In an interview last year with Radian's CEO Mike Jadon -
what's the role for a
Radian Memory SSD? - I learned more about the company's software thinking.
IP-Maker releases Gen 3 NVMe PCIe reference design
July 11, 2016 - for designers of
PCIe SSDs - IP-Maker has released
its new Gen 3 NVMe PCIe reference design which is based on the VC709
evaluation kit by Xilinx. It's integrated
PCIe Gen3 hard IP and a soft DDR3 controller. The
NVMe compliant design uses
a x4 lanes configuration.
SSD glue chips
Mangstor has P2P in NVMe-based PCIe SSDs
June 22, 2016 - Mangstor
announced that its MX-Series family of NVMe-based PCIe SSDs now features
higher capacity 5.4TB usable models (FHHL) as well as Peer-to-Peer (P2P)
capabilities which provide low latency access to an extremely large storage pool
of non-volatile memory, bypassing host resources. The new models have been shown
this week at the ISC High Performance
Conference in Germany.
""Our innovative ability to
separate data control from the data path optimizes I/O, reduces latency spikes
and system bottlenecks, and delivers accelerated performance" said Paul Prince, CTO
Micron samples 2TB 3D 2.5" PCIe NVMe SSDs for desktops
May 31, 2016 - Micron
it is sampling new
2.5" PCIe NVMe
SSDs based on 3D nand for the
2100 with 2TB capacity is expected to be in [roduction in the summer.
2 ASIC roles for PCIe based BiTMICRO SSD controllers
March 25 , 2016 -
5 years ago when
unveiled an earlier generation of its high performance enterprise SSD
controller architecture - it was clear that their preference was for a chipset
which included 2 different types of functionality.
This kind of
thinking wasn't unique at that time - as I'd seen similar things in
designs before but (unlike BiTMICRO) those other designs were captive and not
offered as COTS SSD
How many controller chips do you really need for a
a new blog
today BiTMICRO explains why its current generation of controllers continues
using a 2 ASIC architecture with one acting as a flash array extender and the
other as the main PCIe host interface controller.
Among other things
the blog says "To increase flash channel bandwidth and capacity, more flash
channel expander chips can be instantiated and connected to the main controller."
As noted in the SSD
design heresies - SSD vendors often have different implementation
architecture approaches which compete in similar application slots. When
evaluating different types of offerings it can be useful to ask yourself - which
direction is my own design likely to stretch in future? (Towards more
performance? lower cost? bigger scale? adjacent application role? etc.)
BiTMICRO's blog clarifies where they see their strengths in the market. ...read the
PCIe 4.0 chips milestone
Editor:- March 14, 2016 -
Signs of onwards and upwards progress towards future
PCIe SSD speeds
emerged today in an
have demonstrated electrical interoperability for PCIe 4.0 with "robust
signal integrity" (BER below 10-15) at 16Gbps with 4 lanes running
Seagate promises faster PCIe SSDs
8, 2016 - Seagate
it will ship a 16 lane NVMe PCIe SSD with 10GB/S throughput in the summer. No
further details are available at this time.
in 2015 the fastest production PCIe SSDs were probably the
NVRAM from Microsemi
and the MX6300
2016 the fastest
motherboard SSDs will be
SSDs rather than PCIe SSDs - which potentially will be able to deliver
twice the performance of 16 lane PCIe SSDs - but which are more
capacity and fault
It's not worth paying more for SLC reliability in PCIe SSDs says
Google field study
editor:- February 26, 2016, 2016 - A 6 year
PCIe SSDs used by
Google (spanning millions of drive days and chips from 4 different flash
vendors) concluded that SLC drives were not more reliable than MLC.
An important conclusion re RAS is the importance of being able to map
out bad chips within the SSD architecture. This is because somewhere between
2% to 7% of enterprise PCIe SSDs (depending on where they were used) developed
at least bad chip during the first 4 years in the field - which without such
remapping would necessitate replacing the failed SSD.
The source is -
Reliability in Production: the Expected and the Unexpected (pdf) - by Bianca
Schroeder University of Toronto, Raghav Lagisetty and
Arif Merchant, Google.
This is just one of a set of
which was presented February 22 - 25 , 2016 at the
14th USENIX Conference on
File and Storage Technologies.
Editor's comments:- For more
like this scroll down this page to see the June 2015 story about a large
scale study of PCIe SSD failures within Facebook.
new TB PCIe SSD on M.2 from OCZ
Editor:- January 5,
2016 - OCZ says it
will show new NVMe PCIe SSDs with 1TB capacity and up to 2.4GB/s of bandwidth on
M.2 module at
CES this week.
Memblaze gets more funding
Editor:- December 21,
2015 - Memblaze
it had received "tens of millions of US dollars" in Series C
funding. Among other things the company said it will now expand its sales
efforts in international markets.
Mangstor gets $10 million series B funding
for fastest NVMe
Editor:- September 17, 2015 - Mangstor today
it has closed $10 million in Series B funding which will be used to fuel
growth in sales and engineering and business development.
investment confirms our leadership in NVMe over fabrics technology which
delivers an order of magnitude higher performance at the lowest latency verses
legacy iSCSI and Fiber Channel" said Trevor Smith
CEO and co-founder of Mangstor .
comments:- last week I had a 90 minutes one on one with Trevor which we
spent talking about gaps in the enterprise SSD market, Mangstor's technology
and the competitive positioning of its PCIe SSDs and systems. You can see some
of our discussion topics in their profile page.
M.2 NVMe PCIe SSD good
4 is better says Memoright
September 3, 2015 - Memoright
discussed the possibilities arising from using an array of 4x
NVMe PCIe SSDs packed onto a single PCIe switch card as a way of building
competitively priced SSD accelerated servers - in a paper
a PCIe-Based Switch Module to Enhance Enterprise Storage Architecture (pdf)
presented at the recent Flash
Memory Summit. Each of the new M.2 SSDs from Memoright would have upto 1TB
capacity and R/W IOPS upto 81K / 54K respectively.
comments:- And another recent example of something similar - may be
Kingston's E1000 -
hardware - where it was described as a PCIe SSD which integrates an "array
of 4 NVMe SSDs". (Possibly 4x Kingston's
would be my guess.)
This is part of a new trend in enterprise SSDs.
years ago companies which at that time didn't have their own
architecture (OCZ etc)
were able to compete in raw throughput performance with those who did (Fusion-io,
Virident etc) by
designing PCIe SSDs which were populated by an array of SATA SSDs - routed
together by an onboard RAID controller.
The modern way of doing
something similar is to have an array of small form factor NVMe SSDs on a
The (wishful thinking) marketing argument for this is
that the small NVMe SSDs have their costs and design roadmaps attached to a
much faster changing and more competitive groove in the SSD market than the big
proprietray designs - which reduces the risks of customers being tied to an
expensive single source design.
Although these different design
approaches can deliver SSDs which overlap in standalone
capability - there are usually
factors involved -
customer risk perceptions and preferences - which determine the SSD chosen.
Seagate enters 2.5" NVMe SSD market
August 11, 2015 - Seagate
details of 2 new families of
NVMe SSDs which
will be available in 2.5"
(October) and M.2 (in early
2016) form factors. Also new - the Seagate Nytro XP6500 - a
PCIe SSD accelerator
(which is currently available) delivers the lowest write latency within
Editor's comments:- until this announcement
it wasn't clear how Seagate would deal with an issue which has been
problematic for other competitors too - that of introducing SSDs in form factors
and interface types which can (at the fringes) compete with pre-existing product
But because there is clear customer demand for both
SAS SSDs and
PCIe SSDs in 2.5"
form factors for example - then it would be a mistake for any vendor with large
scale market ambitions to opt out of supplying such products despite the
potential risk of some cannibalization.
Nevertheless - the order in
which we see enterprise SSDs being introduced in various form factor and
interface combinations (different for each supplier) tells us what they consider
are their native strongholds.
Virtium launches industrial M.2 PCIe SSDs
July 28, 2015 - Virtium
it has expanded its StorFly range of industrial SSDs with new
PCIe SSDs (gen 2)
available in both M.2 and
Mini Card form factors with capacities from 16GB upto 480GB.
is upto 3.3 petabytes of writes (about 3.7
DWPD for 5 years).
Virtium's new SSDs have full BOM control with up to 5 years of uninterrupted
bath tub curve is not the most useful way of thinking about
PCIe SSD failures - according to a large scale study within Facebook
June 15, 2015 - A recently published research study -
Study of Flash Memory Failures in the Field (pdf) - which analyzed
failure rates of PCIe
SSDs used in Facebook's infrastructure over a 4 year period - yields some
very useful insights into the user experience of large populations of
Among the many findings:-
- Read disturbance errors - seem to very well managed in the enterprise SSDs
The authors said they "did not observe a statistically
significant difference in the failure rate between SSDs that have read the
most amount of data versus those that have read the least amount of data."
- Higher operational temperatures mostly led to increased failure rates,
but the effect was more pronounced for SSDs which didn't use aggressive data
throttling techniques - which could prevent runaway temperatures due to
throttling back their write performance.
- More data written by the hosts to the SSDs over time - mostly resulted in
more failures - but the authors noted that in some of the platforms studied -
more data written resulted in lower failure rates.
attributed to the fact some SSD software implementations work better at
reducing write amplification when they are exposed to more workload patterns.
- Unlike the classic bathtub curve failure model which applies to hard drives
- SSDs can be characterized as having early an warning phase - which comes
before an early failure weed out phase of the worst drives in the population
and which precedes the onset of predicted endurance based wearout.
this aspect - a small percentage of rogue SSDs account for a disproportionately
high percentage of the total data errors in the population.
report contains plenty of raw data and graphs which can be a valuable resource
for SSD designers and software writers to help them understand how they can
tailor their efforts towards achieving more reliable operation. ...read
the article (pdf)
Memblaze is #1 enterprise PCIe SSD supplier in China market
May 22, 2015 - Memblaze
accounted for 60% market share of the
enterprise PCIe SSD
market segment in China in 2014 - according to CEO Yin Xuebing
who made this comment in a recent
release announcing the availability of Memblaze's new PBlaze4 700 Series
NVMe PCIe SSDs which was demonstrated at CeBIT
in March 2015.
OCZ's power envelope programmable 2.5" hot swap NVMe
May 20, 2015 - OCZ
today revealed more details about the new models shipping in its NVMe
compatible PCIe SSD family - which was first announced last
had already heard before these new models include 2.5" hot swappable
Today OCZ said this model - the
Z-Drive 6300 SFF
will be available with usable capacities of 800GB, 1.6TB and 3.2TB (in
this quarter) followed by 6.4TB (later this year).
performance is upto 2.9GB/s and 1.4GB/s respectively. Random R/W
700K IOPS and 120K IOPS. Latencies are 30µs (write) and 80µs (read).
options are 1 or 3 DWPD.
availability and reliability features
The new Z-Drive 6000 models
are dual ported so that 2 host systems can concurrently access the same SSD.
Additionally, the Z-Drive 6000 Series supports hot swapping of 2.5"
drives, pre-set power thresholds and temperature throttling to support many
types of enterprise ecosystems.
Editor's comments:- for various reasons to do with a
combination of standardization
efforts and changes
of ownership for nearly every major enterprise PCIe SSD company in the
market - you've had to wait 3 years since the idea of this kind of product was
first discussed seriously on
these pages and at
What has become clear to systems architects is that these
new products offer far more flexibility in their roles than merely performance
upgrades to high end SAS
SSDs and traditional storage arrays.
Among other things these new
types of products will enable lower cost mini SSD server clustering at
PCIe latencies which will spur growth in the SDS market. At the high end - they
could become the new building blocks inside the world's most powerful computer
Power consumption and heat in these NVMe SSDs?
know from talking to systems architects that the electrical power and thermal
footprints of 2.5" NVMe SSDs is a critical detail when considering the
design of dense storage arrays so I asked Scott Harlin,
OCZ for more information these factors. Here's what Scott said.
Zsolt you are correct the 2.5" drives can get a little toasty
packing in the higher densities into this form factor -- typical power
consumption of the Z-Drive 6000 series is 25W active and 9W idle. So we
included a few items to address these concerns:
1. - Temperature sensing and thermal throttling to maintain
consistent operating conditions even under adverse temperature variances
2. - User-selectable power envelopes, in 15W, 20W and 25W
settings, reduces wasted power when maximum performance is not required while
efficiently addressing temperature requirements in support of a variety of
3. - An innovative 'flow-through' case design enables more
airflow to critical components, keeping the device cool while reducing airflow
Editor's comments:- that user selectable power envelope- in graduated
steps - seems like a really useful design attribute. So I'll be watching out
for it in future arrayable SSD launches.
CoreRise ships new smaller BladeDrive PCIe SSD
April 20, 2015 - CoreRise
customer shipments of a new version of its BladeDrive family of gen 2 x8
PCIe SSDs - the E24
- which has a smaller form factor than the earlier E28.
based implementation supports upto 1.6TB capacity, 275K IOPS (4KB) and 2GB/s
throughput in half-height half-length. Software support includes Windows
Server, Linux and virtualization such as Xen, Hyper-v, as well as TRIM.
SanDisk comments on customer drift away from its PCIe SSDs
April 16, 2015 - You should be careful not to misconstrue SanDisk's recent
statement about enterprise PCIe SSD replacement by SATA SSDs as a general trend
for all customers and all types of PCIe SSDs. I think some of these customer
migrations are probably more specific to some large scale customers for the
Fusion-io product line rather than the whole industry.
results related conference call (transcript)
this week SanDisk's
President and CEO - Sanjay Mehrotra
- said that in the enterprise market it was seeing some of the business
which had traditionally been implemented by its customers using
PCIe SSDs was moving
towards arrays of SATA
Editor's comments:- another emerging market is
arrays of 2.5"
PCIe SSDs which you could argue will take away market share from
SAS SSDs - as well as
enabling a new market.
Factors at work in the
rackmount flash array
complex due to very different customer needs and different ways of judging
the same tech attributes due to users' own business values related to their core
activities and what they've done before.
8TB 2.5" PCIe SSDs sampling soon from Novachips
March 4, 2015 - Are you interested in a world's first
8TB (15mm height) PCIe Gen2 x4 SSD (pdf) with a single controller developed
was an email I got recently from Sean Oh, in
Germany, who is the sales representative for these products in Europe.
would you say?
I did the same. After reading up the info he attached
too) I asked some questions about availability. Here's what Sean said.
- The working engineering samples have been available since last year.
- The 1st customer sample comes out in the next 30 days.
- We plan to start a mass production in 2nd half of this year.
comments:- part of this story has its roots in a news story from
May 2007 - when a
company called MOSAID was talking about a new, light capacitive load, ring
based, flash memory topology called HLNAND. To make it work they needed a
controller. Novachips collaborated on the design and recently acquired the
assets and patents.
- What do you think ? Is it something cool, isn't it ?
Greenliant enters PCIe SSD market
12, 2015 - Greenliant
Systems - has entered the entry level enterprise
NVME PCIe SSD market -
of its new G7100 (pdf)
series MLC gen 2 x4 PCIe SSDs - upto 2.75 TB raw capacity, Full Height, Half
Length form factor, 130K / 60K
of upto 10 DWPD for 5
Internally Greenliant's new PCIe SSD has a
RAID protected array of
miniature NANDrive SSDs which use the company's own controllers.
our in-depth flash memory knowledge and volume-proven NAND controller expertise,
Greenliant is addressing the industry's increasing need for higher reliability,
higher performance and larger capacity flash-based storage solutions," said
CEO of Greenliant Systems.
Editor's comments:- ever since the
company was founded - in 2010 - Greenliant's focus has been on the
embedded SSD market. So it's surprising to see this new product aimed at
the "enterprise" market. However it's part of an
trend in the market.
Samsung mass producing gen 3 PCIe SSDs for notebooks
January 7, 2015 - 19 months after launching its first M.2 PCIe gen 2 SSDs
aimed at the
(pdf) in June
2013) - Samsung
said it is now mass producing the follow-up SM951 - which supports gen 3
Samsung says - "For ultra-slim notebooks and workstations
the SM951 can read and write sequentially at 2,150MB/s and 1,550 MB/s
The SM951 is the first SSD to adopt L1.2 low
power standby mode (the PCIe
SSD equivalent of the power saving
mode in SATA SSDs) .
When hibernating in
mode, the SM951's power consumption is less than 2mW.
Toshiba shows early version of BGA PCIe SSD
January 7, 2015 - Toshiba
it will showcase a prototype of the world's first PCIe single package SSD -
with up to 256GB in a single BGA package at
CES this week.
compatible device fits into 16mm x 20mm x 1.65mm and weighs under 1g. See
also:- BGA SSDs
Samsung mass produces 3TB 3D PCIe SSDs
September 25, 2014 - Samsung
today announced it has started mass producing 3.2 TB NVMe
PCIe SSDs (HHHL)
based on its 3D flash memory
technology, for use in enterprise systems.
new NVMe PCIe SSD, SM1715 provides a sequential R/W speeds upto 3GB/s and
2.2GB/s respectively with
rated at 10 DWPD for 5
Editor's comments:- Samsung supplied an
for the new SSD - but forgot to include the block size to which it relates - so
I've asked for clarification.
Seagate launches new improved Nytro PCIe SSDs
September 10, 2014 - Seagate
2 new PCIe SSDs -
which are based on the SSD product lines and brand assets of the recently
acquired SSD business of LSI.
XP6302 is a HHHL, gen 3 PCIe SSD - which provides up to 1.75 TB of
usable eMLC capacity with 200 microseconds average latency, and 295K/79K
(8KB) and rated for 0.9 DWPD
endurance for 5 years.
XP6210 is a FHHL gen 2 PCIe SSD with 1.86TB usable 19nm cMLC
capacity, with 50 microseconds average latency 185K/120K R/W IOPS (8KB),
and rated at 1.6 DWPD
endurance for 5 years.
HGST announces 2nd generation clustering software for FlashMAX
Editor:- September 9, 2014 - HGST today
a new improved version of the
clustering capability previously available in the
PCIe SSD product line
acquired last year from Virident.
allows clustering of up to 128 servers and 16 PCIe storage devices to deliver
one or more shared volumes of high performance flash storage with a total usable
capacity of more than 38TB.
HGST says its Virident HA provides a "high-throughput,
low-latency synchronous replication across servers for data residing on FlashMAX
PCIe devices. If the primary server fails, the secondary server can
automatically start a standby copy of your application using the secondary
replica of the data."
For more details see -
Virident Software 2.0 (pdf)
Editor's comments:- This
capability had already been demonstrated last year - and
ESG reported on the
technology in January
But at that time - the clustering product called vShare -
was restricted to a small number of servers - and the data access fabric was
restricted to Infiniband
With the rev 2.0 software - the number of connected devices has
increased - and users also have the lower cost option of using
Ethernet as an alternative
Plextor's M.2 PCIe SSD wins award at FMS
August 7, 2014 - Plextor
- an M.2 PCIe SSD - has
won Best of Show for most Innovative flash memory technology at the Flash Memory Summit.
M6e SSD combines a multi-core
Marvell PCIe 9183
controller and Toshiba
toggle NAND flash with firmware developed by Plextor's in-house team.
say hello to Shannon Systems
Editor:- August 6, 2014
- I hadn't heard of Shannon
Systems before. But I got a nice email this morning from Xueshi Yang,
CEO and co-founder who said he has been reading StorageSearch.com "for
quite a number of years now" and also said that his company is showing
their products at Flash Memory
Among other things - Xueshi Yang said - "Shannon
System is a startup I co-founded in 2011 in China after I left
company is dedicated to the enterprise flash storage market. Currently, we focus
on the high performance PCIe market with our proprietary controllers and
software systems. In April this year, we
the industry first 6.4TB PCIe SSD with a single controller, which boasts 67us
read access latency and 9 us write access latency (all in 4KB, random). While in
June, we introduced a PCIe SSD with SFF-8639 interface, which is hot-pluggable.
We currently serve over 100 customers in China, including Tier 1 internet
companies, as well as other named customers such as China Mobile, China Telecomm
eASIC supports Mobiveil's NVMe platform
August 6, 2014 - eASIC
announced support for Mobiveil's
NVMe platform (pdf)
implemented in eASIC devices.
"eASIC is enabling the rapid
deployment of SSD technology at substantially lower cost and up to 70% lower
power than alternative solutions", said Jasbinder Bhoot,
VP of Worldwide Marketing at eASIC. "By working with Mobiveil, customers
will have access to a complete NVMe solution running in cost, power and
performance optimized eASIC devices."
HGST rekindles concept of a PCM based PCIe SSD
August 4, 2014 - HGST
it will demonstrate a PCM PCIe SSD concept at the Flash Memory Summit. HGST says
the demonstration model delivers 3 million random read IOPS (512 Bytes) and
a random read access latency of 1.5 microseconds.
funded the world's first enterprise PCM PCIe SSD demo 3 years ago (in
June 2011). The
storage density of PCM resulted in an SSD which had pitifully low capacity
compared to flash memory at that time - and earlier this year (in January 2014)
that Micron had temporarily abandoned this idea.
Is HGST really
going to wander into memory space where even the memory makers don't want to go?
Or is this just a market signal that HGST isn't just looking at short term SSD
Lite-On enters enterprise M.2 PCIe SSD market
July 30, 2014 - Lite-On
will be showing a new M.2 PCIe
SSD for the enterprise market next week at the Flash Memory Summit.
The P1P is a
PCIe SSD with capacities of 1TB and
protection circuit all in a small form factor.
The small form
factor and high capacity allow enterprise customers to pack more storage in a
smaller footprint. The P1P Series can deliver sequential R/W speeds up to
610MB/s and 520MB/s while random R/W speeds can be upto 95K/15K
drive has a MTBF
of 2 million hours and an
rating of up to 1 drive write
per day for 5 years.
|archived versions of this PCIe SSDs news page
|PCIe SSD milestones from
|August 2007 -
Violin Memory launched the world's fastest 2U SSD - with a PCIe interface.|
September 2007 -
Fusion-io launched the ioDrive - a PCIe form factor flash SSD with upto 640GB
capacity and 100K IOPS performance.
April 2008 -
StorageSearch.com published a new market directory of PCIe SSDs.
January 2009 -
StorageSearch.com ran the world's first online ads for enterprise SSDs in the
PCIe module form factor. That advertiser (the industry changing
its ads for PCIe SSDs here on these pages right upto the time it was acquired
by SanDisk in July
2014). See what this
2009 PCIe SSD page looked like back then as captured by Archive.rog.
April 2009 -
StorageSearch.com announced that Fusion-io was the
Top SSD Company
researched by readers in
September 2009 -
StorageSearch.com disclosed that search volumes for PCIe form factor SSDs by
its readers had surpassed that for 2.5" SSDs for the 1st time ever.
2011 - Virident Systems demonstrated 1 million IOPS performance in a 1U
server rack using just 2 of its tachIOn PCIe SSDs at a system list price of less
than $.05 per IOPS.
2012 - Micron unveiled a future hot swappable 2.5" PCIe SSD for
use in Dell servers.
February 2013 -
Virident Systems announced beta availability of a software suite which enabled
low latency shared server-side storage and high availability when used with the
company's range of PCIe SSDs.
June 2011 -
NVSL demonstrated the world's first PCIe PCM (phase-change memory) SSD.
2013 - LSI said it was #2 in PCIe SSD shipments in US - having shipped
over 40,000 units in the previous 12 months.
June 2013 -
Samsung has entered the PCIe SSD market with an
M.2 form factor model with
a sequential read performance of 1,400MB/s, and capacity up to 512GB.
2014 - A3CUBE unveiled a PCIe memory fabric for 10,000 node-class
architectures at the heart of which was a shared reflective RAM SSD engine.
2014 - Seagate agreed to acquire LSI's PCIe SSD and SSD controller
business for $450 million.
June 2014 -
SanDisk agreed to acquire Fusion-io for $1.1 billion.
January 2015 -
Toshiba demonstrated a prototype of the world's first PCIe single package SSD -
with up to 256GB in a single BGA package.
February 2016 -
A webscale reliability report by Google concluded it wasn't worth paying
more for SLC rather than MLC in enterprise PCIe SSDs.
|the 3 fastest flash
PCIe SSDs - list / lists|
|Are you tied up in
knots trying to shortlist flash SSD accelerators ranked according to
published comparative benchmarks?|
You know the sort of thing I mean -
where a magazine compares 10 SSDs or a blogger compares 2 SSDs against each
It would be nice to have a shortlist so that you don't have to
waste too much of your own valuable time testing unsuitable candidates
StorageSearch's long running
fastest SSDs directory
typically indicates 1 main product in each form factor category but those
examples may not be compatible with your own ecosystem.
my article -
the 3 fastest PCIe
SSDs list (or is it really lists?) may help you cut that Gordian
knot. Hmm... you may be thinking that StorageSearch's editor never gives easy
answers to SSD questions if more complicated ones are available.
||But in this case you'd be
wrong. (I didn't say you'd like the answers, though.) ...read the article|
SSDs Technical Pros and Cons|
|The great attraction of PCIe for SSD
oems is that it can support a wide range of performance options with throughput
upto 16GB/s, and much lower attachment costs than the
older busses like PCI and cPCI also provide performance which is adequate for
Bus connected SSDs have been around since the
of the SSD market.
The advantage of this approach is high
throughput and low latency compared to SSDs connected via traditional hard disk
style interfaces like
there are disadvantages too which include:-
1 - Bus style
interfaces reduce the available market for the SSD oem. Because older servers
may not have the interface, or perhaps the interface (for example Sun's SBus) is
proprietary and is only available in a small range of models.
2 - Bus
interfaces tend to have shorter permissable cable lengths - which restrict how
such SSDs can be connected.
3 - Bus interfaces usually don't include
intrinsic end to end error detection and correction. If the physical arrangement
of the SSD pushes the speed and cable lengths too far - then errors can arise in
the bus connect - which have to be dealt with in the associated driver.
May 13, 2009 - Dolphin's
CTO, Venkat Krishnan emailed this article correction.
StorExpress addresses concerns of PCIe direct attached SSDs raised in (2) above.
It includes support for different types of PCIe interfaces (ExpressModule, AMC,
etc.). Multiple PCIe SSD cards can be used without requiring multiple PCIe slots
in the server. The storage can be collocated at distances of up to 300m from the
server and can also be potentially shared by more than one server."