PCIe SSDs for use in enterprise server acceleration have been shipping
in the market since
one of the
most popular SSD
subjects pursued by our readers and has been (and has been since
2009 - when we
first told you that searches for PCIe SSDs on
StorageSearch.com - had overtaken
searches for 2.5"
SSDs on this site for the first time in
With over 100 million enterprise PCIe ports already shipped in
the market (upto January 2014) and with waiting-to-be-exploited fabric
capabilities which chipmaker PLX
(by which they mean PCIe as a fabric outside the box but still within the
cabinet) the fast changing and converging PCIe SSD / server market is
well positioned to expand into new applications.
Over 40 companies
already ship enterprise accelerator PCIe SSDs (see list below).
That will rise to over 100 companies as the availability of more
SSD controller chips,
other SSD related chip sets and
IP and SSD software
for this market will make it even easier than it already is for newcomers to
enter the PCIe SSDs market.
PCIe SSDs come in several shapes and
sizes. The most familiar form factor is cards, modules and racks. But a new
form factor - for
2.5" PCIe SSDs
which emerged in
2012 are opening up new applications - such as the displacement of fast
the 2nd half of 2013 - we also saw the start of another type of deployment for
the PCIe interface in the M.2 form factor which is aimed at the
consumer SSD market
notebooks. These consumer PCIe products have throughputs similar to the
enterprise products of 5-6 years earlier - but aren't rated for heavy
Nevertheless I wouldn't be surprised to see them appear as
in some future arrays in read-intensive design slots.
competitive alternative to PCIe SSDs - at the low latency end of the
applications spectrum inside (rather than
outside) the server box -
which started shipping in volume in 2014 - is a new class of low latency flash
SSDs which plug into DIMM sockets which were originally designed for
read more about these products (which go by various names such as TeraDIMMs,
ULLtraDIMM SSDs, eXFlash DIMMs etc) in the
|don't all PCIe SSDs
look pretty much the same?|
|When you look at the
photos and headline specs for high speed PCIe SSDs - it's easy to come away with
the impression that they all look the same and have about the same performance.|
all - how different can they be?
But don't let the experience of the
SAS markets - in which
enterprise SSD vendors cluster close together in performance rankings - give
you the wrong idea about PCIe SSDs.
In this market the performance
limits and capabilities of the SSD aren't set by fossilized
hard disk interface
and package limitations. They're limited only by the imagination of the
designers - tempered by the
educated guesses of
marketers who are trying to predict the optimum (most salable) features for
an ideal server side enterprise SSD.
|how fast can your
SSD run backwards?|
|SSDs are complex devices and there's a
lot of mysterious behavior which isn't fully revealed by benchmarks, datasheets
and whitepapers. |
Underlying all the important aspects of SSD behavior
which arise from the intrinsic technologies and architecture inside the SSD.
|Today's commodity MLC flash
has raw wear-out in the 2,000 to 3,000 write cycle range. And the future
direction is downwards (towards worse).|
|SSD endurance - the
oems list |
|This is a (partial list) from
past SSD news and
Mach Xtreme Technology,
Renice Technology ,
, Texas Memory
|the Problem with
Write IOPS in flash SSDs |
|Random "write IOPS"
in many of the fastest flash SSDs are now similar to "read IOPS"
- implying a performance symmetry which was once believed to be impossible.|
why are flash SSD IOPS such a poor predictor of application performance?
why are users still buying
RAM SSDs which cost an
order of magnitude more than SLC? (let alone
MLC) - even
when the IOPS specs look superficially similar?
tells you why the specs got faster - but the applications didn't.
||And why competing SSDs with
apparently identical benchmark results can perform completely differently.
|many years ago -
|BiTMICRO & CENATEK collaborate on PCI SSDs|
2002 - BiTMICRO.
announced today, a technology and marketing partnership to investigate
developing a hybrid solid state disk storage solution that brings together the
best in flashdisk storage and PCI bus-attached SSD technology.
Prices are likely to
range between $1,000 to $2,000 per gigabyte.
Editor's later comments:- the fast PCI SSDs which later
emerged from CENATEK - can be seen as immediate ancestors of the modern PCIe
But the concept of using fast bus based SSDs as storage
accelerators wasn't a new idea. It went back 3 to 4 decades.
earlier market experiments with solid state storage were always short lived -
because the expensive SSD storage in each product generation was always
competing with fast changing improvements in CPU clock speeds, bus memory
throughput or faster external magnetic storage media.
The changes in
the modern era of
SSDs - which started about 2003 - was that due to those other computer
technologies stagnating and not getting any faster - the only competitor which
killed an SSD from about that time - was another SSD.
By that time -
the new computer bus was known to be PCI express. But it was the failure of the
alternatives to solid state storage to get faster - which made the big
difference to the business viability of SSDs - rather than any inate
characteristic of PCIe SSDs.
|2.5" PCIe SSDs |
the fastest SSDs
SSD endurance -
what changed in SSD
fast can your SSD run backwards?
PCIe SSDs versus
memory channel SSDs
don't all PCIe SSDs
look pretty much the same?
SSDs - the Survive and Thrive Guide
conversation with PLX about PCIe fabric etc
what's so different about the
design of Fusion-io's PCIe SSDs?
SSDs - state of the market & technology- FMS 2014 (pdf)
news below / SAS SSD
news / other SSD news|
Fusion-io's Atomic SSD|
Editor:- October 21, 2014 - Earlier this
year - in June 2014
- and just weeks before SanDisk announced it
was acquiring the company - Fusion-io launched a
new generation of PCIe
SSDs - the Atomic Series - which was an overdue catching up exercise to
use cheaper 20nm flash.
At the time the headline product specifications seemed to me to be
adequate - rather than spectacular.
It's probably safe to assume they
were intended to restore the competitiveness of Fusion-io's products rather
than push performance boundaries.
But what are they like? And how
do they compare?
A new article -
ioMemory PX600 Review - published in StorageReview.com provides the
answers to how it performs and how it compares on popular OS platforms and
benchmarks. ...read the
how fast can your
SSD run backwards?, SSD
testing & analyzer news
Samsung mass produces 3TB 3D PCIe SSDs
September 25, 2014 - Samsung
today announced it has started mass producing 3.2 TB NVMe
PCIe SSDs (HHHL)
based on its 3D flash memory
technology, for use in enterprise systems.
new NVMe PCIe SSD, SM1715 provides a sequential R/W speeds upto 3GB/s and
2.2GB/s respectively with
rated at 10 DWPD for 5
Editor's comments:- Samsung supplied an
for the new SSD - but forgot to include the block size to which it relates - so
I've asked for clarification.
Diablo is the #1 SSD company in September (so far)
September 23, 2014 - This page is about PCIe SSDs - right?
So why am I
going to use this space to briefly talk about Diablo? - pioneer of
I give you 2 reasons in
point is that - due to recent developments in PCIe SSD fabric... having
SSDs located in a DIMM socket in one server - doesn't preclude that very same
data being accessed by another server as if it were just a locally installed
PCIe SSD any more.
That's created a lot of buzz ...read more in SSD news
Seagate launches new improved Nytro PCIe SSDs
September 10, 2014 - Seagate
2 new PCIe SSDs -
which are based on the SSD product lines and brand assets of the recently
acquired SSD business of LSI.
XP6302 is a HHHL, gen 3 PCIe SSD - which provides up to 1.75 TB of
usable eMLC capacity with 200 microseconds average latency, and 295K/79K
(8KB) and rated for 0.9 DWPD
endurance for 5 years.
XP6210 is a FHHL gen 2 PCIe SSD with 1.86TB usable 19nm cMLC
capacity, with 50 microseconds average latency 185K/120K R/W IOPS (8KB),
and rated at 1.6 DWPD
endurance for 5 years.
HGST announces 2nd generation clustering software for FlashMAX
Editor:- September 9, 2014 - HGST today
a new improved version of the
clustering capability previously available in the
PCIe SSD product line
acquired last year from Virident.
allows clustering of up to 128 servers and 16 PCIe storage devices to deliver
one or more shared volumes of high performance flash storage with a total usable
capacity of more than 38TB.
HGST says its Virident HA provides a "high-throughput,
low-latency synchronous replication across servers for data residing on FlashMAX
PCIe devices. If the primary server fails, the secondary server can
automatically start a standby copy of your application using the secondary
replica of the data."
For more details see -
Virident Software 2.0 (pdf)
Editor's comments:- This
capability had already been demonstrated last year - and
ESG reported on the
technology in January
But at that time - the clustering product called vShare -
was restricted to a small number of servers - and the data access fabric was
restricted to Infiniband
With the rev 2.0 software - the number of connected devices has
increased - and users also have the lower cost option of using
Ethernet as an alternative
OCZ samples hot swap, fast 2.5" NVMe SSDs
September 9, 2014 - OCZ
that this month it will begin sampling a new
swappable enterprise PCIe SSD - the Z-Drive 6000 - is a native PCIe
3.0 NVMe 1.1 solution - which the company says "provides industry-leading
IOPS per dollar". It has a SFF-8639 connector, internal RAID, power
loss data protection, "consistent low latency", and encryption.
comments:- Although OCZ demonstrated the SSD industry's first working 3.5"
PCIe SSD prototype 4 years ago - in
August 2010 - the
company didn't follow through to establish an early lead in its natural
successor - the 2.5" enterprise PCIe market - where current leaders today
are Micron and
Samsung. The main
reason for that was financial problems in OCZ in the years which
immediately followed - which mitigated against products which didn't have a
short term guaranteed market.
Now, however, with OCZ having been
almost a year as a Toshiba group company - the small form factor enterprise
NVMe market looks like a natural fit for OCZ - as an extension of its long
running conventional form factor PCIe SSD accelerator business.
StorageSearch.com updates 10 key SSD ideas in 2014
September 5, 2014 - Are SSD years getting faster? It's not just the SSDs.
Anyway it seemed to me that an end of year SSD market update was due
even though 1/3 of this year hasn't happened yet.
That's why StorageSearch.com
this week published a new home page blog -
updating 10 key SSD ideas in 2014.
As you might expect - PCIe SSDs come into the picture. But there's
more to it than that. If you're impatient - you don't have you don't have to
wait till January 2015. ...read
Plextor's M.2 PCIe SSD wins award at FMS
August 7, 2014 - Plextor
- an M.2 PCIe SSD - has won Best of Show for most Innovative flash memory
technology at the Flash Memory
The M6e SSD combines a multi-core
Marvell PCIe 9183
controller and Toshiba
toggle NAND flash with firmware developed by Plextor's in-house team.
say hello to Shannon Systems
Editor:- August 6, 2014
- I hadn't heard of Shannon Systems
before. But I got a nice email this morning from Xueshi Yang,
CEO and co-founder who said he has been reading StorageSearch.com "for
quite a number of years now" and also said that his company is showing
their products at Flash Memory
Among other things - Xueshi Yang said - "Shannon
System is a startup I co-founded in 2011 in China after I left
company is dedicated to the enterprise flash storage market. Currently, we focus
on the high performance PCIe market with our proprietary controllers and
software systems. In April this year, we
the industry first 6.4TB PCIe SSD with a single controller, which boasts 67us
read access latency and 9 us write access latency (all in 4KB, random). While in
June, we introduced a PCIe SSD with SFF-8639 interface, which is hot-pluggable.
We currently serve over 100 customers in China, including Tier 1 internet
companies, as well as other named customers such as China Mobile, China Telecomm
eASIC supports Mobiveil's NVMe platform
August 6, 2014 - eASIC
announced support for Mobiveil's
NVMe platform (pdf)
implemented in eASIC devices.
"eASIC is enabling the rapid
deployment of SSD technology at substantially lower cost and up to 70% lower
power than alternative solutions", said Jasbinder Bhoot,
VP of Worldwide Marketing at eASIC. "By working with Mobiveil, customers
will have access to a complete NVMe solution running in cost, power and
performance optimized eASIC devices."
HGST rekindles concept of a PCM based PCIe SSD
August 4, 2014 - HGST
it will demonstrate a PCM PCIe SSD concept at the Flash Memory Summit. HGST says
the demonstration model delivers 3 million random read IOPS (512 Bytes) and
a random read access latency of 1.5 microseconds.
funded the world's first enterprise PCM PCIe SSD demo 3 years ago (in
June 2011). The
storage density of PCM resulted in an SSD which had pitifully low capacity
compared to flash memory at that time - and earlier this year (in January 2014)
that Micron had temporarily abandoned this idea.
Is HGST really
going to wander into memory space where even the memory makers don't want to go?
Or is this just a market signal that HGST isn't just looking at short term SSD
Lite-On enters enterprise M.2 PCIe SSD market
July 30, 2014 - Lite-On
will be showing a new M.2 PCIe
SSD for the enterprise market next week at the Flash Memory Summit.
The P1P is a
PCIe SSD with capacities of 1TB and
protection circuit all in a small form factor.
The small form
factor and high capacity allow enterprise customers to pack more storage in a
smaller footprint. The P1P Series can deliver sequential R/W speeds up to
610MB/s and 520MB/s while random R/W speeds can be upto 95K/15K
drive has a MTBF
of 2 million hours and an
rating of up to 1 drive write
per day for 5 years.
the Top SSD Companies in Q2 2014
Editor:- July 28,
2014 - StorageSearch.com
today published the new
29th quarterly edition of the
Top SSD Companies -
based on metrics in Q2
OCZ announces availability of ZD-XL SQL Accelerator 1.5
July 15, 2014 - OCZ
this week announced availability of
1.5 of its ZD-XL SQL Accelerator (PCIe SSD and caching software bundle) the
beta version of which was announced in
continuity of PLX's PCIe box platform?
10, 2014 - Yesterday a reader asked if I thought that the PCIe fabric system
- which I had written about last month in the article -
conversation with PLX - would still be available as a product after the
acquisition of PLX
by Avago Technologies closed?
pointed out that Avago avoids competing with its own systems customers (which is
why they divested the SSD business from LSI) and also Avago isn't a systems
Here's what I said...
The points you make are valid
and I had wondered the same thing myself.
But I think the PCIe fabric box (ExpressFabric solution)
will continue in some form or other because it started out as a system
design kit for demonstrating what could be done with the next generation of PCIe
chips and software stacks. So it's an essential sales tool for the chip
Before the announcement from Avago the PLX guys had already
said that any of their customers would be welcome to use as much or as little as
they wanted from the SDK box as PLX wasn't originally advancing this as a
But I could see from my own judgement (and PLX
confirmed this) that for some customers having availability of such a
box as a product would be a convenient tool.
Incidentally - that's
how Intel got into the systems business - with their
SBC product range in the 1970s - which was a response to customers asking
if they could buy the boards which were in the early microprocessor development
Returning to Avago and the PCIe fabric SDK
I agree -
it's possible that Avago may decide not to get into the volume supply
business of these boxes but in that case I think the boxes (and design
IP for these boxes) would continue to be available in some form from a
designated source otherwise lack of this integration concept tool would
slow down market adoption of the new PCIe fabric chips.
It's in the
interests of Avago and its oem customers to ensure that the ExpressFabric SDK
remains available as a software reference architecture which is at the center
of this new ecosystem.
Later:- I asked for an official response about
this (just in case I had missed anything) but - as I expected - the answer
was - it will come in a future press release.
Avago agrees to acquire PLX
Editor:- June 23, 2014 -
announced that it
has agreed to be acquired by Avago
Technologies for approximately $309 million.
SanDisk to buy Fusion-io
Editor:- June 16, 2014 -
a definitive agreement to acquire Fusion-io in
all-cash transaction valued at approximately $1.1 billion.
comments:- What will SanDisk really get from Fusion-io?
the short version from me.
"The ability to get more enterprise
petabytes out from the same raw flash chips in - by shipping it through better
architecture - is a more significant business factor in the flash memory market
today than the ability to do another cell geometry shrink - or to add a few more
toppings on the 3D pizza. "
For the much longer version(s) - see
my article in SSD news
PLDA enters the PCIe SSD controller market with 6.6 Gb/s NVMe
Editor:- June 9, 2014 - PLDA today
immediate availability of a new
SSD controller aimed
at the PCIe SSD market.
IP is a high performance, low latency, highly-configurable PCI Express 3.0
soft IP supporting endpoint, root port, switch and bridge configurations. The
certified design has been engineered for both ASIC/SoC and FPGA implementations.
comments:- PLDA has written a product description -
challenges of PCIe with NVMe in order to deliver highly competitive Enterprise
PCIe SSDs (pdf) - which contains an overview of the design. Here are some
- power saving features? - PLDA implements L1 PM Sub-states, reducing power
consumption without affecting performance.
Here are some questions I put to the
- performance? - 270K write IOPS, 12µS write latency write (4KB) using
the FPGA-based reference design.
- is there a minimum order quantity or cost associated with licensing your
No minimum quantity.
- What is your deliverable?
An Encrypted Source code file that can
be downloaded or sent on CD
- How does a customer evaluate your IP?
We provide 1 month Free Evaluation IP
new article re PCIe and SSDs
Editor:- June 5, 2014 -
published a new article -
conversation with PLX.
That concept diagram they've been talking about for the past 3 years -
may soon become a 1U PCIe fabric box reality which you can use to glue your
servers and SSDs together.
new PCIe SSDs from Fusion-io
Editor:- June 5, 2014 -
2 new gen 2.0 x8 standard height, half length PCIe SSD product lines - which
they call the Atomic Series
- which are available in 2 different price and application categories.
series - upto 5.2TB usable capacity, with 92µS / 15µS R/W
latency, 2.7GB/s / 2.1GB /s R/W throughput and 375K write IOPS - designed for
high end servers - with 5 years warranty.
Fusion-io says that more than 7,000 customers in
over 80 countries use its products to reduce latency and speed the flow of
- upto 6.4TB usable capacity - with similar latency to PX600 - but lower
throughput (2.6GB/s / 1.2GB/s R/W throughput) and 16% lower power consumption -
designed and priced for scale-out architectures and hyperscale environments -
with 3 years warranty.
Hynix buys Violin's PCIe SSD product line
May 29, 2014 - Today in line with previous guidance signals - Violin Memory (which in
January 2013 had
been observed to tentatively dip its toes as a late entrant into the raging
torrents of the PCIe SSD
market - but which had not been regarded by me as very likely to survive
a rigorous swim)
that it has sold its PCIe SSD product line to Hynix which paid $23
million in cash and assumed certain liabilities totaling $0.5 million.
Marvell samples controller for SATAe SSDs
May 21, 2014 - If you're designing SATA Express SSDs then Marvell today
it is sampling the 1st SSD
controller specifically designed for the SATAe market - which will
enable the design of 2.5"
PCIe SSDs at costs which could be
fast SATA SSDs.
88SS1083 is a 2 lane PCIe Gen2 SSD controller with transfer rates up to
1GB/s. It supports DevSleep and L1.2 PCIe low power state - to minimize power
consumption in notebook
and enterprise array
EMC acquires (memory channel SSD in a box but connected via PCIe
Editor:- May 5, 2014 - EMC has acquired a stealth
mode rackmount SSD company - DSSD
- it was announced
Products based on the new DSSD architecture - which connects
to servers via PCIe - are expected to be available in 2015.
comments:- for a competitive analysis and technology perspective see
rackmount SSD news
the Top SSD Companies in Q1 2014
Editor:- April 30,
2014 - StorageSearch.com
today published the 28th
quarterly edition of the
Top SSD Companies List
based on metrics in Q1 2014. Newcomers to the list included Maxta and A3CUBE and there were
significant movements in the top 10 companies. ...read the article
LSI's Nytro works with BPE in SQL Server 2014
April 24, 2014 - A lot of SSD companies have announced that their products
are compatible with Microsoft's SQL Server 2014. Another noteworthy company in
that category is LSI
- who today
flash accelerator cards (PCIe SSDs) work with the new
Pool Extension feature.
OCZ launches new desktop PCIe SSD
Editor:- April 24,
2014 - OCZ today
a new gen 2 PCIe SSD
for high performance desktop uses such as gaming and workstation. The Linux and
Windows compatible RevoDrive
350 (240GB, 480GB, and 960GB capacity models) has R/W speeds upto 1.8 / 1.7
GB/s and 140,000 4K random write IOPS.
Editor's comments:- a
key difference between this type of prosumer desktop SSD and the kind you need
in intensive 24x7 server slots is
OCZ says the RevoDrive 350 is rated at "upto 50GB of host writes
per day for 3 years". That may be OK for a desktop - but if you translate
that into enterprise jargon - DWPD
for 5 years - it translates into 0.1 - which is 40x lower than a typical
enterprise PCIe drive.
Fusion-io says - we don't regard NVM DIMMs as a threat to our
PCIe SSD business
Editor:- April 23, 2014 - Fusion-io today
revenue up 14% yoy to $100 million but losses have grown too - to $30
million for the recent quarter.
Editor's comments:- Here are
some technology points which emerged from the investor conference call.
- Threat to FIO's PCIe SSD business from
said - he thought NVM DIMMs were a validation of what FIO had already done -
but he regarded NVM DIMMs (in their current form) as a niche market. He went
on to say - "We don't believe that DDR-3 and the memory interface is the
bottleneck. It's about managing the nand and getting the best performance out of
- Re 3rd generation iodrives? - These are being qualified. FIO's 3rd
generation products - and the 5th generation of nand flash geometry used by the
company - will be available in the next month or so.
if memory channel SSDs cost nothing - would there still be a
market for PCIe SSDs?
Editor:- April 17, 2014 - StorageSearch.com today published
a new blog - memory channel SSDs versus
PCIe SSDs are these really different markets? - which revisits (from a
current perspective) some of the key questions I posed a year ago about
Memory Channel SSDs.
Are these really different markets? And - at
of price - what are
the different application roles - where one type of SSD is better than the
other? ...read the article
Editor:- April 16, 2014 - BiTMICRO recently
announced that Bharadwaj
Pudipeddi has joined the company as VP of Engineering and Lead
Architect. Pudipeddi's past design roles in notable SSD companies include 2
years at Violin Memory.
what's the market for Plextor's PCIe SSD?
April 7, 2014 - I asked Plextor's virtual
marketing representative in the US - Andrew Erickson at
Alaniz Marketing to tell me more
about Plextor's thinking about routes to market for their entry level (Gen2)
PCIe SSD market
offering - the
said - "The M6e is marketed directly to consumers as an end-user upgrade
and mostly via newegg and major e-tailers."
Andrew went on to say - "Plextor is definitely moving into the
enterprise space and we're expecting to see a couple of read intensive and
multi-use ssd products later this year. It sounds like some of the enterprise
firmware advances may already be evident in the M6e - though it's not being
marketed in that way. I have seen some Plextor drives sold in high end gaming
rigs, but I'm not aware of current plans to sell OEM at this time."
Samsung's new 2.5" NVMe PCIe SSD
March 25, 2014 - Although Samsung's own
meat navigation inspired
SSD selection page doesn't list this product yet (at the time of writing
this) Samsung is engaged in the
2.5" PCIe SSD
market - and is shipping a 1.6TB NVMe PCIe SSD rated at
7 DWPD for 5
years to Dell for use
in its PowerEdge
PCIe Switching - new article in EnterpriseTech.com
March 13, 2014 - A new article in EnterpriseTech.com
Switching Takes on Ethernet, InfiniBand - reviews the antecedents and
current state of the PCIe fabric market.
This should be of interest
to anyone thinking about the emerging architectural influences which may
impact their plans within the
PCIe SSD market.
of the many vendors discussed in the article - PLX - says it's "not
targeting warehouse-scale datacenters... but is rather thinking on a smaller
scale, from hundreds to thousands of nodes." ...read
a new SSDserver nomenclature?
Editor:- March 5, 2014
- In a new home page blog today - StorageSearch.com
published proposals for a new shorthand for SSD enhanced enterprise
servers in the context of complex architectural dialogs. ...read the article
(You won't need this if all your projects look the same, or if your plans only
include small numbers of SSDservers.)
OCZ launches Z-Drive 4500 - 19nm enterprise PCIe SSD
March 4, 2014 - OCZ
is still using LSI's
SandForce SSD controllers
SF-2582 enterprise SATA (pdf)) in its newest PCIe SSD - the
Z-Drive 4500 Series
today - which has upto 3.2TB of usable
19nm flash, R/W bandwidth
of 2.9GB/s and 2.2 GB/s respectively, and 252K / 76 K R/W
in a FHHL form factor and is integrated with OCZ's new WXL caching software.
4500 briefing notes (pdf)
Editor's comments:- As I
reported in an interview
last November - OCZ's VXL bundles have been very successful in small to
medium scale enterprise deployments. The evolution of this product line -
supporting as it does another new generation of (lower cost) memory - will
further extend its reach.
A3CUBE unveils PCIe memory fabric for
PCIe SSD architectures
Editor:- February 25, 2014 -
PCIe SSDs can now
access a true PCIe connected shared memory fabric designed by A3CUBE - which exited
stealth today with the
of their remote shared broadcast memory network -
RONNIEE Express -
which provides 700nS (nanoseconds) raw latency (4 byte message) and which
enables message throughput - via standard PCIe - which is 8x better
Editor's comments:- I spoke to the
recently - who say they intend to make this an affordable mainstream
The idea of using PCIe as a fabric to share data at low
latency and with fast throughput across a set of closely located servers
isn't a new one.
The world's leading PCIe chipmaker
PLX started educating
designers and systems architects about these possibilities
few years ago - as a way to elegantly answer a new set of scalability
problems caused by the increasing adoption of PCIe SSDs. These questions
- how do you make this expensive resource available to more servers?
the least year or so - we've seen most of the leading vendors in the enterprise
PCIe SSD market leverage some of the new features in PCIe chips - to
implement high availability SSDs with low latency.
- how do you enable a simple to implement failover mechanism - so that data
remains accessible in the event of either a server or SSD fault?
But although there
are many ways of doing this - the details are different for each vendor.
- until now - if you wanted to share data at PCIe-like latency across a bunch
of PCIe SSDs from different companies - located in different boxes - the
simplest way to do that was to bridge across ethernet or infiniband. - And even
though it has been technically possible with standard software packages - the
integration, education and support issues - compared to legacy SAN or NAS
techniques would be extremely daunting.
That's where A3CUBE comes into
the picture. Their concept is to provide a box which enables any supported PCIe
device to connect to any other - at low latency and with high throughput -
in an architecture which scales to many thousands of nodes.
heart of this is a shared broadcast memory window - of 128Mbytes - which can be
viewed simultaneously by any of the attached ports.
ever used shared remote memory in a supercomputer style of system design at
any time in the past 20 years or so - you'll know that the critical thing is how
the latency grows as you add more ports. So that was one of the questions I
Here's what I was told - "The latency is related to the
dimension of the packet for example: In a real application using a range of
64-256 bytes of messages the 3D torus latency doubled after 1,000 nodes.
With larger packets, the number of nodes to double the latency becomes grater.
But the real point is that the latency of a simple p2p in a standard 10GE is
reached after 29,000 nodes.
"A more clear example of the scalability of the system is this.
Imagine that an application experiences a max latency of 4 us with 64 nodes, now
we want to scale to 1,000 nodes the max latency that the same application
experience will became 4.9 us. 0.9 us of extra latency for 936 more nodes."
Editor again:- Those are very impressive examples - and demonstrates that the
"scalability" is inherent in the original product design.
didn't want to say publicly what the costs of the nodes and the box are at this
stage. But they answered the question a different way.
Their aim is to
price the architecture so that it works out cheaper to run than the legacy
(pre-PCIe SSD era) alternatives - and they're hoping that server oems and fast
SSD oems will find A3CUBE's way of doing this PCIe fabric scalability stuff -
is the ideal way they want to go.
There's a lot more we have to learn
- and a lot of testing to be done and software to be written - but for users
whose nightmare questions have been - how do I easily scale up to a 10,000
PCIe SSD resource - and when I've got it - how can I simplify changing
suppliers? - there's a new safety net being woven. Here are the
PLX is ready with 1st Gen3 compliant PCIe switches
February 4, 2014 - PLX
announced it is
the 1st PCIe switch vendor to have achieved Gen3 compliance having passed the
compliance testing procedures of PCI-SIG.
Editor's comments:- the new standard doubles the maximum
data bandwidth of PCIe SSDs to 1GB/s per lane in each direction - which enable
32GB/s total throughput for a x16 link.
SSD interface chips,
Express 3.0 Integrators List
ESG reports on clustering Virident's PCIe SSDs
January 23, 2014 - ESG
today published a
report for Virident
/ HGST's FlashMAX
II (PCIe SSDs)
which validated the ability of this product family and its
(vShare ) to be configured for useful operation in an Oracle RAC environment
availability configurations which were clustered via
ESG said performance was good they also commented on some current
limitations of the product suite for this type of application. In particular:-
- the lack of a graphical interface for setup and performance mintoring,, and
- the lack of support for other supported fabrics such as 10GbE (mentioned
in the report as a future option), and PCIe fabric (which was not mentioned at
all in this report).
This is what it takes to be #2 in PCIe SSDs
January 22, 2014 - LSI
said today in its
financial report it has shipped over 100,000
PCIe SSDs since it
entered this market in April 2012.
Editor's comments:- LSI
shipped 60,000 units in the 8 months from May to December 2013 (a figure
calculated by subtracting the numbers contained in an
Despite those impressive numbers - LSI itself
estimates that it is still the #2 company in this market. Which gives you an
idea of how large this market has grown.
IBM launches first memory channel SSD based servers
January 16, 2014 - IBM
today disclosed preliminary test results comparing the latency of
SSDs to PCIe SSDs - in the
press release of its new X6 architecture for X86 servers.
- "Our evaluators are seeing 5-10 microseconds write latency for eXFlash
DIMMs in preliminary testing vs. 15-19 microseconds latency for PCIe-based flash
storage from Fusion IO,
Virident, and 65
microseconds latency for Intel
S3500 and S3700 SSDs."
IBM also announced enhancements and sales
figures for the rackmount
SSD product line it acquired from
Texas Memory Systems a
year ago. See SSD news for
more about the latter.
reasons to expect more PCIe SSDs in 2014 PC Market
December 16, 2013 - A new
in DRAMeXchange discusses the issue of
SATA 3 SSDs being
replaced by PCIe SSDs
in the consumer SSD
market in future designs of PCs. ...read
how to avoid hot pluggable PCIe SSD failures
December 3, 2013 - What happens if you test
PCIe SSDs for their
sensitivity to data
corruption or even failure
- in the event of
You'd think that with more hot pluggable products coming
into the market -especially in the
2.5" form factor
- that the experimental outcomes would be known by the designers and problems
debugged so that users wouldn't have to worry.
In August 2013 Quarch Technology
launched some special
test equipemnt to inject power related faults into PCIe SSDs - and the
company today announced
it has extended this range to automate power line error testing of PCIe SFF
Technical Director at Quarch Technology told me today that "Almost every
combination of test kit we have tried in Quarch (a number of friendly customers
lent us kit and eval drives to get the new module up and running) has failed in
some way. Sometimes failing to come back up again nicely, sometimes with a full
BSOD which will almost
certainly have risked data loss."
estimating the size of the server-side SSD market etc...
October 8, 2013 - Before you make that next presentation about what's happening
in the business world of enterprise flash, or before you commit to any future
datelines for hard drives being sold into the enterprise - you'd be well
meet Ken and the
enterprise SSD software event horizon - the (long anticipated) new home
page blog on StorageSearch.com.
new route to market for FIO's ioScale
September 26, 2013 - IBM
- which has been offering Fusion-io's ioDrive
(PCIe SSD) technology
in its product line since
December 2009 has
first server oem to remarket the
ioScale - a product
described as having been optimized (price and feature set-wise) for huge
end users in the SSD
dark matter segment at the time of its launch in
Demartek blog re NVMe
Editor:- September 19, 2013 -
PCIe SSD market was
threatened with standards
which might tame it several years ago. The state of play is assessed in a new
IDF2013 and NVMe written by Dennis Martin, President
LinkedIn uses Virident SSDs
Editor:- August 28, 2013
- Virident Systems
that LinkedIn is a customer of its
Skyera offers PCIe attach to petabyte class SSD
August 13, 2013 - Skyera
that. among other things, it will introduce
PCIe connectivity (to
the existing FC and
iSCSI) as well as
replication in the next version of its
rackmount SSDs -
the skyEagle - which will ship in the first half of 2014 - offering 500TB
uncompressed (2.5TB deduped and compressed) in a 1U form factor at a record
breaking list price
expected to be under $2,000 per uncompressed terabyte.
SMART starts sampling ULLtraDIMM SSDs
8, 2013 - SMART
Storage Systems today
it has begun sampling the first memory channel SSDs compatible with the
interface and reference architecture created by Diablo Technologies.
first generation enterprise ULLtraDIMM SSD (ULL = ultra-low latency) can be
deployed via any existing DIMM slot and provides 200GB or 400GB of enterprise
class flash SSD memory with upto 1GB/s and 760MB/s of sustained read/write
performance, with 5 microseconds write latency. Throughput, IOPS and memory
capacity scale with the number of ULLtraDIMM deployed in each server.
Samsung enters the 2.5" PCIe SSD market
July 18, 2013 - Samsung
today announced its entry into the
2.5" PCIe SSD
market. Its new NVMe SSD has upto 1.6TB capacity, read throughput upto
3GB/s, and up to 740K
PCIe topologies for enterprise SSDs
17, 2013 - PLX Technology
recently published a white paper -
and PCI Express - which gives an overview of past, current and future PCIe
SSD connection topologies along with a list of detailed reference articles.
with the clear text and authoritative viewpoint (PLX is the leading supplier of
PCIe chips to the SSD industry) the article includes a complete set of
attractive pictures too - which make it easier to visualize the many different
interconnection methods being discussed. Below is an example from a section in
the paper - called - Taking PCIe Out of the Box.
In recent years we've seen the start of a growing
diversity in both
the type and functionality of
PCIe SSDs. PLX's new
article provides a good introduction to what can be done with PCIe in an SSD
context - and may make you rethink your ideas about the roles of this interface
interface chips and IP,
7 silos for enterprise
Samsung enters PCIe SSD market
Editor:- June 17,
2013 - Samsung
has entered the PCIe SSD
market with an M.2 form factor model (80mm x 22mm) aimed at
Samsung's XP941 - which weighs less than 6g - has a sequential read
performance of 1,400MB/s, and capacity up to 512GB.
market began the year before PCIe SSDs started being used in the
But in the first 5 years of its history (2006-2010) the
notebook SSD market was a disappointment to SSD evangelists like me - because
integration with PCs was so bad. And for years on these pages I ranted that
notebooks using SSDs would never be able to reach their true potential as long
as they were still wasting their inherently light CPU resources and latency
advantages by talking to the CPU via old fashioned
hard disk interfaces
exciting thing about today's announcement by Samsung is that consumer grade
PCIe SSDs for notebooks will enable a dramatically different user experience
which will help to create new markets.
Will there be a crossover into
the enterprise market?
It's inevitable that some people will ask -
what would an array of consumer priced PCIe SSDs look like in a box? And no
doubt you will probably see such products coming onto the market. And that
might lead to a temporary state of user confusion about expectations for PCIe
But setting aside for the moment the obvious considerations at
the single drive level of differences in
characteristics - I think the key differentiators of enterprise PCIe SSDs
compared to consumer
PCIe SSDs are the different degree of
(higher for the enterprise),
management and support for fault tolerance.
Violin's new blog on PCIe vs FC SAN SSDs
June 11, 2013 - How much confidence do you have in the
of the SSD system which you're deploying? - And how different are the
reliability costs when you scale
PCIe SSDs compared to
are some of the issues discussed in a new blog by Violin -
Memory: too Array or to Card - written by the company's CTO of Software -
Goldick - who warns that when you're estimating the latency
advantages between different ways of connecting SSDs - "Always be
cognizant that you may just be moving the
to the software."
Micron turns up the heat for adoption of 2.5" PCIe SSDs
May 3, 2013 - Micron
it's sampling a new model in the
hot swappable 2.5"
PCIe SSDs market - the
has upto 1.4TB MLC capacity and can deliver 750K R IOPS.
QLogic enters the enterprise SSD market - with SAN IOPS caching
Editor:- March 22, 2013 - QLogic yesterday
entered the enterprise
SSD market (in the PCI
SSD and SSD ASAPs
segments) with the
its first product - the
10000 Series adapter (pdf) - which provides transparent sharable and
clusterable caching for FC SANs.
The 2 card set (upto 400GB flash,
and 2x 8Gbps FC ports) can deliver upto 310,000 initiator IOPS and
supports upto 2,048 concurrent logins.
QLogic says - FabricCache uses
just a single standard adapter driver per OS, the same QLogic field-proven
adapter driver stack that has shipped with more than 13 million FC ports. The
FabricCache adapter appears logically to the server as a single QLogic HBA which
integrates seamlessly into current enterprise server and infrastructure
Editor's comments:- from the business and
marketing perspective QLogic's new Mt. Rainier branded SSD is a tardy,
pragmatic and cunning response to the threats and challenges posed by SSDs
to traditional storage ecosystems vendors.
Tardy - because
the first SSD ASAP appliances - which used SSDs to automatically and
transparently accelerate access to
SAN storage first shipped
in 2009. And
the first enterprise PCIe SSDs started shipping in 2007.
- for a bunch of reasons
- Pragmatic - because QLogic's new products provide a credible way for the
company to participate and get customer experience in the crowded high
value PCIe SSD market - by recycling IP from (what would otherwise soon be
downgraded to) a commodity IP base (FC SAN physical adapters).
my 2011 article - don't
all PCIe SSDs look pretty much the same? - SSD ASAPs (like QLogic's
FabricCache ) were discussed as one of the many options for market
- because the implied message to users here is:- "if you want to fool
around with SSD auto-accelerating your SAN storage - you can do it without
haveing to learn about new software. - All the software you need to know about
is already running your interface HBAs."
- Pragmatic - for another reason too - because with 50 or more different
flavors of auto-caching/tiering software schemes already in the market - QLogic
has neatly side-stepped the issue of participating as yet another new
SSD software company
which user have got to learn about - and instead has cleverly positioned its
product as needing no new software. (The FabricCache adapters learn everything
they need to know from being acticated by standard FC HBA drivers.)
This is kind of sneaky -
because in reality - if users are investing in this product - they will soon
start to care about the internals of the software - and to ponder on the
ability of QLogic to continue delivering good
symmetry across upcoming new generations of back end SAN storage systems as
these SAN racks themselves change their internal make-up from hard drives to
Another consideration is that - accelerating data I/O between a
traditional server SAN stack and storage on the SAN - may give better results
than without any flash - but may not be as good an ROI as the competing
alternative of accelerating apps performance using similar flash capacity
but with an interface agnostic bunch of algorithms.
QLogic's new SSDs provide a "low perceived risk" way for users -
who don't want to plunge deeply into the mystical and changeable details of
SSD acceleration - to experiment and fool around with the benefits of SSD
Before I get any emails from investors asking - is
this good for QLogic? or - is it bad for
Virident)? Here are
Companies in the server SAN infrastructure market have no
future without an SSD core. So having a plausible SSD product - even at this
late stage of the enterprise SSD market development game- upgrades QLogic from
a future as a mere commodity interface chip supplier to a plausible player.
I doubt if any PCIe SSD makers feel in any way threatened by QLogic's new
See also:- QLogic's informative white paper -
Mt. Rainier Integrates SSDs with SAN Connectivity (pdf)
9 million IOPS in a single PCIe SSD
5, 2013 - Fusion-io
it has achieved 9.6 million
byte) from a single 365GB
ioDrive2 (PCIe SSD).
performance is made possible using APIs in
SDK (such as Auto-Commit Memory) which integrate flash into host systems,
allowing data to bypass normal bottlenecks in the OS.
FIO says its
APIs have been embraced by dozens of industry-leading
software companies to
enhance their applications.
|archived versions of this PCIe SSDs news page
|the SSD Buyers Guide|
where are we now with SSD software?
SSDs for newbies - what are they etc
|the 3 fastest flash
PCIe SSDs - list / lists|
|Are you tied up in
knots trying to shortlist flash SSD accelerators ranked according to
published comparative benchmarks?|
You know the sort of thing I mean -
where a magazine compares 10 SSDs or a blogger compares 2 SSDs against each
other. It would be nice to have a shortlist so that you don't have to waste too
much of your own valuable time testing unsuitable candidates wouldn't it?
StorageSearch's long running
fastest SSDs directory
typically indicates 1 main product in each form factor category but those
examples may not be compatible with your own ecosystem.
If so a
new article -
the 3 fastest PCIe
SSDs list (or is it really lists?) may help you cut that Gordian
knot. Hmm... you may be thinking that StorageSearch's editor never gives easy
answers to SSD questions if more complicated ones are available.
||But in this case you'd be
wrong. (I didn't say you'd like the answers, though.) ...read the article|
|How are fault tolerant
PCIe SSD designs supported in chips?|
enterprise SSD designs - this video by PLX includes an
introductory tutorial into PCIe and its performance and architectural
capabilities for SSDs including automatic failover and multi-host capabilities.|
|PLX's switch chips also supports failover if
the fault occurs in the PCIe switch fabric chips themselves.
extract - "...And in case one of the hosts fails
and you want to connect the SSDs - or the devices connected to that host - to
another host - that can be done automatically as well - and the surviving host
can attach the devices that were attached to the failing host to itself and
control it so that the system doesn't go down and the data stored in these
devices doesn't get isolated from the main system."
|How Big is the Market for
|In 2010 I wrote here that I confidently
expect that PCIe SSDs will become a multi-billion dollar a year market.|
now the standard SSD
Since 2010 most of
the top 10 SSD companies
also sold PCIe SSDs.
For more about PCIe SSDs fit into the enterprise SSD landscape see -
introduction to SSD
|"This is a tsunami warning
event for SSD vendors addressing the enterprise server acceleration market."|
|...editor's comments (September 24, 2009 ) when
I alerted readers and vendors to the fact that search volume for PCIe SSDs
had surpassed that for 2.5" SSDs for the first time. |
type of search spike had been a reliable advance
predictor for new
interfaces - such as SATA
and iSCSI - in earlier
phases of storage
history - and search volume has also been a good predictor for
successful SSD companies
SSDs Technical Pros and Cons|
|The great attraction of PCIe for SSD
oems is that it can support a wide range of performance options with throughput
upto 16GB/s, and much lower attachment costs than the
older busses like PCI and cPCI also provide performance which is adequate for
Bus connected SSDs have been around since the
of the SSD market.
The advantage of this approach is high
throughput and low latency compared to SSDs connected via traditional hard disk
style interfaces like
there are disadvantages too which include:-
1 - Bus style
interfaces reduce the available market for the SSD oem. Because older servers
may not have the interface, or perhaps the interface (for example Sun's SBus) is
proprietary and is only available in a small range of models.
2 - Bus
interfaces tend to have shorter permissable cable lengths - which restrict how
such SSDs can be connected.
3 - Bus interfaces usually don't include
intrinsic end to end error detection and correction. If the physical arrangement
of the SSD pushes the speed and cable lengths too far - then errors can arise in
the bus connect - which have to be dealt with in the associated driver.
May 13, 2009 - Dolphin's
CTO, Venkat Krishnan emailed this article correction.
StorExpress addresses concerns of PCIe direct attached SSDs raised in (2) above.
It includes support for different types of PCIe interfaces (ExpressModule, AMC,
etc.). Multiple PCIe SSD cards can be used without requiring multiple PCIe slots
in the server. The storage can be collocated at distances of up to 300m from the
server and can also be potentially shared by more than one server."