High end solo enterprise PCIe SSDs are under threat of being squeezed in
the gap between DIMM wars memories and SATA SSDs. Cloud compatible roadmaps
and array friendly collaborative software customization are bigger market
opportunites than artificial benchmark performance.
the changing shape of the enterprise PCIe SSD marketby
editor - StorageSearch.com
- July 12, 2016
|If you knew the
enterprise PCIe SSD market really well 4 years ago (in
2012) and if
your attention had been distracted elsewhere in the intervening years -
you'd hardly recognize it today as the same market you once knew. |
not just the names of the key suppliers which have changed but the PCIe SSD
market - which itself was a disruptive influence on the server and storage
markets from the earliest product shipments in 2007 - has not been immune
itself from further disruptive forces from within the SSD market.
think it's not too strong to say that the enterprise PCIe SSD market (as we once
knew it) has exploded and fragmented into many different directions. (And some
parts of the PCIe SSD visionary outlook have shrunk too.)
Here are the
main factors which have contributed to these extra layers of complexity.
- commoditization... Although the enterprise PCIe SSD market started life as
a high value, proprietary technology market it was inevitable that once the PCIe
flash market got big enough (and especially as PCIe SSDs started to be designed
for consumer SSD
roles) many of those technology and cost factors would come back to taunt the
parent enterprise market with cost comparisons which would undermine the
original foundations of many pioneer PCIe SSD business plans. The predictable
incumbent assassin tools included:- simpler (single chip
frameworks (like NVMe) and smaller form factors (like
M.2 - which we see being
used singly and also in arrays on PCIe carrier boards).
smaller, cheaper solutions became available and proven in the market - they
changed the cost structure and roadmap assumptions of the enterprise PCIe SSD
market and also created new application roles which included physically
smaller spaces into which the first generation products couldn't go. The
traditional way for such older product designs to go would up market - but
upmarket was becoming a more competitive space too for similar and other
- new ways of server acceleration... As the enterprise SSD market became
more confident in using SSDs - it became clear that the value of SSD
acceleration - even within a single type of nominated CPU server - could be
far more nuanced and
finely grained and adaptable to different application roles (from a small
SDS cluster upto a cloud
scale data center) by using standard SSDs with different interfaces, latency
and cost characteristics.
Traditional high performance (big form
factor) PCIe SSDs still had useful work they could do in this wider framework
of acceleration - but they were no longer the only solution, or the easiest -
but instead one of many possible solutions - in which electrical power
consumption, capacity, scalability and affordability of the server were
increasingly being seen as more desirable attributes than the peak performance
of any single SSD.
At the high end of the performance range for PCIe
SSDs we're now seeing the market space being squeezed by
SCM DIMM wars
In the current market therefore the traditional big
bang ultrafast PCIe SSD doesn't seem as impressive as it used to be and instead
is part of a continuum of possible performance options in the market rather
than the best game in town.
The notes above show some of
the reasons why the traditional PCIe slot form factor enterprise SSD market
has not followed a single and predictable pattern of market growth.
- New ways of storage box acceleration... When it comes to providing easy to
integrate low latency flash based cache storage in arrays of storage drives -
there have been 3 factors which have between them reduced the market demand for
traditional PCIe SSDs in the kind of roles we saw in hybrids in 2010, 2011 etc.
enterprise has entered or passed the early majority adoption phase of SSD
storage. This means when all the bulk storage drives in a big box are already
cheap SSDs (SATA or SAS) rather than HDDs - the optimum technology for local
caching in such boxes is more likely to look like a small amount of memory (whatever memory may be) rather
than a big fat PCIe SSD front ending the array.
For general purpose
applications some big SSD users have reported that using arrays of SATA SSDs
with their servers gives them similar application performance (and lower cost)
to having PCIe SSD rich servers (SSDserver rank
2) operating in
tandem with earlier generation HDD arrays. The business consequence has been
that follow up deployments of infrastructure across a wide range of cloud
scale uses have tended to use lower quantities of traditional PCIe SSDs than
would have been forecast from the earlier market adoption models.
significant growth is beeing seen in smaller form factors within the PCIe SSD
In new high performance storage arrays the drives in the
array are likely to be 2.5"
PCIe SSDs. This removes the need or benefit of having big form factor PCIe
SSDs in such storage boxes.
In value based hybrid arrays - if there is
still a need for some kind of low latency flash cache (in the traditional PCIe
SSD role) the need is likely to satisfied by
M.2 SSDs rather than a
larger form factor product.
In SDS applications - everything which has
been said above for servers applies too.
complicating factors have been the growing sophistication of the
SSD aware software market
and its ability to encompass many different SSD types (by interface and form
factor) and use the varied SSD latency and R/W characteristics within pools of
abstracted virtual storage. Again this kind of software feature was initially
pioneered in PCIe SSDs and associated with such products. The decoupling of
server application agility from SSD form factor and specific product lines has
increased competitive choice for SSD integrators to the extent that they can now
get PCIe-SSD -like capabilities without having to have PCIe SSDs physically
PCIe SSDs will still be very useful components in the
enterprise market. And the advantages of latency and ease of integration have
migrated into other form factors as already noted above (2.5", M.2 and also
BTW SSDs on a chip too).
The SSD, storage and server market has learned
many valuable lessons from the early phases of the PCIe SSD market (2007 to
2012) about data architecture, how to use SSDs as storage and memory and how to
improve infrastructure utilization. And the system shaping techniques so
learned now form part of the conventional wisdom of the industry. But continuing
adaptation of those new tricks is not tied or bound to the original past
physical form factors or the companies which first experimented with them.
Instead the PCIe SSD market and its associated ideas have exploded into many
different shapes and sizes and segments.
It's still an interesting
market to keep your eyes on - but is no longer at the vortex of architectural
disruption. Instead you could say it provides a sanity check and places cost
zone limits on the future direction of the DIMM wars market (with which it
overlaps in capabilities at the lower latency end of the PCIe SSD product
spectrum) and the PCIe SSD market is itself prevented from getting too greedy by
competing with the wide range of alternative technologies which (from the user
point of view) provide similar and competing system capabitites.
|how fast can your
SSD run backwards?|
|SSDs are complex devices and there's a
lot of mysterious behavior which isn't fully revealed by benchmarks, datasheets
and whitepapers. Underlying all the important aspects of SSD behavior are
which arise from the intrinsic technologies and architecture inside the SSD.
|You don't have to
understand the internal details of how these individual techniques work. And
with hundreds of patents already pending in this topic there's a high
probability that the SSD vendor won't give you the details anyway (not even
under NDA). It's enough to get the general idea.|
care management & DSP ECC IP in SSDs|
|the Problem with
Write IOPS in flash SSDs |
|Random "write IOPS"
in many of the fastest flash SSDs are now similar to "read IOPS"
- implying a performance symmetry which was once believed to be impossible.|
why are flash SSD IOPS such a poor predictor of application performance?
why are users still buying
RAM SSDs which cost an
order of magnitude more than SLC? (let alone
MLC) - even
when the IOPS specs look superficially similar?
tells you why the specs got faster - but the applications didn't.
||And why competing SSDs with
apparently identical benchmark results can perform completely differently.
|many years ago -
|BiTMICRO & CENATEK collaborate on PCI SSDs|
2002 - BiTMICRO.
announced today, a technology and marketing partnership to investigate
developing a hybrid solid state disk storage solution that brings together the
best in flashdisk storage and PCI bus-attached SSD technology.
Prices are likely to
range between $1,000 to $2,000 per gigabyte.
Editor's later comments:- the fast PCI SSDs which later
emerged from CENATEK - can be seen as immediate ancestors of the modern PCIe
But the concept of using fast bus based SSDs as storage
accelerators wasn't a new idea. It went back 3 to 4 decades.
earlier market experiments with solid state storage were always short lived -
because the expensive SSD storage in each product generation was always
competing with fast changing improvements in CPU clock speeds, bus memory
throughput or faster external magnetic storage media.
The changes in
the modern era of
SSDs - which started about 2003 - was that due to those other computer
technologies stagnating and not getting any faster - the only competitor which
killed an SSD from about that time - was another SSD.
By that time -
the new computer bus was known to be PCI express. But it was the failure of the
alternatives to solid state storage to get faster - which made the big
difference to the business viability of SSDs - rather than any inate
characteristic of PCIe SSDs.
the fastest SSDs
2.5" NVMe PCIe SSDs
is remanence in
NVDIMMs a new security risk?
the big SSD ideas which emerged in 2016?
where are we
heading with memory intensive systems and software?
|Foremay ships aerospace
capable 8TB 2.5" U.2 NVMe SSD|
Editor:- September 26, 2016 -
volume production of 8TB models in its rugged secure 2.5" U.2 NVMe SSD
product range - which with PCIe x4 lanes has R/W speeds up to 1.2GB/s with
latency as little as 25 microseconds. Optional features of the SC199 hi rel
- Military secure erase and fast erase features.
- Rugged designs with anti-shock and anti-vibration, meeting MIL-STD-810G/F
also:- military SSDs
- Anti-radiation and anti-emission, both electrical and magnetic, for
aerospace applications subject to the customer's specifications.
Liqid controller inside fastest 2.5" NVMe flash SSD
August 9, 2016 - In a joint
release today Liqid
gave details of the "fastest 2.5" PCIe NVMe flash SSD ever
benchmarked" - which they're showing this week at FMS.
2.5" PCIe SSDs,
- upto 3.9 TB of capacity
- Mixed R/W: ~ 5.5 GB/s (Full Duplex)
- Seq read ~ 3.6 GB/s Seq write ~ 3.6 GB/s
- mixed random R/W ~ 1.15 M
IOPS (4 K,
Radian will sample PCIe based competitor to NVDIMMs in October
August 2, 2016 -
it will be sampling in October a new byte addessable,
(NVRAM-flash cache ratio), 12TB
PCIe SSD which has
on-board host controlled tiering between its flash and NVRAM.
comments:- Radian is positioning the new product as a cost effective alternative
competitor to hybrid
NVDIMMs and similar emerging products in the
SCM DIMM wars
In an interview last year with Radian's CEO Mike Jadon -
what's the role for a
Radian Memory SSD? - I learned more about the company's software thinking.
IP-Maker releases Gen 3 NVMe PCIe reference design
July 11, 2016 - for designers of
PCIe SSDs - IP-Maker has released
its new Gen 3 NVMe PCIe reference design which is based on the VC709
evaluation kit by Xilinx. It's integrated
PCIe Gen3 hard IP and a soft DDR3 controller. The
NVMe compliant design uses
a x4 lanes configuration.
SSD glue chips
Mangstor has P2P in NVMe-based PCIe SSDs
June 22, 2016 - Mangstor
announced that its MX-Series family of NVMe-based PCIe SSDs now features
higher capacity 5.4TB usable models (FHHL) as well as Peer-to-Peer (P2P)
capabilities which provide low latency access to an extremely large storage pool
of non-volatile memory, bypassing host resources. The new models have been shown
this week at the ISC High Performance
Conference in Germany.
""Our innovative ability to
separate data control from the data path optimizes I/O, reduces latency spikes
and system bottlenecks, and delivers accelerated performance" said Paul Prince, CTO
QLogic to be acquired
Editor:- June 15, 2016 - QLogic
it has agreed to be acquired by Cavium
in a transaction worth approximately $1.36 billion.
Micron samples 2TB 3D 2.5" PCIe NVMe SSDs for desktops
May 31, 2016 - Micron
it is sampling new
2.5" PCIe NVMe
SSDs based on 3D nand for the
2100 with 2TB capacity is expected to be in [roduction in the summer.
enterprise PCIe SSD shipments grew 16% Q-Q
May 18, 2016 - TrendFocus
publication of its
2016 nand/SSD quarterly market report. TrendFocus says the enterprise SSD
market saw Q-Q growth in all segments.
storage market research
2 ASIC roles for PCIe based BiTMICRO SSD controllers
March 25 , 2016 -
5 years ago when
unveiled an earlier generation of its high performance enterprise SSD
controller architecture - it was clear that their preference was for a chipset
which included 2 different types of functionality.
This kind of
thinking wasn't unique at that time - as I'd seen similar things in
designs before but (unlike BiTMICRO) those other designs were captive and not
offered as COTS SSD
How many controller chips do you really need for a
a new blog
today BiTMICRO explains why its current generation of controllers continues
using a 2 ASIC architecture with one acting as a flash array extender and the
other as the main PCIe host interface controller.
Among other things
the blog says "To increase flash channel bandwidth and capacity, more flash
channel expander chips can be instantiated and connected to the main controller."
As noted in the SSD
design heresies - SSD vendors often have different implementation
architecture approaches which compete in similar application slots. When
evaluating different types of offerings it can be useful to ask yourself - which
direction is my own design likely to stretch in future? (Towards more
performance? lower cost? bigger scale? adjacent application role? etc.)
BiTMICRO's blog clarifies where they see their strengths in the market. ...read the
PCIe 4.0 chips milestone
Editor:- March 14, 2016 -
Signs of onwards and upwards progress towards future
PCIe SSD speeds
emerged today in an
have demonstrated electrical interoperability for PCIe 4.0 with "robust
signal integrity" (BER below 10-15) at 16Gbps with 4 lanes running
Seagate promises faster PCIe SSDs
8, 2016 - Seagate
it will ship a 16 lane NVMe PCIe SSD with 10GB/S throughput in the summer. No
further details are available at this time.
in 2015 the fastest production PCIe SSDs were probably the
NVRAM from Microsemi
and the MX6300
2016 the fastest
motherboard SSDs will be
SSDs rather than PCIe SSDs - which potentially will be able to deliver
twice the performance of 16 lane PCIe SSDs - but which are more
capacity and fault
It's not worth paying more for SLC reliability in PCIe SSDs says
Google field study
editor:- February 26, 2016, 2016 - A 6 year
PCIe SSDs used by
Google (spanning millions of drive days and chips from 4 different flash
vendors) concluded that SLC drives were not more reliable than MLC.
An important conclusion re RAS is the importance of being able to map
out bad chips within the SSD architecture. This is because somewhere between
2% to 7% of enterprise PCIe SSDs (depending on where they were used) developed
at least bad chip during the first 4 years in the field - which without such
remapping would necessitate replacing the failed SSD.
The source is -
Reliability in Production: the Expected and the Unexpected (pdf) - by Bianca
Schroeder University of Toronto, Raghav Lagisetty and
Arif Merchant, Google.
This is just one of a set of
which was presented February 22 - 25 , 2016 at the
14th USENIX Conference on
File and Storage Technologies.
Editor's comments:- For more
like this scroll down this page to see the June 2015 story about a large
scale study of PCIe SSD failures within Facebook.
TrendFocus says all enterprise SSD interface types enjoyed
double digit market growth in Q4 2015
Editor:- February 22, 2016
- Unit shipments in the enterprise
PCIe SSD market
increased 20% in Q4 2015 compared to Q3 according to a
report today from TrendFocus.
Shannon indicates sales of enterprise PCIe SSDs doubled
February 12, 2016 - Apparently Shannon Systems is
seeing high growth in the
PCIe SSD market and
has been saying that its sales in this category nearly doubled in the recent
quarter. This was learned in a recent
of its Direct-IO range by Tom's IT Pro.
Microsemi completes acquisition of PMC-Sierra
January 15, 2016 - Microsemi
it has completed the acquisition of PMC-Sierra for $2.5
new TB PCIe SSD on M.2 from OCZ
Editor:- January 5,
2016 - OCZ says it
will show new NVMe PCIe SSDs with 1TB capacity and up to 2.4GB/s of bandwidth on
M.2 module at
CES this week.
PCIe SSDs still dominate reader thoughts
January 4, 206 - PCIe SSDs remained at the forefront of interface types and
form factors researched by the readers of
December - as you can see in the
Top 50 SSD
articles seen by readers.
That's the 6th year in a row that PCIe
SSDs have held this lead in reader fascination.
As a matter of interest
the topic of memory
channel SSDs has crashed in relative reader interest. This may be partly
due to this application segment having recently become very much less clear than
it appeared to be (upto about 5 months ago) as the idea sinks in that many
new vendors have announced their intentions to offer products for the DIMM
slot accelerator space and they're offering more than 3 distinctly different
combinations of memory technologies to get there.
The illusion of a
single dominant supplier was a simplifying assumption which boosted the
popularity of this idea. Fragmentation within the DIMM big memory / storage
market is the new reality. ...see the popular
Memblaze gets more funding
Editor:- December 21,
2015 - Memblaze
it had received "tens of millions of US dollars" in Series C
funding. Among other things the company said it will now expand its sales
efforts in international markets.
PMC-Sierra agrees to be acquired for $2 billion
October 6, 2015 - PMC-Sierra
has agreed to be acquired by Skyworks
for $2 billion it was
"With our acquisition of PMC, Skyworks will be uniquely
positioned to capitalize on the explosive demand for high performance solutions
that seamlessly connect, transport and store Big Data," said David J. Aldrich,
chairman and CEO of Skyworks. "Specifically, we plan to leverage PMC's
innovative storage systems, flash controllers, optical switches and network
infrastructure solutions to expand our engagements with some of the world's
leading OEMs and ODMs as well as
data center customers."
Mangstor gets $10 million series B funding
for fastest NVMe
Editor:- September 17, 2015 - Mangstor today
it has closed $10 million in Series B funding which will be used to fuel
growth in sales and engineering and business development.
investment confirms our leadership in NVMe over fabrics technology which
delivers an order of magnitude higher performance at the lowest latency verses
legacy iSCSI and Fiber Channel" said Trevor Smith
CEO and co-founder of Mangstor .
comments:- last week I had a 90 minutes one on one with Trevor which we
spent talking about gaps in the enterprise SSD market, Mangstor's technology
and the competitive positioning of its PCIe SSDs and systems. You can see some
of our discussion topics in their profile page.
M.2 NVMe PCIe SSD good
4 is better says Memoright
September 3, 2015 - Memoright
discussed the possibilities arising from using an array of 4x
NVMe PCIe SSDs packed onto a single PCIe switch card as a way of building
competitively priced SSD accelerated servers - in a paper
a PCIe-Based Switch Module to Enhance Enterprise Storage Architecture (pdf)
presented at the recent Flash
Memory Summit. Each of the new M.2 SSDs from Memoright would have upto 1TB
capacity and R/W IOPS upto 81K / 54K respectively.
comments:- And another recent example of something similar - may be
Kingston's E1000 -
hardware - where it was described as a PCIe SSD which integrates an "array
of 4 NVMe SSDs". (Possibly 4x Kingston's
would be my guess.)
This is part of a new trend in enterprise SSDs.
years ago companies which at that time didn't have their own
architecture (OCZ etc)
were able to compete in raw throughput performance with those who did (Fusion-io,
Virident etc) by
designing PCIe SSDs which were populated by an array of SATA SSDs - routed
together by an onboard RAID controller.
The modern way of doing
something similar is to have an array of small form factor NVMe SSDs on a
The (wishful thinking) marketing argument for this is
that the small NVMe SSDs have their costs and design roadmaps attached to a
much faster changing and more competitive groove in the SSD market than the big
proprietray designs - which reduces the risks of customers being tied to an
expensive single source design.
Although these different design
approaches can deliver SSDs which overlap in standalone
capability - there are usually
factors involved -
customer risk perceptions and preferences - which determine the SSD chosen.
Seagate enters 2.5" NVMe SSD market
August 11, 2015 - Seagate
details of 2 new families of
NVMe SSDs which
will be available in 2.5"
(October) and M.2 (in early
2016) form factors. Also new - the Seagate Nytro XP6500 - a
PCIe SSD accelerator
(which is currently available) delivers the lowest write latency within
Editor's comments:- until this announcement
it wasn't clear how Seagate would deal with an issue which has been
problematic for other competitors too - that of introducing SSDs in form factors
and interface types which can (at the fringes) compete with pre-existing product
But because there is clear customer demand for both
SAS SSDs and
PCIe SSDs in 2.5"
form factors for example - then it would be a mistake for any vendor with large
scale market ambitions to opt out of supplying such products despite the
potential risk of some cannibalization.
Nevertheless - the order in
which we see enterprise SSDs being introduced in various form factor and
interface combinations (different for each supplier) tells us what they consider
are their native strongholds.
BiTMICRO has new faster, denser PCIe SSD
August 11, 2015 - BiTMICRO
announced the release of the company's MAXio Z-Series enterprise
PCIe SSDs (gen 3 8
lanes) which are controlled by its patented Talino ASIC architecture. The new
Z-Series provides up to 8.8TB in a PCIe edge card form factor and also includes
military-class erase technology. For 4KB blocks the R/W IOPS and latencies are
respectively - 430K 50µS and 150K 30µS respectively.
comments:- BiTMICRO which 10 years ago was a pioneer in the high performance
enterprise flash SSD market got kind of swept aside from mainstream market
view for many years as the market expanded and the rate of memory innovation
seen in such SSDs changed too fast for most companies to keep up.
years the company retrenched back into its safe roots in industrial military
applications - and like many other SSD companies - it has had business setbacks.
But it's also had some customer successes too with its newer SSD technologies
in the China market in partnership with
Toshiba's new NVMe PCIe SSDs will be available in 4 form
Editor:- August 10, 2015 - Toshiba today
details of 3 new NVMe PCIe SSD families which will sample in the next quarter.
of these are aimed at the
and come in variants of the M.2
form factor - with upto 4 lanes of PCIe 3.
More interesting, however,
are the new enterprise products - model PX04P - in 2.5" or HHHL form
Virtium launches industrial M.2 PCIe SSDs
July 28, 2015 - Virtium
it has expanded its StorFly range of industrial SSDs with new
PCIe SSDs (gen 2)
available in both M.2 and
Mini Card form factors with capacities from 16GB upto 480GB.
is upto 3.3 petabytes of writes (about 3.7
DWPD for 5 years).
Virtium's new SSDs have full BOM control with up to 5 years of uninterrupted
bath tub curve is not the most useful way of thinking about
PCIe SSD failures - according to a large scale study within Facebook
June 15, 2015 - A recently published research study -
Study of Flash Memory Failures in the Field (pdf) - which analyzed
failure rates of PCIe
SSDs used in Facebook's infrastructure over a 4 year period - yields some
very useful insights into the user experience of large populations of
Among the many findings:-
- Read disturbance errors - seem to very well managed in the enterprise SSDs
The authors said they "did not observe a statistically
significant difference in the failure rate between SSDs that have read the
most amount of data versus those that have read the least amount of data."
- Higher operational temperatures mostly led to increased failure rates,
but the effect was more pronounced for SSDs which didn't use aggressive data
throttling techniques - which could prevent runaway temperatures due to
throttling back their write performance.
- More data written by the hosts to the SSDs over time - mostly resulted in
more failures - but the authors noted that in some of the platforms studied -
more data written resulted in lower failure rates.
attributed to the fact some SSD software implementations work better at
reducing write amplification when they are exposed to more workload patterns.
- Unlike the classic bathtub curve failure model which applies to hard drives
- SSDs can be characterized as having early an warning phase - which comes
before an early failure weed out phase of the worst drives in the population
and which precedes the onset of predicted endurance based wearout.
this aspect - a small percentage of rogue SSDs account for a disproportionately
high percentage of the total data errors in the population.
report contains plenty of raw data and graphs which can be a valuable resource
for SSD designers and software writers to help them understand how they can
tailor their efforts towards achieving more reliable operation. ...read
the article (pdf)
Memblaze is #1 enterprise PCIe SSD supplier in China market
May 22, 2015 - Memblaze
accounted for 60% market share of the
enterprise PCIe SSD
market segment in China in 2014 - according to CEO Yin Xuebing
who made this comment in a recent
release announcing the availability of Memblaze's new PBlaze4 700 Series
NVMe PCIe SSDs which was demonstrated at CeBIT
in March 2015.
OCZ's power envelope programmable 2.5" hot swap NVMe
May 20, 2015 - OCZ
today revealed more details about the new models shipping in its NVMe
compatible PCIe SSD family - which was first announced last
had already heard before these new models include 2.5" hot swappable
Today OCZ said this model - the
Z-Drive 6300 SFF
will be available with usable capacities of 800GB, 1.6TB and 3.2TB (in
this quarter) followed by 6.4TB (later this year).
performance is upto 2.9GB/s and 1.4GB/s respectively. Random R/W
700K IOPS and 120K IOPS. Latencies are 30µs (write) and 80µs (read).
options are 1 or 3 DWPD.
availability and reliability features
The new Z-Drive 6000 models
are dual ported so that 2 host systems can concurrently access the same SSD.
Additionally, the Z-Drive 6000 Series supports hot swapping of 2.5"
drives, pre-set power thresholds and temperature throttling to support many
types of enterprise ecosystems.
Editor's comments:- for various reasons to do with a
combination of standardization
efforts and changes
of ownership for nearly every major enterprise PCIe SSD company in the
market - you've had to wait 3 years since the idea of this kind of product was
first discussed seriously on
these pages and at
What has become clear to systems architects is that these
new products offer far more flexibility in their roles than merely performance
upgrades to high end SAS
SSDs and traditional storage arrays.
Among other things these new
types of products will enable lower cost mini SSD server clustering at
PCIe latencies which will spur growth in the SDS market. At the high end - they
could become the new building blocks inside the world's most powerful computer
Power consumption and heat in these NVMe SSDs?
know from talking to systems architects that the electrical power and thermal
footprints of 2.5" NVMe SSDs is a critical detail when considering the
design of dense storage arrays so I asked Scott Harlin,
OCZ for more information these factors. Here's what Scott said.
Zsolt you are correct the 2.5" drives can get a little toasty
packing in the higher densities into this form factor -- typical power
consumption of the Z-Drive 6000 series is 25W active and 9W idle. So we
included a few items to address these concerns:
1. - Temperature sensing and thermal throttling to maintain
consistent operating conditions even under adverse temperature variances
2. - User-selectable power envelopes, in 15W, 20W and 25W
settings, reduces wasted power when maximum performance is not required while
efficiently addressing temperature requirements in support of a variety of
3. - An innovative 'flow-through' case design enables more
airflow to critical components, keeping the device cool while reducing airflow
Editor's comments:- that user selectable power envelope- in graduated
steps - seems like a really useful design attribute. So I'll be watching out
for it in future arrayable SSD launches.
which market uses the highest capacity PCIe SSDs?
May 12, 2015 - Unlike the enterprise market where nearly everyone uses
technologies which are recognizably similar to everyone else - we don't hear so
much about advances in embedded
- because a technique which pushes the state of the art for one industrial
customer may be of no interest to nearly everyone else.
hearing more stories in which increasing SSD capacity in confined rugged
spaces is the difference.
Yesterday it was a
customer story from Waitan about designing
and supplying high capacity PCIe SSD modules for use on a drilling platform.
That was an application where paying more for the SSD to fit was cheaper than
making more space for the SSD.
If you've got an interesting story
about custom high capacity PCIe SSDs - outside the usual context of the cloud
and the static datcenter (as opposed to the mobile datacenter) send me an
email with the details.
still the same positioning for new generation of SanDisk inside
Fusion ioMemory PCIe SSDs
Editor:- April 28, 2015 - Back in the
2012 - at
which time the PCIe SSD
market was already well served by many strong (and soon to be acquired)
competitors (as you can see in
archived news page from September 2012) I came up with a short phrase to
summarize the positioning of Fusion-io's products (in ads here) which ran like
"the standard for enterprise PCIe SSDs by which all
others are judged:- ioDrives from Fusion-io".
About a year
later (in about
2013) when I was seeking for a way to refine that description - I updated
that summary to...
"(still) the standard for enterprise PCIe SSDs
by which all others are judged:- ioDrives from Fusion-io."
reason I mention that is to show how phrases which appear in ads can stick in
readers' minds years later and resurface in everyday conversations.
is something which (as I've got no formal qualifications in writing) I had never
really thought about until I heard a lecture on this very point by Professor
in an episode of his audio series "Building
Great Sentences" - a series which I dip into from time to time -
originally triggered by the hope that I might learn some useful tricks I
could apply to my scribbles on this site but a series which as I heard more
episodes sustained my interest more so because it took me to unexpected
panoramas of word writing virtuosity which have probably given me more
pleasure as a spectator than the haphazard impressionistic experiments
in recycling his word craft which I sometimes inflict on you .
can imagine my delight at seeing a recent
release from SanDisk
about a new generation of
Fusion ioMemory PCIe
SSDs which among other things included this statement by John Scaramuzzo,
senior VP and GM, Enterprise Storage Solutions, SanDisk.
technology fundamentally transformed expectations about data center performance
when it debuted 8 years ago and it remains the standard by which all other
PCIe products are judged."
Also in this news story we learn
that over 250,000 of these accelerators have been deployed by over 7,000
customers. And that having been re-engineered to use SanDisk memory - the new
models cost a lot less than the previous generation ioDrive2 product, and
are 2x faster at reads.
Silicon Motion acquires Shannon Systems
April 24, 2015 - Silicon
it will acquire Shannon
Systems for a total purchase price of $57.5 million.
has become the largest internet market in the world and its internet companies
have already widely adopted SSDs for their data centers and hyperscale servers.
More recently, Chinese corporates and government organizations have started
using SSDs for data virtualization. These operators of corporate and internet
infrastructure have increasingly been turning to Shannon for world-class,
locally developed and supported SSD solutions," said Wallace Kou,
President and CEO of Silicon Motion.
CoreRise ships new smaller BladeDrive PCIe SSD
April 20, 2015 - CoreRise
customer shipments of a new version of its BladeDrive family of gen 2 x8
PCIe SSDs - the E24
- which has a smaller form factor than the earlier E28.
based implementation supports upto 1.6TB capacity, 275K IOPS (4KB) and 2GB/s
throughput in half-height half-length. Software support includes Windows
Server, Linux and virtualization such as Xen, Hyper-v, as well as TRIM.
SanDisk comments on customer drift away from its PCIe SSDs
April 16, 2015 - You should be careful not to misconstrue SanDisk's recent
statement about enterprise PCIe SSD replacement by SATA SSDs as a general trend
for all customers and all types of PCIe SSDs. I think some of these customer
migrations are probably more specific to some large scale customers for the
Fusion-io product line rather than the whole industry.
results related conference call (transcript)
this week SanDisk's
President and CEO - Sanjay Mehrotra
- said that in the enterprise market it was seeing some of the business
which had traditionally been implemented by its customers using
PCIe SSDs was moving
towards arrays of SATA
Editor's comments:- another emerging market is
arrays of 2.5"
PCIe SSDs which you could argue will take away market share from
SAS SSDs - as well as
enabling a new market.
Factors at work in the
rackmount flash array
complex due to very different customer needs and different ways of judging
the same tech attributes due to users' own business values related to their core
activities and what they've done before.
Sonnet launches "Fusion PCIe Flash Drive"
by) a useful clarification re the Fusion brand in storage
April 7, 2015 - In a move which I think will lead to inevitable confusion -
has today launched a new consumer storage module called the "Fusion
PCIe Flash Drive."
"The Fusion PCIe Flash Drive leverages the
latest advancements in PCIe SSD design and Thunderbolt 2 technology, enabling
Sonnet to offer a storage device that fits neatly in the palm of your hand yet
delivers the blazing-fast performance of a multi-drive RAID storage system many
times its size," said Robert Farnsworth,
Sonnet Technologies CEO. "We think this will become an indispensable
accessory for the creative professional."
Editor's comments:- Inside the box - is an
M.2 SSD. If it had
anything to do with Fusion-io
- the best known brand in the
PCIe SSD market - then
I think we would have heard about it before.
I often wonder what goes
through the minds of product marketers when they name new SSD products. Many of
the clever words have already gone as you can see in my
SSD brand name
Later corrections and clarifications
contacted Sonnet's CEO - Robert Farnsworth about this name confusion issue.
said - "Robert is there any connection between your new Fusion PCIe drive
and Fusion-io? I would be surprised if there was. Didn't you anticipate
confusion when naming this product?"
Robert told me - "Sonnet
Technologies has been using the Fusion name for our storage product since
before Fusion-io existed as a company. None of Fusion-io products carry the
Fusion name because Sonnet owns the trademark."
I got more of
the background history about the Fusion storage trademark issue from Greg LaPorte,
VP Sales and Marketing - Sonnet Technologies - whose detailed notes make
Greg said - "Sonnet was actually the first
to use, and trademark, 'Fusion' in association with storage back in 2005 and
have since used it for all Sonnet storage products. With respect to Fusion-io,
yes, Sonnet began using the trademark 'Fusion' for storage products before
Canvas Technologies became FusionMultiSystems in June of 2006 and well before
they named their first ioDrive in September 2007. Later they evolved to
Fusion-io for a company name but did not name their actual products with Fusion.
They tend to use "io" in the majority of their product names."
email also included this related story. "When Apple came out with the
Fusion Drive, we brought this to their attention; Was Apple infringing on a
Sonnet trademark? Their answer was that Fusion is so commonly used for product
names, they felt it fell into the "non-trademarkable" category. We...
don't agree with that assessment but we have a good relationship with Apple."
Memblaze enters 2.5" PCIe SSD market
March 16, 2015 - Memblaze
it is showing a 2.5"
NVMe PCIe SSD variant of its PBlaze series at
CeBIT in Germany.
Memblaze was the
27th most highly searched SSD company by readers of StorageSearch.com in
another design win for Seagate's Nytro in China cloud market
March 12, 2015 - QingCloud mentioned
high capacity and low cost among the reasons for selecting Seagate's
(pdf) (PCIe SSD)
as components to build the low latency SSD infrastructure of its cloud
services for the China market - in a
Editor's comments:- who are the new cloud
companies in China?
China's Cloud Innovators - a blog by Charlie Dai, Principal
Analyst - Forrester
the big market impact
of SSD dark matter
8TB 2.5" PCIe SSDs sampling soon from Novachips
March 4, 2015 - Are you interested in a world's first
8TB (15mm height) PCIe Gen2 x4 SSD (pdf) with a single controller developed
was an email I got recently from Sean Oh, in
Germany, who is the sales representative for these products in Europe.
would you say?
I did the same. After reading up the info he attached
too) I asked some questions about availability. Here's what Sean said.
- The working engineering samples have been available since last year.
- The 1st customer sample comes out in the next 30 days.
- We plan to start a mass production in 2nd half of this year.
comments:- part of this story has its roots in a news story from
May 2007 - when a
company called MOSAID was talking about a new, light capacitive load, ring
based, flash memory topology called HLNAND. To make it work they needed a
controller. Novachips collaborated on the design and recently acquired the
assets and patents.
- What do you think ? Is it something cool, isn't it ?
Greenliant enters PCIe SSD market
12, 2015 - Greenliant
Systems - has entered the entry level enterprise
NVME PCIe SSD market -
of its new G7100 (pdf)
series MLC gen 2 x4 PCIe SSDs - upto 2.75 TB raw capacity, Full Height, Half
Length form factor, 130K / 60K
of upto 10 DWPD for 5
Internally Greenliant's new PCIe SSD has a
RAID protected array of
miniature NANDrive SSDs which use the company's own controllers.
our in-depth flash memory knowledge and volume-proven NAND controller expertise,
Greenliant is addressing the industry's increasing need for higher reliability,
higher performance and larger capacity flash-based storage solutions," said
CEO of Greenliant Systems.
Editor's comments:- ever since the
company was founded - in 2010 - Greenliant's focus has been on the
embedded SSD market. So it's surprising to see this new product aimed at
the "enterprise" market. However it's part of an
trend in the market.
Samsung mass producing gen 3 PCIe SSDs for notebooks
January 7, 2015 - 19 months after launching its first M.2 PCIe gen 2 SSDs
aimed at the
(pdf) in June
2013) - Samsung
said it is now mass producing the follow-up SM951 - which supports gen 3
Samsung says - "For ultra-slim notebooks and workstations
the SM951 can read and write sequentially at 2,150MB/s and 1,550 MB/s
The SM951 is the first SSD to adopt L1.2 low
power standby mode (the PCIe
SSD equivalent of the power saving
mode in SATA SSDs) .
When hibernating in
mode, the SM951's power consumption is less than 2mW.
Toshiba shows early version of BGA PCIe SSD
January 7, 2015 - Toshiba
it will showcase a prototype of the world's first PCIe single package SSD -
with up to 256GB in a single BGA package at
CES this week.
compatible device fits into 16mm x 20mm x 1.65mm and weighs under 1g. See
also:- BGA SSDs
What next - after PCIe SSDs everywhere?
December 2, 2014 - What next - when PCIe SSDs are already everywhere? You
know you need them. But you need more too.
Signs of interesting times ahead in 2015 are revealed in the new
edition of the
Top SSD Companies -
based on market metrics in
Q3 2014 -
researched and published by StorageSearch.com
over 5 years - Fusion-io
had occupied the top #1 spot in this list as the SSD company which
was most researched by our readers. That was inevitably going to change. And it
did. But not in the way you might have expected.
The new #1 SSD
company is Diablo
Technologies - creator of the Memory Channel Storage platform.
that's not the only sign of change.
7 years after the
2007 Year of SSD
revolutions - is the enterprise SSD market about to begin a new revolution
in server centric SSD architecture? ...read the article
the key band members - which made Fusion-io so very famous -
have reunited at Primary Data
Editor:- November 19, 2014 - "With
Woz on the
team along with Lance
we now have the band back together, and I'm amped to be reunited at Primary Data"
said Rick White,
Data today - referring to the fact that 4 of the best known people in
(including the 2 founders) were now at the helm of the new company on the day it
emerged from stealth mode. ...read
more in SSD news
why did Memblaze demonstrate 14µS latency PCIe SSDs in
Europe before US?
Editor:- October 31, 2014 - Something I saw in
a press release this week by Memblaze Technology
made me think I'd better pause to check the significance behind the words
before mentioning it here.
The story - from their
PR agency -
A3 Communications - was about
the launch of a new PCIe
SSD - the EBlaze3 - in Europe - which contained the phrase.
will be the first time the technology is available outside China."
wondered if - behind that statement - there might have been an implied message
that the new product will be sold preferentially in Europe - rather than the
US - as we have seen evolving in stories about one of the world's biggest
enterprise companies - to have emerged from China -
this wording - and the showing of Memblaze's EBlaze3 at a trade show this
week in Germany - simply due to the timing of having booked booths at trade
shows and the availability of the new product - rather than an expression of
preference about which market - the US or EU - is more important for their new
enterprise SSD to be seen in?
So I asked Ron MacLeod - who is VP
of marketing for Memblaze in Europe - to clarify these points.
- "The product will be available in the US in the same way it is available
in Europe. And with regard to timing indeed it was just coincidence that the
Powering the Cloud event was running and at which was the first time Memblaze
was public in Europe. There is no favouritism over markets except the US is
potentially larger - so no hidden messaging. We value the European business and
other potential global partners equally."
Editor's comments:- So that's why I delayed mentioning this before.
- The Memblaze EBlaze3 (upto 2.4TB in HHHL and 4.8TB in FHHL single slot) is
a new version of the
card - which is aimed at, Hyperscale and OCP (Open Compute Project
market). It offers an ultra-low latency of 14µs and enables
customers to define and alter key settings to ensure optimum performance in
their specific applications and platforms.
StorageReview.com tests Fusion-io's Atomic SSD
October 21, 2014 - Earlier this year - in
June 2014 - and
just weeks before SanDisk
announced it was acquiring the company - Fusion-io launched a
new generation of PCIe
SSDs - the Atomic Series - which was an overdue catching up exercise to
use cheaper 20nm flash.
At the time the headline product specifications seemed to me to be
adequate - rather than spectacular.
It's probably safe to assume they
were intended to restore the competitiveness of Fusion-io's products rather
than push performance boundaries.
But what are they like? And how
do they compare?
A new article -
ioMemory PX600 Review - published in StorageReview.com provides the
answers to how it performs and how it compares on popular OS platforms and
benchmarks. ...read the
how fast can your
SSD run backwards?, SSD
testing & analyzer news
Samsung mass produces 3TB 3D PCIe SSDs
September 25, 2014 - Samsung
today announced it has started mass producing 3.2 TB NVMe
PCIe SSDs (HHHL)
based on its 3D flash memory
technology, for use in enterprise systems.
new NVMe PCIe SSD, SM1715 provides a sequential R/W speeds upto 3GB/s and
2.2GB/s respectively with
rated at 10 DWPD for 5
Editor's comments:- Samsung supplied an
for the new SSD - but forgot to include the block size to which it relates - so
I've asked for clarification.
Seagate launches new improved Nytro PCIe SSDs
September 10, 2014 - Seagate
2 new PCIe SSDs -
which are based on the SSD product lines and brand assets of the recently
acquired SSD business of LSI.
XP6302 is a HHHL, gen 3 PCIe SSD - which provides up to 1.75 TB of
usable eMLC capacity with 200 microseconds average latency, and 295K/79K
(8KB) and rated for 0.9 DWPD
endurance for 5 years.
XP6210 is a FHHL gen 2 PCIe SSD with 1.86TB usable 19nm cMLC
capacity, with 50 microseconds average latency 185K/120K R/W IOPS (8KB),
and rated at 1.6 DWPD
endurance for 5 years.
HGST announces 2nd generation clustering software for FlashMAX
Editor:- September 9, 2014 - HGST today
a new improved version of the
clustering capability previously available in the
PCIe SSD product line
acquired last year from Virident.
allows clustering of up to 128 servers and 16 PCIe storage devices to deliver
one or more shared volumes of high performance flash storage with a total usable
capacity of more than 38TB.
HGST says its Virident HA provides a "high-throughput,
low-latency synchronous replication across servers for data residing on FlashMAX
PCIe devices. If the primary server fails, the secondary server can
automatically start a standby copy of your application using the secondary
replica of the data."
For more details see -
Virident Software 2.0 (pdf)
Editor's comments:- This
capability had already been demonstrated last year - and
ESG reported on the
technology in January
But at that time - the clustering product called vShare -
was restricted to a small number of servers - and the data access fabric was
restricted to Infiniband
With the rev 2.0 software - the number of connected devices has
increased - and users also have the lower cost option of using
Ethernet as an alternative
StorageSearch.com updates 10 key SSD ideas in 2014
September 5, 2014 - Are SSD years getting faster? It's not just the SSDs.
Anyway it seemed to me that an end of year SSD market update was due
even though 1/3 of this year hasn't happened yet.
That's why StorageSearch.com
this week published a new home page blog -
10 key SSD ideas in 2014.
As you might expect - PCIe SSDs come into the picture. But there's
more to it than that. If you're impatient - you don't have you don't have to
wait till January 2015. ...read
Plextor's M.2 PCIe SSD wins award at FMS
August 7, 2014 - Plextor
- an M.2 PCIe SSD - has
won Best of Show for most Innovative flash memory technology at the Flash Memory Summit.
M6e SSD combines a multi-core
Marvell PCIe 9183
controller and Toshiba
toggle NAND flash with firmware developed by Plextor's in-house team.
say hello to Shannon Systems
Editor:- August 6, 2014
- I hadn't heard of Shannon
Systems before. But I got a nice email this morning from Xueshi Yang,
CEO and co-founder who said he has been reading StorageSearch.com "for
quite a number of years now" and also said that his company is showing
their products at Flash Memory
Among other things - Xueshi Yang said - "Shannon
System is a startup I co-founded in 2011 in China after I left
company is dedicated to the enterprise flash storage market. Currently, we focus
on the high performance PCIe market with our proprietary controllers and
software systems. In April this year, we
the industry first 6.4TB PCIe SSD with a single controller, which boasts 67us
read access latency and 9 us write access latency (all in 4KB, random). While in
June, we introduced a PCIe SSD with SFF-8639 interface, which is hot-pluggable.
We currently serve over 100 customers in China, including Tier 1 internet
companies, as well as other named customers such as China Mobile, China Telecomm
eASIC supports Mobiveil's NVMe platform
August 6, 2014 - eASIC
announced support for Mobiveil's
NVMe platform (pdf)
implemented in eASIC devices.
"eASIC is enabling the rapid
deployment of SSD technology at substantially lower cost and up to 70% lower
power than alternative solutions", said Jasbinder Bhoot,
VP of Worldwide Marketing at eASIC. "By working with Mobiveil, customers
will have access to a complete NVMe solution running in cost, power and
performance optimized eASIC devices."
HGST rekindles concept of a PCM based PCIe SSD
August 4, 2014 - HGST
it will demonstrate a PCM PCIe SSD concept at the Flash Memory Summit. HGST says
the demonstration model delivers 3 million random read IOPS (512 Bytes) and
a random read access latency of 1.5 microseconds.
funded the world's first enterprise PCM PCIe SSD demo 3 years ago (in
June 2011). The
storage density of PCM resulted in an SSD which had pitifully low capacity
compared to flash memory at that time - and earlier this year (in January 2014)
that Micron had temporarily abandoned this idea.
Is HGST really
going to wander into memory space where even the memory makers don't want to go?
Or is this just a market signal that HGST isn't just looking at short term SSD
Lite-On enters enterprise M.2 PCIe SSD market
July 30, 2014 - Lite-On
will be showing a new M.2 PCIe
SSD for the enterprise market next week at the Flash Memory Summit.
The P1P is a
PCIe SSD with capacities of 1TB and
protection circuit all in a small form factor.
The small form
factor and high capacity allow enterprise customers to pack more storage in a
smaller footprint. The P1P Series can deliver sequential R/W speeds up to
610MB/s and 520MB/s while random R/W speeds can be upto 95K/15K
drive has a MTBF
of 2 million hours and an
rating of up to 1 drive write
per day for 5 years.
|archived versions of this PCIe SSDs news page
|PCIe SSD milestones from
|August 2007 -
Violin Memory launched the world's fastest 2U SSD - with a PCIe interface.|
September 2007 -
Fusion-io launched the ioDrive - a PCIe form factor flash SSD with upto 640GB
capacity and 100K IOPS performance.
April 2008 -
StorageSearch.com published a new market directory of PCIe SSDs.
April 2009 -
StorageSearch.com announced that Fusion-io was the
Top SSD Company
researched by readers in
September 2009 -
StorageSearch.com disclosed that search volumes for PCIe form factor SSDs by
its readers had surpassed that for 2.5" SSDs for the 1st time ever.
2011 - Virident Systems demonstrated 1 million IOPS performance in a 1U
server rack using just 2 of its tachIOn PCIe SSDs at a system list price of less
than $.05 per IOPS.
2012 - Micron unveiled a future hot swappable 2.5" PCIe SSD for
use in Dell servers.
February 2013 -
Virident Systems announced beta availability of a software suite which enabled
low latency shared server-side storage and high availability when used with the
company's range of PCIe SSDs.
June 2011 -
NVSL demonstrated the world's first PCIe PCM (phase-change memory) SSD.
2013 - LSI said it was #2 in PCIe SSD shipments in US - having shipped
over 40,000 units in the previous 12 months.
June 2013 -
Samsung has entered the PCIe SSD market with an
M.2 form factor model with
a sequential read performance of 1,400MB/s, and capacity up to 512GB.
2014 - A3CUBE unveiled a PCIe memory fabric for 10,000 node-class
architectures at the heart of which was a shared reflective RAM SSD engine.
2014 - Seagate agreed to acquire LSI's PCIe SSD and SSD controller
business for $450 million.
June 2014 -
SanDisk agreed to acquire Fusion-io for $1.1 billion.
January 2015 -
Toshiba demonstrated a prototype of the world's first PCIe single package SSD -
with up to 256GB in a single BGA package.
February 2016 -
A webscale reliability report by Google concluded it wasn't worth paying
more for SLC rather than MLC in enterprise PCIe SSDs.
|the 3 fastest flash
PCIe SSDs - list / lists|
|Are you tied up in
knots trying to shortlist flash SSD accelerators ranked according to
published comparative benchmarks?|
You know the sort of thing I mean -
where a magazine compares 10 SSDs or a blogger compares 2 SSDs against each
It would be nice to have a shortlist so that you don't have to
waste too much of your own valuable time testing unsuitable candidates
StorageSearch's long running
fastest SSDs directory
typically indicates 1 main product in each form factor category but those
examples may not be compatible with your own ecosystem.
my article -
the 3 fastest PCIe
SSDs list (or is it really lists?) may help you cut that Gordian
knot. Hmm... you may be thinking that StorageSearch's editor never gives easy
answers to SSD questions if more complicated ones are available.
||But in this case you'd be
wrong. (I didn't say you'd like the answers, though.) ...read the article|
SSDs Technical Pros and Cons|
|The great attraction of PCIe for SSD
oems is that it can support a wide range of performance options with throughput
upto 16GB/s, and much lower attachment costs than the
older busses like PCI and cPCI also provide performance which is adequate for
Bus connected SSDs have been around since the
of the SSD market.
The advantage of this approach is high
throughput and low latency compared to SSDs connected via traditional hard disk
style interfaces like
there are disadvantages too which include:-
1 - Bus style
interfaces reduce the available market for the SSD oem. Because older servers
may not have the interface, or perhaps the interface (for example Sun's SBus) is
proprietary and is only available in a small range of models.
2 - Bus
interfaces tend to have shorter permissable cable lengths - which restrict how
such SSDs can be connected.
3 - Bus interfaces usually don't include
intrinsic end to end error detection and correction. If the physical arrangement
of the SSD pushes the speed and cable lengths too far - then errors can arise in
the bus connect - which have to be dealt with in the associated driver.
May 13, 2009 - Dolphin's
CTO, Venkat Krishnan emailed this article correction.
StorExpress addresses concerns of PCIe direct attached SSDs raised in (2) above.
It includes support for different types of PCIe interfaces (ExpressModule, AMC,
etc.). Multiple PCIe SSD cards can be used without requiring multiple PCIe slots
in the server. The storage can be collocated at distances of up to 300m from the
server and can also be potentially shared by more than one server."