|
|
Test to Fail
Too
often, testing groups tend to run tests intended to pass a storage peripheral
without understanding what a passing result means. As a storage community, we
need to commit to designing tests that push the limits of the technology. This
means continually evaluating your test plan and increasing the intensity and/or
complexity of a test until failures can consistelntly be generated and real
watermarks for pass/fail criteria can be established. Creating tests that are
too lenient or that circumvent known issues only leads to poor product quality
and customer dissatisfaction which is the main reason we test in the first
place. Testing to fail also requires the intentional injection of errors to test
the robustness and recovery capabilities of your device. The more failure
scenarios that you test for the better prepared your data storage solution will
be for the real world, where the unexpected will happen due to the uncontrolled
environment. Being able to gracefully manage these scenarios is a sign of
quality that will be appreciated by your customers.
Test Smart
One
of the key ingredients in knowing where to begin in your test design is know the
limitations of both the products being tested and the tools available for test.
Depending on the peripheral being tested these limitations could include
maturity, interoperability, speed or a host of other issues. Identify these
limitations in the peripheral to create a clear understanding of the test
objectives and a realistic expectation of the results. Limitations exist in test
solutions as well such as multi-threading, tagged queuing, remote control
access, automation capabilities and low level protocol control. Understand which
features are necessary in a testing solution for your particular testing
application and make an informed decision on which solution is right for you.
Another important facet of "smart" testing is creating test
plans. Test plans should be created for each stage of your peripheral's
development from Engineering Verification Testing all the way through to
Manufacturing and Field Service. This ensures that no matter which stage of the
peripheral's life cycle you are in, there is a process in place to ensure that
it is functioning to the level expected. In qualifying a storage peripheral, it
is important to note that without statistical significance, there is no
significance to your results. This means that you need to test a relatively
large sample of product (when applicable) to ensure that once testing is
complete, enough data, variety of data, mechanical motion etc. has been
exercised through each component to draw a significant, confident conclusion.
Manage
Your Vendors
Managing vendors is sometimes a forgotten art. I can't
even begin to count the number of instances where a call to a vendor application
engineer could have quickly solved problems that had been dragging on at various
customer sites. Developing a good working relationship with vendor support
engineers / application engineers is key to receiving timely updates and
avoiding days of needless debugging.
Thorough testing accompanied by the proper documentation to back up
your assertions of problems is the key to establishing a rational discussion
with a vendor and getting the problems taken seriously and resolved quickly.
Provide descriptions of what was being tested at the time, a copy of your test
(for reproduction), any error logs from the resulting failure and of course
protocol traces if applicable. This enables your vendor to do their job more
quickly yielding in more rapid root cause failure analysis and problem
resolution.
Besides product updates, vendor engineers can also provide useful tips
and tricks for testing their products as well as lists of outstanding and
resolved issues which can help to pinpoint problems in the field and in the lab.
Remember that vendor data for MTBF (Mean Time Between Failures), read/write
error rates and all other statistics presented should be used as temporary
watermarks until real results can be ascertained in your own test
environments.
Summary
As we have discussed in this
paper, testing is an interactive, multifaceted process that requires an
understanding of the quality required and determines step by step processes to
achieve and maintain that quality in all stages of a products life cycle. It
should be a dynamic process that evolves as the technology being tested
necessitates. It must push the test subject to its limits and ensure that the
quality customers expect is found in each and every product sold. Testing should
not be viewed as a necessary evil but rather a means of achieving quality,
confidence, and ultimately customer satisfaction. ...Extreme Protocol Solutions
profile |
|
|
|
| |
|
|
|
|
|
|
 |
TMC-The Mate Company,
with headquarters in
Glendale, California, manufactures an extensive
range of cables, adapters, converters and
terminators for fibre-channel
and SCSI systems. | |
|
|
|
|
|
| |
