Now in 2012 it seems clear that solid state storage will be the solution to balance the network-compute-storage triangle and provide the IO necessary to not only virtualize the easy applications but now tackle the hard (latency-sensitive) IO bound applications that have been fine tuned to run on dedicated servers.

The easiest way for incumbent array vendors to get in the game is to use solid state in the HDD form factor (SSD) and plug them in to the traditional storage array. Yes, you can take a Fibre Channel or SAS SSD and place them in an enterprise disk array and you will most likely see a performance increase but nowhere near the datasheet expectations of the SSD. The main issue is latency.  The big array has multiple layers of controllers (RAID, DeDup, shelf controllers, etc) to make the “array of unreliable HDD’s” appear reliable in the first place.  These layers will always reduce the “potential” offered by the SSD and in most cases only a few slots in shelf can be filled with SSDs before the shelf controller is saturated.  This obviously creates a strange looking array if you tried to fill it will all SSD’s as 80 percent of the slots would be empty because the array will be max’ed at the limits of the performance it was designed to support  At the same time the price paid for these SSD’s from the array vendor is much higher than the street prices you will find online.  It’s all about protecting those margins.

It is like buying the engine from a race car and putting it in a minivan, sure it will go faster, but not much faster, it just isn’t built to handle it.

Lets look a little deeper at why those SSD’s don’t really fit in today’s storage Array. First all of these devices use disk protocols:

• SAS (Serial Attached SCSI)
• Fibre Channel (really Fibre Channel Protocol for SCSI) SCSI
• Perhaps SATA (very slow)

Some of these newer interfaces may be faster than older disk interfaces, but they are still disk interfaces and limit the bandwidth and increase latency when used with SSDs that are capable of so much more performance.

Stepping back a bit – if you only put a handful of SSDs in a disk array they are usually used as a cache or tier.  This can be a valid strategy, but one that assumes a very good cache hit rate.  In a simple case where a single cache miss might add 10ms of latency –  it only takes a 1% miss rate to cut the performance of a single threaded application in half.

If you take a look at the specs of any enterprise storage array, it will usually have more front side bandwidth (facing the hosts) than it has backside bandwidth (facing the disks). This is a perfectly good choice for these systems since they all have large DRAM caches that serve data to users and stage writes into the disks.  The disks are rarely run at full utilization even when under high user load.  In fact, I recall being told the difference between a consumer RAID controller and an enterprise array was the consumer device had more bandwidth available to the disks than to the host while the enterprise array had more bandwidth from the cache to the users than it did to the disks. While this was probably a good design for disk systems I will show you that it’s not true for flash, and as a result the legacy disk array simply is the wrong place to put flash storage.

Remember that the reason you were considering putting SSDs in the storage array is the existing cache isn’t performing and the disks are running at full utilization. By implication this means the disks are running out of IOPs not bandwidth, so there still should be capacity for pref-etching from the SSDs, or maybe not? Remember, if the access patterns were such that IOs weren’t pre-fetchable in the first place, then you are still going to take large latency hits at the start of every access to be loaded in the SSDs. If you are lucky you will be performing large accesses to allow some amortization of the access cost, but you must be reading multiple megabytes to overcome the pre-fetch cost. And if you are reading multiple megabytes sequentially then you probably didn’t need the SSDs in the first place.

It is also likely that your SSD, like some PCIe cards, has its own internal RAID because it is possible for entire blocks to fail and on rare occasion even entire NAND die. Despite all the fancy feature names and white papers predicting ominous consequences if you don’t use a specific vendor drives with their Super Extra Special Enterprise Strength Enhanced Storage Enabled Safety Engine aka (SE)^5 ™, recovering from failed flash blocks is little different than remapping failing disk sectors which disk drives have been doing for decades. At Violin, we have a whole host of mechanisms designed to ensure every piece of data you give us is protected as one would expect from a piece of enterprise storage equipment.

But back to the beginning, the SSD has no idea if it is being used in a RAIDed array or not and must implement its own internal RAID to recover from such failures to provide the sort of data protection required for enterprise storage. So like the PCIe card, when you use the SSD in a RAIDed storage array you are paying twice for RAID.

Another issue is disks only have a limited amount of remapping space available and most arrays controllers are designed to handle a head crash in a disk where all the bad sectors are in the same place.  They aren’t designed to deal with a drive that can suddenly have thousands, or even millions bad sectors appear randomly all over a drive.  Unlike a head crash where all the bad sectors are in the same place, when a block, or worse a whole flash die fails it will contain sectors with logical block address spread over the whole address range.

Like the PCIe card is to the server, the SSD is really is just an expensive way to try boost the cache performance of an enterprise disk array and provide the incumbents an incremental introduction to the capabilities of solid state while maintain their existing profit margins and data center footprint.

If one were to take the next logical step – the flash Memory Array (Violin 3000 & 6000) is a “ground up” design to aggregate flash in most economical and reliable way to provide a path to low latency “memory speed” performance in a footprint measured in 3RU  (rack units) rather than multiple racks of short stroked hard disks.

Putting SSDs in legacy arrays is just like putting a race car engine in a minivan, the only bang you are likely to get for your buck is the sound of the under-built transmission exploding the first time you step on the gas.

You can view more Storage Switzerland videos on YouTube

George also put together the below video where he gives a short chalk talk on where to use solid state storage.

• Integration standards
• Distributed services
• Higher-performance compute, networking, and storage
• Implementing low-latency services
• Maintaining performance as you scale

Part 1 (Garbage Collection) and Part 2 (Commodity SSDs)

What about PCIe cards?

Another option is to pack as much flash as possible onto a PCIe card to sit in a high speed slot on a server.  Because of their much higher interface speeds, PCIe cards have much better performance than your typical commodity SSD, but face their own unique issues. To speak with the Operating System (OS)  PCIe cards require specialized software drivers.  With some cards these drivers are so heavy-weight that their vendors don’t even call them drivers anymore but try to convince you they are a great value added software layer. That might be true except for the fact that those cards gain their performance by stealing CPU cycles from the core that’s hosting it, the same core running the business software your trying to accelerate.  At the same time you may be paying software license fees (per core) and now have to evaluate what is serving the application vs what is needed to keep the PCIe card running.  At the same time you will require many more GB’s of DRAM to handle the metadata for the PCIe cards.

Consider the following:

• In most cases a single PCIe flash card costs 3 or 4 times that of a server itself
• Evaluate the  CPU cycles a PCIe card with a heavy driver steals from your server
• Now look at the full cost of the server and software licenses
• Then look at the cost of the PCIe card itself (in perspective)

What was the value proposition again?

To be fair, card vendors have worked to lessen their driver footprint and some have even begun to show smaller write cliffs than the commodity SSDs, but it’s still there.

So do PCie cards have a place in the data center when there are single points of failure everywhere?  Some of the higher end cards have started touting they have spare chips to swap in if they detect a failing component, some even have on-card RAID in the event of a spontaneous die failure. Such protections are of little use if the single controller managing the flash fails or crashes.

You can run multiple PCIe cards in a server but you need to choose between performance and reliability as  multiple cards typically run in a RAID-0 configurations for performance. If you wanted actual RAID data protection, you would have to use multiple cards (assuming you have enough PCIe slots) and  were willing to pay an extra ~50% (assuming 3 RAIDed cards).

Then there is the fact that a flash PCIe card is locked in a server.  You lose access to that  data  if the server crashes. So if you actually *needed* that data, then you don’t have much choice but to mirror your servers, pay a 100% overhead and taking the performance hit for mirroring. Even worse you may be paying more than 100% overhead if the card offers some on-board RAID feature that you don’t need because your already mirroring for availability reasons.

Add to this the inability to service a PCIe card in a running system and it seems clear that PCIe cards are acceptable as a memory extension technology.  Reliable cost effective data storage – probably not.   Properly deployed flash is a strategic data center resource that can’t be locked in a single server – it needs to be shared.

So the answer must be in the Storage Array’s themselves….   (part 4)

It’s easy to see why HDD form factor SSDs seem attractive: they have the same look and feel as a regular disk drive, they do usually weigh less, and they fit in the same HDD connectors you have in your existing server. That’s all you need to see great flash speed, right? Sure they look good in the benchmarks, and while some benchmark sites have become a lot more sophisticated in measuring SSD performance, the SSDs have also become more sophisticated at Specsmanship.

In the end the commodity SSD fails in the enterprise for a lot more reasons than just its inability to live up to exaggerated performance claims. The commodity SSD isn’t going to have the error protection or failed component recovery of an “Enterprise Flash Drive”.  The commodity drive is single bus attached so at best it can be used in simple server settings, but not in proper RAID enclosures with redundant controllers. The algorithms in commodity SSDs are optimized around the access patterns of your standard desktop machine and none of these optimizations are going to help your enterprise server.

Similarly, compression or dedupe built into the SSD sound like great ideas for the average desktop PC (and they sure do improve benchmark numbers) but not necessarily for the enterprise. Enterprise apps and services, or the storage arrays they are plugged into, are often already performing those functions. Additionally, encrypting the data right from the application is also becoming more and more common, which totally defeats these benchmark-oriented optimizations.

The commodity SSD isn’t meant to be sold to someone focused solely on performance, it’s meant to be as cheap as possible. So when a few months later you buy some replacements for a failed drive, or to expand your storage capacity, imagine your surprise when you find they have half the performance they used to have, because they got great pricing on Flash chips that were 2X as big, and so the vendor put half the number of chips in as the ones they sold you last month, half the chips, half the IOPs, but the price per bit is great isn’t it? The price per IOP? Not so great. (And yes, some vendor really did do that.)

So clearly packaging matters, particularly when you are talking about deploying in a demanding enterprise environment. The real objective is not so much about packaging flash, but optimally aggregating flash for the enterprise. I’ll get more into that in the next two posts.

The very adventuresome amongst you might start with, “I’ll just make my own”.

Roll your own

I can’t tell you how many times I’ve heard people, be they end users or large system vendors who thought they could just roll their own flash storage. They wouldn’t think of building their own disk drive, DIMMs or mother-boards, yet some how they think building their own is a perfectly reasonable idea when it comes to flash. Why do they think making their own is a good idea? Two top reasons, price and performance. In the old days, flash was expensive and vendors sold it at very steep premiums, so obviously the answer was to make your own and cut out the middle man and save money! Lets be honest, unless your name is Steve (or now Tim), or you happen to be partnered with a major flash supplier, you can’t just go out in the spot market and buy flash in volume with reliable supply at a price that ever makes building your own drive make sense. Also you would have to learn how to manage the flash, so advanced error-correction, how to make the flash work when the power fails (something even todays large vendors have trouble with)  and how to handle the highly-variable performance issues of consumer-grade MLC. Oh you didn’t know that when the MLC data sheet says “typical”, the metric in question could range from 3x to 1/3 of that “typical” value? Does that complicate things? It “typically” does.

Sure, MLC benchmarks at the advertised speed, for a few minutes, then the garbage collection (grooming, page reclamation, flash block  cleanup…) starts, and then it falls off the write cliff. All that bus bandwidth promised to you, starts being using by the drive to perform garbage collection and that means your performance goes down drastically and your price/performance numbers go up along with your frustration levels. You can make your own flash storage, but you will have to garbage collect once all the empty space is written to, just like everyone else does. Garbage collection is necessary when the drive is busy so that there are always cleared flash blocks available for the application to write to. When this happens, the bandwidth available to the user decreases because it’s spending its time doing reads, writes and more importantly erases which all take time. Accessing flash for reads nominally take 90 microseconds, writes can take a millisecond or more and in the case of erases 5 to 20 milliseconds. While the SSD is busy erasing a block, the entire flash die holding that block cannot be read from, so if you wanted to read a page of flash from a die being erased you may have to wait up to 10 milliseconds for your data. Imagine the number of 90 uS reads that might queue up while that 10 milliseconds counts down to zero – think that might generate a latency spike at the application level?

Latency spikes are far more important than the loss of performance due to the write cliff.  Most benchmarks just throw as many outstanding IOs at a drive at the specified queue depth of the test. For a big enough queue depth of *unrelated* reads, the additional latency doesn’t appear to effect the performance. But real applications are not like benchmarks, when the application reads data from storage it often will issue a handful of reads before it has to wait for a response, then it can issue the next set of reads.  Think about files systems or database search indexes -  those data structures are traversed based on the result of any given read and that read will point to the next location to be read. Most interesting applications are like this and if the latency for a single read is very low (say 90us,) then even with only a single outstanding read an application can deliver 10,000 IOP/s, but if applications’ reads keep getting stuck behind erases, then in the worst case it might only get 100 IOP/s. That would result in a factor of 100 loss of performance. Now will erase blocking always be this bad? No, “typically” it will not be this bad, but sometimes it will, meaning you will have totally unpredictable application performance.

So when people complain that existing SSDs don’t deliver on their promised performance it is usually due to the write cliff and ongoing  garbage collection.   The performance loss for streaming applications and almost all enterprise applications is the result of the large unpredictable spikes in read latency inherent in poor garbage collection strategy (and implementations.)

Now if I were in marketing, after telling you everyone has to do garbage collection, I would tell you how we at Violin have developed some magical technique to make garbage collection go away, and then you would justifiably ignore the rest of what I have to say. But I take the T in CTO seriously, so I’m not going to say we don’t have to do garbage collection, we do, why don’t we have a write cliff,?, well you have to read the rest of the future blogs to find out.

That being said we do have an advanced (patent pending vRAID) technology that makes those giant latency spikes (due to erases) completely disappear under *all* circumstances, which some might consider sufficiently advanced to be indistinguishable from magic. Certainly our customers find the resulting “predictable performance”  application acceleration experience using our Memory Arrays to be pretty magical.

So let’s get back to basics and think about why we buy Flash in the first place. It’s certainly not as a capacity tier where $/GB concerns dominate. Flash’s real value comes from how it accelerates applications through very low latency and very high IOPS, so it’s microseconds and $/IOPS that matter most. You wouldn’t want your applications to slow way down by doubling or tripling IO wait times, that’s what Flash is supposed to solve. So let’s drill down on these features from a Flash point of view.

DDUP in its most basic form takes every application write, computes its hash using something like SHA-1, and looks for a matching hash in a big DRAM table. If there’s a match found a pointer to the existing block is written instead of consuming a new block and writing a pointer to that instead. Note that in some products a hash match is considered to be safe enough to treat the data as identical. Other products read the existing block and compare the bytes to avoid the very small chance of a false match and just pay the added performance penalty to be absolutely safe in mission critical applications. Now to add additional complexity, DDUP can be done inline where the performance impact is added write latency, or in the background which causes a reduction in available IOPS. When evaluating inline DDUP solutions it’s critical to ask how much impact there is on write latency as that goes directly to application performance. When evaluating a background DDUP solution the additional IOPS load needs to be considered carefully, if your applications are not pushing the array anywhere near its IOPS limit this may have no visible impact but otherwise applications will be impacted in direct proportion to the write activity.

Now I would be remiss if I didn’t mention a really big gotcha, block layer DDUP increases the risk of catastrophic data loss. Now a statement like that requires some explanation. While it seems strange to want more than a single copy of data it’s important to know that all file systems intentionally duplicate critical information to avoid single points of failure, see super blocks and ditto blocks. If your DDUP product removes intentional redundancy you run the risk of losing all of your data. This issue can best be addressed when DDUP is run in a file system like ZFS and SIS or in application layer products like Ocarina and Permabit, the block layer just doesn’t have enough knowledge to do the right thing. Alternatively one could configure the block layer DDUP feature to keep at least two copies of every unique block just in case, make sure you have that option before you risk losing everything because of a single point of failure.

Now let’s talk a little about compression. Compression is a computational process that shrinks the data on write and expands it to its original form on read. Now a useful fact to know is that the space savings generally increase as we compress larger blocks, this is the opposite of DDUP wherein the smaller the block the more likely you find a duplicate. If writes are small and random there’s not a lot of opportunity for compression to add much value at the block layer, so beware of optimistic vendor scenarios that assume large writes or mostly sequential IO unless that’s what your application actually does. Note that the file system and application level products mentioned previously again have an advantage because they can work across an entire file and can even use file type-specific algorithms with superior compression rates.

In conclusion, when evaluating any data reduction technology for Flash be sure you don’t give up the primary value proposition of high application performance in search of lower $/GB. The gains you hope for are dependent on the type of data you store, they are not a guarantee. Also, make sure you can turn these features off so you don’t give up performance if your data doesn’t compress well or you consider DDUP too be to risky. Finally, remember that the price you pay for an array is the true $/GB and any hopes of 5-10x is a wait and see.

Well first I have to say that I’m a big believer in Flash as a replacement for high RPM spinning disks. While this is not exactly news in the storage industry, it’s nevertheless the early days of Flash in the Enterprise and I’m excited to have a chance to rethink assumptions on how we’ve always done things. Whenever something gets a hundred times faster and cheaper it’s time to ask yourself “Should we be doing something different?”

I’ve spent decades trying to avoid random IO in applications and file systems and treating every IO as millions of CPU instructions thrown away. I’ve designed algorithms and implementations, used caching and Tiering, tried short-stroking and other unnatural acts of storage, even used DRAM and batteries, all to address the central fact that spinning disk is the enemy of application performance. However, we are now in the Memory Era and the rules have changed. Storage latency has plummeted, IOPS have skyrocketed, and bottlenecks have moved. The performance leading RAID controllers and arrays of last year are easily overwhelmed by even the smallest amount of Flash.

So clearly I’m jazzed about Flash but still, Why Violin? Fundamentally, I believe that this all-Flash external array is what the Enterprise really needs. It has all of the read and write performance advantages of Flash without losing the availability and scalable data management platform that customers have come to expect from external arrays. While the PCIe-attached Flash cards have had OK adoption, few customers can lose access to their data when a server goes down, something that is out of the Flash card’s control. Basically, guaranteed data availability is just as important as high performance. Today, Flash card customers have the not so hidden cost of either synchronously mirroring their data, with the resulting cost and performance hit, or losing access to data, an even bigger impact.

Now I shouldn’t forget to mention the arguably biggest vehicle for Enterprise Flash which is as a performance tier for the disk array vendors. Without revisiting the many entertaining blog wars over who has the best Tiering strategy, it’s clear that the center of gravity for these vendors is spinning disk. Every design assumption from performance to reliability to data management is based on what they could achieve with spinning disks. What is needed is a purpose built flash Memory Array whose performance is predictably excellent, and that’s what Violin has developed.

So Violin has a great array but where do I come in? Well the next battle ground in Flash will be over data management and I get to drive our vision to be the leader in this space. As Memory Arrays achieve wide deployment the winner will be the one that delivers data management without sacrificing performance and I’m here to make sure that Violin is that leader.

Jonathan Goldick
CTO Software
Violin Memory