I can see you have given the subject a great deal of thought, but let me address a few of your comments and perhaps help you further see why I argue the way I do.

As far as cost goes, we don’t all buy the same chips at the same price, and that price is not RAID neutral. And the RAM in my box is a lot cheaper than in your server, I don’t need the speed or density of parts a server needs so its a lot cheaper for me. The deployment increment of PCIe cards almost guarantees higher cost and I’ve never seen PCIe cards deployed with spares which means taking down the server and cracking open the box to replace the failed board. two things most of our customers don’t want to do. Remember its not just the flash that fails, any electronic part can fail, also don’t forget that they will “fail” once or twice a year when you upgrade the software in the server.

Saving on licensing is indeed a great attraction, but I think you will find the sort of configuration you are suggesting is rather more operational complexity than most people would be interested in trying to make work, and why would you want to dedicate CPUs to your storage, didn’t you buy them to compute with?

Duplexing the data to avoid being locked in to a crashed server doubles your storage cost. As I said the prices are not RAID neutral.

Differences between out box and a server with PCIe cards.
a) PCIe cards only attach to one switch chip, single point of failure.
b) PCIe cards means software RAID, high latency, lower performance
c) a host of ‘minor’ things like power and density, serviceability

If you just want to build a one off to play with there are many things you can put together, but if you want to deploy storage in volume in a data center environment all the little things matter. Just being able to share the storage between hosts, whose upgrade cycle need not match that of the storage, and not have to open up servers deploy or service the storage are reasons enough for many of our customers to prefer the separate box, for them its not a downside, its a feature.

I don’t know what you were doing with your storage, but if you only needed 10K IOPs and your dataset was small enough, and reliability wasn’t an issue then that’s not really a data center environment. Funny thing about performance, while you say you only needed 10K IOPs, I have yet to meet a customer who on discovering they could go 10X faster, sat there and said “you know 2X faster is fine”, there is always something more you can be doing with your data if only you had the IOPs to do it.

Regardless of that, if you were looking at equipment priced at €40/GB then you were looking in the wrong place, so when you do find yourself needing to buy enterprise flash storage be sure and come talk to us instead.

I doesn’t fit the current Violin approach as well as it fits those PCIe Flash cards with their “fat block layer OS” you dislike so much.

The problem is, that the optimal point for that intelligence moves with the use case and factors like licensing considerations. If Oracle’s “Smart Scan” was an open API you could implement it and gain a fast immediate advantage, because the compression logic wouldn’t be paid out of a database CPU budget… Exactly the advantage Oracle wants to keep for themselves (and need to offset the horrid 2006 Flash hardware).

And it changes with technology: If I remember correctly, Violin started out with something very similar to an Intel Above Board, a (paged DRAM) memory expansion board designed to alivate 80286 RAM issues, except that the board was in fact a rack box, which offered truly huge amounts of RAM space…sold as DRAM SSD, while it could have also been DRAM, for direct database use.

If I have heard correctly, HDS is turning the VSP design with that internal PCIe switch into a generic blade platform: A kind of “Super UCS” where you can have a practically unlimited number of PCIe slots, you can assign and move slots to any blade, which is then just CPU and RAM. With an eGenera BladeFrame like management tool and sub capacity management acceptable to the likes of Oracle and IBM, one could move that logic to fit the use case.

Your current niche doesn’t look very comfortable to me.

We didn’t need “crazy” IOPS, 10k would fit just fine. And paying €40/GB for enterprise grade PCIe storage was just a little high, when even the daily overwrite rate was below 100%.

Our loads were well within consumer SSD specs, but we wanted more reliability and nobody seemed to address that segment with the proper quality.

With LSI purchasing Sandforce I expect that picture to change. Hybrid approaches like the OCZ Velo and Revo units may offer both default OS integration through “known” RAID and PCIe switch interfaces, while they may augment the Flash visibility and handling with customized/enhanced drivers: A bit like paravirtualized drivers for hypervisors.

With FusionIO entering the appliance market you may have to swallow your pride and offer a PCIe solution, too. Or perhaps something smarter, that physically replaces all those unneeded drive cages while it offers plug-in upgradable Flash storage at PCIe speeds…

For me the important price metric for flash cards is that even for these enterprise grade PCIe cards the price per GB is dropping below €10/GB and therefore in many cases more economical than the SAN boxes they replace.

Also it’s not a true differentiator, because you all need to buy the same chips at similar prices. That price is also RAID neutral. If quality and management of these Flash chips is as perfect as you and that PCIe vendor says, then there is no reason why you’d introduce RAID for redundancy inside the server. Flash will deteriorate in a predictable fashion and you will replace cards before they are exhausted. And those driver and glue chips will fail with exactly the same likelihood as CPUs and RAM (or the chips inside your appliance). That’s why you have another box, if you need to cover that angle.

Same for the RAM: Whether you pay for server RAM or RAM inside your boxes doesn’t make a big difference. There may be issues regarding the expandability of that server RAM, but that’s becoming less of a problem these days.

Licensing: Oracle is more profitable than it is fair when it comes to sub capacity licensing. But then you could use Oracle VM to have that “block storage server software” run on CPU cores, which you don’t pay database licenses for. That may be your strongest point, actually, but I don’t think it would ever be strong enough for me to consider the extra complexity of extra boxes: I’d really have to evaluate the CPU consumption inside the “block server” layer. And then your storage interface may not come for free either (while you both may save on the disk protocol layer required by SAS/SATA SSDs).

Losing access: There is two basic reasons for server crashes: Hardware and software. In the case of a software crash, recovery should be fast and automatic. And you need to deal with those crashes in any case, no matter where your data may be. I don’t like RAC for the highest availability demands, because software bugs in RAC tend to make you unavailable. So when you duplex the data before it enters the database server, there is no issue with locked in data.

And for the hardware case, I see no difference between a Violin box and an enterprise server containing PCI flash cards. I looked at your hardware on OpenWorld and as far I could tell they are made of the very same electronic components with exactly the same redundancies and likelihood of failure. If the quality and the number of components is the same, math says they will fail just as likely.

Except, that with an internal PCIe card, there may be one set less cables and boxes, than when you move them all into Violin.

For me the main differentiator is expandability and how far you can go. Current servers designs still favor disk bays over PCIe slots, but perhaps that will change. But with 10TB in on one card I can typically cover everything I need. While I don’t think flash storage replacement will be as frequent as hard disk replacements, the just-in-time capacity upgrades enabled by these 16 drive bays on most 2H servers still look attractive.

I am hoping for some improvement here, where “drive bays” do not imply SAS/SATA speeds and overheads, while mechanical issues are “non existing”. When replacing only whole card buys you another class of reliability I may well be tempted for truely critical workloads.

But nevertheless I am relaxing in a way I haven’t been able to for years, because storage bottlenecks seem to become a very small issue indeed.

We have currently deployed a Coraid ESAN on an Arista Networks layer 2 infrastrcuture.

How would Violin Memory products benefit a solution that replaced a Coraid ESAN?

Are you working with Nexanta Systems? I would like to leverage the NexantaStor product in our VMware 5.0 virtualization infrastructure to migrate and/or support Coraid (SAS.SATA) and Violin Memory SSD Arrays concurrently.

Thank you.

Ron W. Szpak