For the last 12 months people have been saying that all-flash and hybrid configurations are getting really close in terms of pricing. During the many conversations I have had with customers it became clear that this is not always the case when they requested quotes from server vendors and I wondered why. I figured I would go through the exercise myself to see how close we actually are and to what we are getting close. I want to end this discussion once and for all, and hopefully convince all of you to get rid of that spinning rust from your VSAN configurations, especially those who are now at the point of making their design.
For my exercise I needed to make up some numbers, I figured I would use an example of a customer to make it as realistic as possible. I want to point out that I am not looking at the price of the full infrastructure here, just comparing the “capacity tier”, so if you received a quote that is much higher than that does make sense as you will have included CPU, Memory, Caching tier etc etc. Note that I used dollar prices and took no discount in to account, discount will be different for every customer and differ per region, and I don’t want to make it more complex than it needs to be. This applies to both the software licenses as the hardware.
What are we going to look at:
- 10 host cluster
- 80TB usable capacity required
- Prices for SAS magnetic disks and an all-flash configuration
I must say that the majority of my customers use SAS, some use NL-SAS. With NL-SAS of course the price point is different, but those customers are typically also not overly concerned about performance, hence the SAS and all-flash comparison is more appropriate. I also want to point out that the prices below are “list prices”. Of course the various vendors will give a substantial discount, and this will be based on your relationship and negotiation skills.
- SAS 1.2TB HDD = $ 579
- SAS 960GB SSD = $ 1131
The above prices were taken from dell.com and the SSD is a read intensive SSD. Now lets take a look at an example. With FTT=1 and RAID-1 for Hybrid and RAID-5 for all-flash. In order to get to 80TB usable we need to calculate the overhead and then divide it by the size of the device to find out how many devices we need. Multiply that outcome by the cost of the device and you end up with the cost of the 80TB usable capacity tier.
80TB * 2 (FTT factor) = 160TB / 1.2TB device = 133.33 devices needed 134* 579 = $ 77586
80TB * 1,33 (FTT factor) = 106.4TB / 0.960TB device = 110.8 devices needed 111 * 1131 = $ 125542
Now if you look at the price per solution, hybrid costs are $ 77.5k while all-flash is $125.5k. A significant difference, and then there also is the 30k licensing delta (list price) you need to take in to account, which means the cost for the capacity is actually $155.5k. However, we have not included ANY form of deduplication and compression in to account. Of course the results will differ per environment, but I feel 3-4x is a reasonable average across hundreds of VMs with 80TB of usable capacity in total. Lets do the math with 3x just to be safe.
I can’t simply divide the cost by 3 as unfortunately dedupe and compression do not work on licensing cost, unless of course the lower number of devices would result in a lower number of hosts needed, but lets not assume that that is the case. In this case I will divide 111 devices required by 3 which means we need 37 devices in total:
37 * 1131 = $ 41847
As you can see, from a storage cost perspective we are now much lower than hybrid, 41k for all-flash versus 78k for hybrid. We haven’t factored in the license yet though, so if you add 30K for licensing then (delta between standard/advanced * 20 CPUs) it means all-flash is 71k and hybrid 78k. The difference being $ 7000 between these configurations with all-flash being cheaper, well that is not the only difference, the biggest difference of course will be the user experience, much higher number of IOPS but more importantly also an extremely low latency. Now, as stated the above is an example with prices taken from Dell.com, if you do the same with SuperMicro then of course the results will be different. Prices will differ per partner, but Thinkmate for instance charges $379 for a 10K RPM 1.2TB SAS drive, and $549 for a Micron DC510 with 960GB of capacity. Which means that the base price with just taking RAID-5 in to consideration and no dedupe and compression benefit will be really close.
Yes, the time is now, all-flash will over take “performance” hybrid configurations for sure!
I wonder what the result would be when taking into account the power/cooling savings across larger such deployments? Pretty significant I’d guess…
Exactly, that is another point. I did not take this in to account as it is too complex to lay that out in a short blog post, but there are big advantages in that space as well.
The main unknown factor is the deduplication/compression factor. There is no white paper stating 3x as an average, so it’s difficult to factor in the numbers into a budget proposal.
There is no paper indeed and there won’t be one either. We cannot guarantee anything without a solid understanding of your data set. I used 3x just as an example to show when we get close. Whether that is realistic for your environment is definitely not up to me to decide.
I’m just starting to look into VSAN as a possible alternative to traditional SAN. If I am reading this correctly, you are saying Raid 1 and no dedupe for spinning disk vs raid 5 and dedupe for flash. I’m just curious as to why. Apologies if I missed he reasoning in the article.
I would recommend reading this article to start with: https://blogs.vmware.com/virtualblocks/2016/02/12/the-use-of-erasure-coding-in-virtual-san-6-2/
In summary: the IO amplification and cost associated with these data services just make them unfit for hybrid in our opinion. Especially at the rate of which flash prices are comig down.
Good stuff on what VSAN 6.2 is enabling. Cant with for a 15TB SSD in a vsan node.Just one thing
We always need to be careful when your “banking” on dedupe. Persistent desktops vs Linked mode pools vs DB workloads, vs alot of Windows web servers all deliver a crazy variance of ratios. The danger on the partner side, is the customer usually is looking for that guarantee..
ANd I fully agree with you, I am taking an example here where All-Flash is on-par with Hybrid, but of course the use case will determine whether you get 2x or 10x data reduction (or anything in between). I will not make guarantees, and I will definitely not mention unrealistic 30x / 40x data reduction numbers.
So basically All-Flash is approaching hybrid because VMware has decided to put space efficiency features only in their higher cost all flash tier. I’m tired of buying VMware products and then having VMware tell me that I need to buy a higher tier of licensing to get any new features.
No, we didn’t enable space efficiency only for all-flash for that reason. It is about IO amplification etc. In order to get the same performance on a hybrid configuration with dedupe/compression/R5 or R6 you need to buy more spindles. Saving capacity while needing more spindles to provide the same performance doesn’t make sense. I would recommend reading this article to start with: https://blogs.vmware.com/virtualblocks/2016/02/12/the-use-of-erasure-coding-in-virtual-san-6-2/
As a customer; I like the approach explained here as its one of the closest.
However, a VSAN vs Hybrid discussion is also a ‘node’ vs ‘array’ mind shift.
Your example for VSAN lacks Hotspares and shelf redundancy. Those will be in the Hybrid systems.
What happens when one of your VSAN nodes is in maintenance mode? In your case 10% of your raw / parity are lost while it is out of service.
We’re getting closer, but this will never be a definitive ‘once and for all’ answer.
@vTrooper
I am not comparing “hybrid SAN” with all-flash VSAN, but hybrid VSAN with all-flash VSAN. With VSAN there is no such a thing as hotspares, your whole cluster functions as a hotspare, however you need to keep that in mind of course during the design. if you have 3 nodes and do maintenance well then there is nowhere to go when you want to evac data. So you still have to think about it indeed.
Am I missing something here , what happened to the dedupe
On hybrid options !!!?? .. That’s a bit of strange maths to include dedepue just for all flash . Any person with experience in storage will release majority of data is cold , so a hybrid system with ssd to no-sas will be much more cost effective .
These data services are not available for hybrid VSAN as explained in other comments, hence I cannot include them in the calculations.
I read the article you referenced, “The Use Of Erasure Coding In VMware Virtual SAN 6.2.” So after reading this I just had a quick question on the node requirements. Before you could have a minimum of 3 hosts or 2 with a remote “watcher” node. Now if you choose the Erasure coding RAID-5 you need 4 hosts and if you go the RAID-6 you need 6 hosts?
Thanks for the help,
Mike
Yes, minimum is:
RAID-5 = 3+1 = 4 nodes
RAID-6 = 4+2 = 6 nodes
I think you chose a SATA drive instead of a SAS driver for the SSD. This is the current list from the Dell site
960GB Solid State Drive SATA Read Intensive TLC 6Gbps 2.5in Hot-plug Drive, PM863 [$1,131.45]
The closest I am seeing for a SAS SSD is
800GB Solid State Drive SAS Read Intensive MLC 12Gbps 2.5in Hot-plug Drive [$1,310.94]
Kaminario is the only all flash provider on the market to offer a written guarantee on effective capacities- we guarantee a minimum of 3.5 to 1 data reduction in mixed workload shops. If we don’t hit it- you get free flash to make up the difference. Ask your flash provider for a written minimum effective capacity guarantee.
Okay thanks… not sure what that has to do with VSAN, but I guess you want to get the exposure as a vendor 🙂
Great post. I have great respect for blog writers that not only spend a lot of time writing quality posts, but help and answer any questions that readers write. Great work Duncan, keep it up!
Hi Duncan,
I did some calculations recently similar to yours (would have been nice if I had seen this beforehand and save me the effort). But I came to similar conclusions. We assumed no dedup on DBs and 5:1 on the rest. Resulting in an average ratio of 2.5 (in a situation where 50% of the data was DB oriented). In our case All Flash was a little bit more expensive, but the difference was so close that it was deemed insignificant. With all the other advantages of flash and the view that the cost of flash would get closer to disks in the future. Flash it was.
It would be nice if you could have different storage tiers (mix of hybrid and all flash) in the same cluster, where based on policies, actual performance, dedup ratios, etc. the data would be stored on one or the other.
Also I included power. We took 2W per flash device (including PUE) and 10W for a disk (including PUE). In your example it would be 1.33kW for Hybrid and 74W for all flash. Almost factor 20 difference.
In cost BTW the impact is there but not that significant. probably around 1000 euro per year on your configuration. Still nice to have though, and always a good story that you cut the cost of power with > 90%, Green IT may not be the hype it was 5 years ago, but still it is always a good message to your organization and beyond.
And compared to a traditional disk based or Hybrid SAN you will see even more power savings (as enclosures, SAN switches, etc. also consume a lot of power).
Hi Duncan,
I have been asking numerous people how do we size the expected de-duplication and compression ratio with VSAN as this is key to your argument above.
For VxRail EMC use Conservative (1.5:1), Moderate (2:1) and Optimistic (3:1) and I was advised to use Conservative as there is no way of guaranteeing the other two numbers.
I just do not believe 3:1 is realistic, but on the other hand I have no tools or guidelines to work with so I am completely in the dark.
Any thoughts would be appreciated – seems to be rather a huge gap in the VSAN all-flash story today.
Best regards
Mark
It is difficult to say indeed without knowing the environment… whether is is 3:1 or 2:1 or even less. As mentioned database typically don’t dedupe well, but the first 30GB of most VM images do, usually because they are all the same. I have seen results ranging from 1.5:1 to 10:1 even, but that fully depends on what you are running. We are definitely not in the business of giving guarantees or making false promises. I understand your request, and I have already asked the engineering team to see if there is a way we can do some form of “hinting”, where we can provide you in sights around how effective the process could be (or not).
Hi Duncan,
Thanks for the feedback.
I appreciate that it is not a precise science, but it would be useful to have a tool that allowed us to specify the type of data (i.e. VM, Exchange, DB, VDI, files, media files, etc), quantity, number of disk groups, etc.
This would allow us to get a fairly accurate estimate and then it is down to the customer to decide how conservative they want to be.
Many thanks
Mark
Hi Duncan!
I am just trying to find the answer for one of the issue, which I faced recently.
We initially built a vSAN cluster with 5 hosts and 5 capacity disks for each disk group.Each host had 1 disk group. and we enabled compression and deduplication in the vSAN cluster. Later we decided to add 2 more hosts to the vSAN cluster and also add 2 more capacity disks to each disk group. I was able to add 2 new disk groups to the vSAN cluster and when I tried to add 2 more capacity disks to each disk group, I got an configuration error stating that compression and deduplication should be disabled before adding disks to each disk group. Once I disabled compression and deduplication in the cluster, which took almost 30 hours, then I was able to add capacity disks to each disk group. after adding the capacity disks, I re-enabled compression and deduplication and that took another 30 hours. I am wondering, if that restriction will be changed in vSAN 6.5?