Yellow Bricks

by Duncan Epping

Enable Storage IO Control on all Datastores!

· ·

This week I received an email from one of my readers about some weird Storage IO Control behavior in their environment.  On a regular basis he would receive an error stating that an “external I/O workload has been detected on shared datastore running Storage I/O Control (SIOC) for congestion management”. He did a quick scan of his complete environment and couldn’t find any hosts connecting to those volumes. After exchanging a couple of emails about the environment I managed to figure out what triggered this alert.

Now this all sounds very logical but probably is one of the most common made mistakes… sharing spindles. Some storage platforms carve out a volume from a specific set of spindles. This means that these spindles are solely dedicated to that particular volume. Other storage platforms however group spindles and layer volumes across these. Simply said, they are sharing spindles to increase performance. NetApp’s “aggregates” and HP’s “disk groups” would be a good example.

This can and probably will cause the alarm to be triggered as essentially an unknown workload is impacting your datastore performance. If you are designing your environment from the ground-up, make sure that all spindles that are backing your VMFS volumes have SIOC enabled.

However, in an existing environment this will be difficult, don’t worry that SIOC will be overly conservative and unnecessarily throttle your virtual workload. If and when SIOC detects an external workload it will stop throttling the virtual workload to avoid giving the external more bandwidth while negatively impact the virtual workload. From a throttling perspective that will look as follows:

32 29 28 27 25 24 22 20 (detect nonVI –> Max Qdepth )
32 31 29 28 26 25 (detect nonVI –> Max Qdepth)
32 30 29 27 25 24 (detect nonVI –> Max Qdepth)
…..

Please note that the above example depicts a scenario where SIOC notices that the latency threshold is still exceeded and the cycle will start again, SIOC checks latency values every 4 seconds. The question of course remains how SIOC knows that there is an external workload accessing the datastore. SIOC uses a what we call a “self-learning algorithm”. It keeps track of historical observed latency, outstanding IOs and window sizes. Based on that info it can identify anomalies and that is what triggers the alarm.

To summarize:

  • Enable SIOC on all datastores that are backed by the same set of spindles
  • If you are designing a green field implementation try to avoid sharing spindles between non VMware and VMware workloads

More details about when this event could be triggered can be found in this KB article.

Reader Interactions

Comments

  1. Hi Duncan,

    i get the same error but within my infrastructure it was the veeam backup that accesses the lun´s, respectively the vm´s over san, which then is an legitimate error.

    Gernot

  3. What is often done is that multiple vmfs LUNs (and vmfs only) are carved out of a single set of spindles. Am I correct to assume SIOC should not trigger on an external workload?

    So what is considered an external workload? Spindle sets used by more than one vCenter instance (eg LUN1 to vCenter1 and LUN2 to vCenter2) would cause both vCenter instances to trigger this external workload alarm I assume?

  4. Your explanation seems plausible, but I wonder why you conclude it must be the spindles. In an IBM SVC architecture the concept of spindles is hidden by the SVC, which presents VDisks on the client (ESX servers) side. It could be the SVC load as well, or the FC switches,…
    Personaly, I don’t think you can set up a modern SAN architecture where you have to dedicate spindles to specific applications.
    Shouldn’t your first conclusion, if I follow your reasoning, say “Enable SIOC only on datastores that have exclusive use of a set of spindles” ?

    Another point that I don’t really get is why we see these non-VI Workload alarms only on some of the nodes in a cluster. Does ESX only detect these anomalies when there is a certain amount of datastore activity ?

    Are there any user-configurable parameters for this algorithm ? Shouldn’t there be some ?
    Are there any metrics exposed that could provide more information ? Shouldn’t the alarm be a metric alarm , where the user can define a threshold, instead of an event alarm ?
    Could a VCB backup impact the algorithm ?

  5. Good article, Duncan; this is related to a discussion that both Scott Drummonds and I had on our respective sites a short while ago:

    http://vpivot.com/2010/11/03/sioc-event-ignore-or-panic/

    http://blog.scottlowe.org/2010/11/01/sioc-event-with-emc-mirrorview/

    In our case, we were specifically discussing the impact of replication technologies on SIOC-enabled datastores, but the resulting error message was the same.

    Interestingly, your comment about how SIOC responds to this sort of event implies that SIOC would automatically disable itself (“stop throttling the virtual workload”) on datastores that are being replicated. Can you confirm if SIOC stops throttling anytime this event is encountered?

    • Yes SIOC will stop throtling the virtual workload when it notices an external workload is more than likely impacting the performance. Which makes sense as other wise something like a replication could push away regular VM IO.

      • If I’m understanding your reply correctly, then this means that SIOC can’t be combined with replication and still be effective. If the presence of the replication technology triggers an SIOC event then SIOC will, in turn, stop throttling. In my mind, this presents a pretty significant design impact. Do you have any thoughts on this?

        • I don’t see why it is a hefty implication Scott. As at the point where it stops throttling you are back at the point before SIOC existed. There is no point in SIOC throttling a workload which is not more than likely not causing the increase in latency.

          In a normal situation latency will not be that high constantly due to replication. When it is it is more than likely due to a large amount of changes that need to be replicated and as stated there is no point in throttling a workload because of that.

  6. “If and when SIOC detects an external workload it will stop throttling the virtual workload to avoid giving the external more bandwidth while negatively impact the virtual workload”

    Well this makes SIOC worthless for anyone using XIV, EVA, etc as these architectures don’t really lend themselves to non-shared spindles. Personally I think there should be a tunable parameter to determine how aggressive SIOC is when it detects an outside workload instead of it just giving up.

    • No, it changes the game slightly. If you work with HP you need to be aware what is running on those spindles. So it is a design consideration. Calling it worthless is overstated.

      Also, it will not throttle the virtual workload because throttling it doesn’t change the latency. It is the external workload that is causing this issue and SIOC can’t throttle that one.

      • So it doesn’t turn the feature off just because it detects another workload but rather stops throttling temporarily if the other workload is the cause of the current latency? Because if so that makes complete sense though I would still wish to have some ability to scale back the VM workload, perhaps assign a SIOC share to external where the external load was always assumed to be 100% of that share value.

  8. I enabled SIOC on all my LUNs in my homelab, just to see what would happen. All LUNs are carved out of the same set of SATA spindles (4+1 R5). All LUNs containing VMs are now in a state of error, even though I am VERY sure no other workload is active.

    I suspect SIOC is not as smart as we were all hoping for: There is no solid link between spindle sets and luns carved out of them anywhere, neither does SIOC seem to try to figure that out for itself.

    I suspect SIOC simply looks on a per-LUN basis. So even if a second vmfs LUN on a set of spindles raises latency on the first vmfs LUN, SIOC will act with the “external workload detected” alarm. Shame, because you hardly ever link a set of spindles to just one single LUN, which seems the only place SIOC should be enabled to me (and NOT have it replicated as well which thins the abilities even more).

    Please correct me if I’m wrong. If not, we all need to work on getting storage and virtualization even closer together so that SIOC might be able to relate luns to spindles automagically.

    Come to talk of it, that would be a nice wannahave anyway; I’d like to see storage maps in vCenter that links VMs to spindlesets, not VMFS volumes. SIOC could ride on that feature as well 🙂

  9. Erik,
    In general, enabling SIOC on all datastores on the same spindle set should reduce the trigger of the alarm. This is because all of them would react to the increase in latency and SIOC will do the throttling on all of them. If the latency decreases with the decrease in queue depth, no alarm should be raised. This is true in theory but in practice we have seen some situations where due to array level IO scheduling, caching, prefetching etc., two LUNs on the same set of spindles may observe somewhat different latencies. For example 30 ms and 36 ms in one of the cases. Now due to such behavior, the LUN with higher latency might do more throttling temporarily, but the latency is kept high on the LUN due to overall contention and sharing. Such cases lead to the trigger of alarm.

    So although enabling SIOC on all LUNs should reduce the probability of the alarm drastically ( to zero in theory) it doesn’t go to zero (in practice). SIOC is based on distributed latency measurements and doesn’t have the knowledge of underlying storage architecture.
    You are right that SIOC would definitely benefit from such information.
    For future releases, we are planning to make SIOC smarter and external workload detection, even more robust in presence of such array level behavior. Would be glad to share more details about it later this year. Thanks for testing it out and posting the results.

  11. I’ve been doing a bit of work with the product manager for SIOC. There are some changes planned to the way that the alarms work to better communicate what is going on. When I first came across this problem I thought we’d somehow mounted a VMFS datastore to another system and were in danger of data corruption. This was an iSCSI environment, so misconfigurations can be easy. It helped us highlight some issues with the network configuration and storage configuration. We needed to upgrade firmware and drivers on the NIC’s. After that, the storage replication was still impacting it. But hasn’t cause us major problems.

  12. I think that SIOC should be more API-aware otherwise we will be seeing that “external load” message all the time…

      • Actually all VMware API’s, I was starting to think VADP, VAAI, but any API that could be leveraged to impact disk.
        Thereby reducing unknown storage issues/messages.

  13. Hi All,

    i had this “Non-VI Workload” sometimes happening and it took me a while to find out that the problem in my case was pathtrashing on an Active/Passive Array. After i found the Host responsible the “Non-VI Workload” was gone.

    regards

    Harry

    • That’s interesting. Thanks for sharing.

      We’ve seen this from some field reports a well where customers were discovering real problems in their setups only because of the event generated by SIOC that they otherwise would not have found.

  14. According to the vmware article it specifies that datastores should be managed by the same vCenter but it does not specify at the cluster level. Let’s say I have 3 clusters using datastores from the same disks (not all cluster’s hosts see the same datastores). Would it still complain or since the clusters are all managed by the same vCenter that it would know the I/o?

  16. As far as I understood RDM disks are not supported by SIOC, and particularly RDM disks for MS Failover clusters. I guess it means that all IOs executed on RDM disks will be considered as “External I/O workload” and alarm will be triggered.

  17. We noticed the same issue as Garret, we have 3 clusters accessing the same LUNS and the SIOC alerts are present across nearly all VMFS datastores.

    A comment on spindles being shares across multiple LUNS, Arrays etc, in reality this may be an efficient use of space but if you need the best performance for a particular type of workload sharing spindles would not be your best solution. If your looking for best utilization of your storage resources, maximize your arrays with the most disks it can handle, the more heads reading and writing the data the better your performance will be but be careful to ensure that you have load balancing setup on your multi-pathing design so as to not saturate any one target. Although ESX is limited to 2TB RAW LUNs this can be overcome with extents when creating your VMFS datastores.

  18. I’ve enabled SIOC on _all_ datastores of our 12 node P4500 Lefthand SAN (vSphere 4.1, 6 ESXi hosts). And get the warning “An external I/O workload is detected on datastore …” every 2-3 minutes for each datastore. No other devices are accessing the LUNs, the iSCIS network is isolated, the SIOC threshold is 30 ms everywhere, there are also no extends. Any idea how to check what might the underlying issue?

  19. That is because Lefthand shares spindles between LUNs so “latency” could be caused by VMs on a different datastore. I would suggest disabling the alarm.

  20. Thanks for your quick answer. I thought this would be handelt by “Enable SIOC on all datastores that are backed by the same set of spindles”. I’ve disabled the alarm already but the events are flooding the log file (Splunk tells me that there have been 4,071 entries in the last 3 hours) which is a bit anoying. But I guess I’ve to live with that.

  21. Hi Duncan,
    Has SIOC been improved on Vsphere 5.1? If the throttling happens at the host level NFS client and that host(s) have also VMs that live on other Datastores that are not suffering from high latencies, would those VMs be affected by the host throttling?

  22. This alarm is occasionally happening in our environment, but the only connected hosts are VMware. However, the LUNs are carved from an EMC Storage pool (with auto tiering enabled). I haven’t been able to pin down the alarms to the time when the AT move process happens, so I’m wondering if the fact it is on a storage pool could be the cause.

  23. Also experiencing this alarm due to reaching our storage controller limits.
    Apparently with a 1000VM’s, whenever Symantec was updating definitions it was starting a quick smart scan. This saturated the controller and all LUN’s got this message at once.