Properly assessing the workload requirements is key to good storage design. This process should be reviewed continuously as requirements change and the infrastructure expands – I/O characteristics are rarely static. In relation to performance a good storage design should account for growth in workload and throughput. In this post we will cover some of the potential areas affecting performance and look at common solutions.
Principle Areas Affecting SAN Performance
- Failure, to correctly assess and understand the workload characteristics and requirements (IO, Bandwidth, IO Profiles – Random vs Sequential)
- Subsystem bottlenecks such as choice of physical disks (whether you have enough back-end IOPs to support workloads).
- Drive response times, this is defined as the time it takes for a disk to execute an I/O request.
- Nearline (NL) provide less I/O performance than 10/15K SAS disks.
- Response time = (queue length + 1) x the service time to complete task.
- Slow array response times might be caused by a number of issues some include:
- SAN fabric fan-in ratios / increasing contention causing throughput bottlenecks.
- Front-end cache saturation (memory storing I/Os before they are de-staged to the backend drives).
- Heavy workloads
- Drive response times, this is defined as the time it takes for a disk to execute an I/O request.
- Large workload I/O sizes can saturate interfaces impacting performance.
- To identify VM I/O sizes use the vscsiStats tool, Cormac Hogan explains it nicely.
- Poor change control mechanisms and workload awareness (Beware of unnecessary operations being performed on the array. New SAN – “Wouldn’t it be cool if we run IOmeter @100% random ops to see what this thing can really do!” – not if its in production.
- I/O path saturation, incorrectly configured path selection policies.
- Poor choice of RAID configuration for the supporting workloads. There is a penalty for every write operation performed, this varies depending on RAID policy (see below). Note that RAID rebuilds can impact performance and depending on the size of the disk can take a long time to complete.
| RAID | I/O Penalty |
| 1 | 2 |
| 5 | 4 |
| 6 | 6 |
| 10 | 2 |
| RAID-DP (NetApp) | 2 |
Fabric Switching, IP & FC Impact on Performance FC Fabric Check physical switches for:
- CRC errors, indicating code violations within FC frames.
- Potential causes are: Faulty SFP’s or damaged cables. The impact is additional CPU overhead caused by retransmissions.
- Loss of Sync – indicative of multiple code violations (character corruption), due to incompatible speeds between initiators and targets (don’t forget your ISL’s).
- Loss of Sync errors, can also be caused by faulty SFP’s or HBA’s.
- Class 3 discards, no acknowledgement from the receiver, FC frames can be discarded due to port congestion (check ISL oversubscription).
- Exchange completion times, An ‘exchange’ is made up of [frames] & [sequences]. Frames include information about source and destination ID’s, originating exchange and sequence ID’s. Often a true measure of performance in switched fabric environments is how long each exchange takes to complete (specifically reads/write operations). Its important to note that latency here, cascades up the stack ultimately impacting the application.
- Issues affecting exchange rates: HBA’s (check firmware), physical server, number of hops, disk speeds, interfaces, configuration issues, I/O transaction sizes (Ref I/O sizes : see Brocade doc on buffer credit management).
- Check Fan-in/Fan-out ratios, ensure there is enough bandwidth available – see Class 3 discards (above).
IP Fabric
- The switch should be able to process and forward a continuous stream of data at full wire speed (1 Gbps) on all ports simultaneously, also known as ‘non-blocking’.
- Adequate port buffering, this is working space for inbound (ingress) and outbound (egress). There are two different modes: 1) Shared port buffering, 2) Dedicated port buffering. Shared port buffering dynamically allocates memory to to ports as needed, where dedicated port buffering has a fixed memory amount per port.
- Best practise: Disable spanning tree on initiator and target ports. If you have to use STP enable rapid spanning tree to reduce the convergence time and set port to immediately forward frames.
- Check switch vendor recommendations regarding flow control (preventing buffer overflow, reducing retransmissions).
- Check switch vendor recommendations for storm control.
- Stacking / Trunking, check ISL is not oversubscribed, be aware of backplane oversubscription.
- Both IP and FC storage traffic can be susceptible to congestion, however when an iSCSI path is overloaded, the TCP/IP protocol drops packets and requires them to be resent. This is typically caused by oversubscription of paths or low port buffer.
- Minimise the amount of hops between initiators and targets, traffic should not be routed and sit on the same subnet.
Lastly, if your switches have the capability use performance counters to identify times of peak workloads, you may be able to cross reference this information with other latency alarms setup across your datacentre (See VMware LogInsight).
Calculating functional IOPs of the array
This is important as it will give you a good idea of the performance capabilities taking into account the RAID write penalty.
First we need to understand the Raw IOPs of the array, this is calculated as drive unit IOPs x n (total amount of drives in the array).
Second we need understand the I/O cross section – Read% vs Write%. This information can be obtained from your SAN management tools or looking at throughput for reads & writes.
Formula: Total Reads (KBps) + Total Writes (KBps) = Total Throughput in KBps
340,000KBps + 100,000KBps = 440,000 KBps 340,000 / 440,000 = 77.27 % Read 100,000 / 440,000 = 22.72 % Write
We can then use the this information to determine the functional IOPs of the array, this value is key in assessing whether or not the SAN is up to the job.
Functional IOPs = (Raw IOPs x Read%) + ((Raw IOPs x Write%) / RAID Write Penalty)
Calculating Throughput Requirements
MBps = (Peak VM Workload IOPS * KB per IO) /1024 (use the VMware vscsiStats tool to output workload IO sizes)
MBps = (2000 * 8) /1024 = 15.625MBps
In the above example where the I/O workload requirement is 2000 IOPS, we would need 16MBps (128 Mbp/s @ 8KB per IO) of throughput to satisfy that requirement.
Note: esxtop can be used to determine SAN performance problems impacting hosts. Here are a couple of latency counters that should be monitored closely:
| Value | Description |
| CMDS/s | This is the total amount of commands per second and includes IOPS (Input/Output Operations Per Second) and other SCSI commands such as SCSI reservations, locks, vendor string requests, unit attention commands etc. being sent to or coming from the device or virtual machine being monitored. In most cases CMDS/s = IOPS unless there are a lot of metadata operations (such as SCSI reservations) |
| DAVG/cmd | This is the average response time in milliseconds per command being sent to the deviceWarning threshold = 25High numbers (greater than 15-20ms) represent a slow or over worked array. |
| KAVG/cmd | This is the amount of time the command spends in the Vmkernel.Warning threshold = 2 High numbers (greater than 2ms) represent either an overworked array or an overworked host. |
| GAVG/cmd | This is the response time as it is perceived by the guest operating system. This number is calculated with the formula: DAVG + KAVG = GAVGWarning threshold = 2 |
Calculating I/O Size and I/O Profile : Random vs Sequential
See VMware vscsiStats tool / Dtrace for linux, SQLIO Windows.
ESX host monitoring
Have a look at the following VMware KB – Using esxtop to identify storage performance issues for ESX / ESXi (multiple versions) (1008205)
Virtual Machine Monitoring
- Setup latency alarms within VMware vCentre to monitor virtual machine total disk latency.
- The default is 75ms however, this should be adjusted depending on your KPI’s (<=25ms).
Host SCSI Latency
- To reduce latency on the host, ensure that the sum of active commands from all virtual machines does not consistently exceed the LUN queue depth.
- Either increase the LUN queue depth or move virtual machines to another LUN.
- Observe vendor best practices for adjusting queue length on HBAs.
- Work with the storage teams to ensure that each LUN is comprised of multiple physical disks.
- To reduce latency on the array, reduce the maximum number of outstanding I/O commands to the shared LUN.
- Distribute workloads across datastores.
- Reduce VM to datastore ratios / reduce logical contention.
- The maximum number of outstanding I/O commands that the array is capable of handling varies by array configuration, see my post on storage design considerations: Link
SCSI Reservations A SCSI reservation is a lock operation on the LUN preventing I/O from other operations. Consistent lock operations add latency measured in milliseconds.
- Operations performed which can create locking metadata:
- Creating a VMFS datastore
- Expanding a VMFS datastore onto additional extents
- Powering on a virtual machine
- Acquiring a lock on a file
- Creating or deleting a file
- Creating a template
- Deploying a virtual machine from a template
- Creating a new virtual machine
- Migrating a virtual machine with vMotion
- Growing a file, for example, a snapshot file or a thin provisioned virtual disk
- For the zeroed thick type of virtual disk the reservation is required only when zeroing the blocks.
In vSphere 4.x VMware released VAAI – VMware API for Array Integration, these primitives included an ATS – Atomic Test & Set which reduces locking.
- For compatibility, check the VMware HCL/SAN Vendor –
From the hosts the following commands can be used to check the internal status ESX4.x
# esxcfg-scsidevs -l | egrep &quot;Display Name:|VAAI Status:&quot;
ESX5.x
# esxcli storage core device vaai status get
Note: You may need to install the binaries on the ESX host, if these have not been included. The binaries come in the form of VIBs, a reboot of the host will be required after installation.
VAAI changes are logged in the VMKernel logs at /var/log/vmkernel (for ESXi 5.0 vmkernel.log) or /var/log/messages).
VMware also provide a great KB for troubleshooting SCSI reversions conflicts on VMware Infrastructure VMware KB: 1005009 VMware KB enabling and troubleshooting VAAI : KB 1021976
Possible Solutions
Correctly plan and identify the workload characteristics and requirements:
- Analyse total throughput, identify I/O profiles (read versus write percentages), I/O types (sequential vs random) and particularly I/O size.
- Understand the workload characteristics, In most cases database workloads are inherently more I/O intensive than say the same number of web servers.
- To reduce contention and latency, avoid oversubscribing by:
- Spreading workloads across different LUNs (use VMware SDRS to automate balance workload across datastores).
- Identify front-end bottlenecks, balance I/O to different storage processors and storage processor ports.
- Use different FC or Ethernet paths.
- Use multiple ESXi storage HBAs and HBA Ports.
- Set appropriate pathing policies based on storage vendor best practises.
- Sometimes poor storage performance can be attributed to bad host and VM management, check VM disk alignment, identify if VM’s are swapping consistently to your SAN , attributing to unnecessary I/O across your storage fabric, use local server-side storage to home VM SWAP if possible.
- In the case of the latter reduce memory overcommit or increase physical RAM/Hosts.
- Set appropriate queue depth values on HBA adapters, follow vendor recommendations. Observe impact to consolidation ratios specifically the number of VMs in a VMFS datastore. Setting queue depths too high can have a negative impact on performance (very large queue depths 64+, tend to just mask issues and are indicative of a larger performance problem).
- Distributing workloads with a large number of transactions per second, you can also reduce latency by reducing the number of hops in the storage path.
- Increase I/O front-end ports if possible (some SAN devices support the installation of additional HBA cards).
- Check if target and initiator ports have negotiated correctly (check port properties on SAN fabric switches).
- For NFS/iSCSI – Use, good quality switches and preferably dedicated stacks (1GbE – Dell PowerConnect 6200’s, Cisco 3750’s), if you are using 10GbE use NIOC to prevent other I/O operations from impacting your storage I/O.
- Not directly related to performance but ensure proper L2 isolation between iSCSI traffic and general network traffic.
- Calculate I/O cost to support the workload, factoring RAID write penalty.
- Increase the number of drives to meet requirements.
- Upgrade controller/SP cache, however in the event cache is saturated you will be reliant on back-end performance of the underlying drives, front-end cache is not a fix for poor back-end storage design.
- To prevent noisy neighbours using all available front-end cache look at array side tools to create I/O polices for different workloads, prioritise those that are business critical.
- VMware Storage IO Control (SIOC) can be used when latency impacts VM workloads by prioritising IO (only invoked during periods of contention).
- Investigate potential use of SSD’s to absorb intensive I/O operations (Server side cache may also help – See PernixData FVP for VMs).
- Investigate sub-lun tiering mechanisms to move hot blocks of data to drives with faster performance characteristics and less used blocks to slower storage (EMC FAST, HP 3PAR AO).
- Use array multipathing policies, either native or thirdparty such as EMC powerpath. These policies can help by distributing I/O across all available storage paths in a more effective manner.
- I/O Analyzer (VMware fling) can measure storage performance in a virtual environment and to help diagnose storage performance concerns. I/O Analyzer, supplied as an easy-to-deploy virtual appliance.
- VMware Log Insight can aggregate and perform deep analysis of system logs identifying trends in many metrics in a fully customisable package. This is particularly useful when investigating storage related problems. http://www.vmware.com/products/vcenter-log-insight/
Recommended Reading
VMware KB / Troubleshooting Storage Performance Issues with VMware Products
Brocade FOS Admin Guide / Buffer Credit Management
/> vi Kernel 