VCDX5-DCV – Resources

The journey to achieving the VCDX certification is multifaceted, requiring highly developed skills in several areas. To give guidance and structure to my approach, I am compiling a reference library in addition to some of the great resources provided by the VMware community.

Overview

VCDX Community Resources

VCDX Holder Blog Resources (tagged “VCDX” search)

Reference Documents:

Books:  Most of these i’ve read a few times over and a few which I have only covered lightly. There is only so much you can store mentally from just reading. My personal study method is to: Read, comprehend, write, evaluate, lab (if required). The more I can do of this the better. Its a fairly logical approach which has helped me in the past.

  • vSphere 5.1 Clustering Deepdive – Duncan Epping and Frank Denneman
  • VMware vSphere Security Cookbook – Mike Greer
  • VMware vCloud Security – Prasenjit Sarkar
  • VMware vSphere Design SE – Scott Lowe, Forbes Guthrie and Kendrik Coleman
  • The Art of Network Architecture: Business Driven Design – Russ White and Denise Donohue
  • Virtualising Microsoft Business Critical Applications on VMware vSphere – Matt Liebowitz and Alexander Fontana
  • Networking for VMware Administrators – Chris Wahl and Steve Pantol
  • VMware vCloud Architecture Toolkit (vCAT) VMware Press : Technical and Operational Guidance for Cloud Success : vCAT Team
  • Troubleshooting vSphere Storage – Mike Preston
  • VMware vSphere Resource Management – Jonathan Frappier
  • Managing and Optimising VMware vSphere – Sean Crookston and Harley Stagner
  • Storage Implementation in VSphere 5.0 – Mostafa Khalil
  • Data Center Virtualization Fundamentals: Understanding Techniques and Designs for Highly Efficient Data Centers with Cisco Nexus, UCS, MDS, and Beyond
  • VMware vSphere Design Best Practises – Brian Bolander
  • Virtualizing SQL Server with VMware: Doing it Right (Vmware Press Technology) – Michael Corey, Jeff Szastak, Michael Webster
  • VMware vSphere 5, Building a Virtual Datacenter: Integration into the Datacenter – Eric Mallie, Rene-Francois Menneceir
  • Mastering vSphere 5.5 – Scott D Lowe, Nick Marshall, Forbes Guthrie, Matt Liebowitz and Josh Artwell

VCAPs completed….what’s next ?

There is value in trying to achieve the tasks we find the hardest – it’s through the process we learn and improve. For me this is what achieving the VCDX is all about.

So why do I want to pursue it ?

A good question, and one that every candidate should ask of themselves. At first this was driven by my need to be one of those select few, the industry experts of which there are less than 200 world-wide. As I started on the VMware certification ladder towards preparing for the VCDX, from getting the VCP and then moving on towards the more advanced VCAP exams, my objective shifted. It was simply to be the best I could possibly be. Towards the end of 2014, this culminated in me transitioning to a solutions focussed company, with employees who had achieved their VCDX or were striving towards the same goals.

Ultimately I see this as an investment in myself – there is intrinsic value in the process, it is not purely about the end result for me.

The Impact

Many have said it – the impact to your personal life and family should not go without thought. This is not just in the writing of your defence paper but also in qualifying exams up until you start the VCDX process. I have a very supportive family, led by the most amazing wife anyone could ask for, and is the true measure behind my success.

Moving Forward

The community has never been stronger or more prepared to support those pursuing their VCP, VCAPs and VCDX.
I am currently formulating my defence, for which my design contributes a large part of my ongoing learning process.

Good luck to all those starting and those chosen to defend this year.

TOGAF study resources

The Open Group Architecture Framework (TOGAF) provides process and procedures for the design, implementation and governance of enterprise architecture. The framework can be used as a set of building blocks to develop an enterprise architecture for use within an organisation.

TOGAF is developed and maintained by members of The Open Group, working within the Architecture Forum. I plan to use elements of the framework to supplement and rationalise my design theory, which I hope will aid me with my VCDX defence.

Resources:
An introduction to TOGAF version 9.1 – Guidelines, Architecture Development Method (ADM), content framework & reference models.
TOGAF 9 Components 
TOGAF 9 Template Deliverables, Set 1
TOGAF 9 Template Artifacts and Deliverables, Set 2
TOGAF Publications, webinars and whitepapers
IT Architecture and Design Patterns/Assets: An Initial Assessment
TOGAF 9 Certification Program
TOGAF Benefits
Enterprise Architecture Standards

Books:
TOGAF 9 Foundation Study Guide

Additional Content:
The Open Group IT4IT™ Reference Architecture, Version 1.3
Service Orientated Reference Architecture
Study Information Landing Page
Hitchhikers guide to TOGAF exam preparation

Infrastructure Design & Project Framework

Successfully planning, designing and implementing a virtualisation project can be a very rewarding experience. Whether you are working alone or in a team you may find the task initially daunting, be unsure of where to start or lack the appropriate framework from which to work from. Hopefully this information will support you, if you have been given the task or have successfully completed a virtualization project, but want to identify ways to make the next upgrade or implementation more efficient.

Infrastructure design is a deep subject with many facets and interlinking dependencies of design choices. The four pillars, referred to as compute (see my compute design post), storage (see my storage design post), networking and management, can be very complex to integrate successfully, when considering all the options. A great deal of emphasis should be placed on understanding design decisions, as poor planning can lead to additional costs, the project not meeting organisation goals and ultimately a failure to deliver. Through each part of the design process it is important that you validate design decisions against requirements identified through the information gathering process.

Furthermore, design decisions should be continually evaluated against infrastructure qualities such as availability, manageability, performance, recoverability and security.

Project Framework

Use the following key points/stages to plan and build your project:

1. Information Gathering
2. Current State, Future State and Gap Analysis
3. Conceptual, Logical & Physical Design Process
4. Migration and Implementation Planning
5. Functional Testing / Quality Assurance
6. Continuous Improvement
7. Monitoring Performance, Availability and Capacity

1. Information Gathering: Information should be gathered from stakeholders / C-level executives, application owners and subject matter experts to define and identify:

  • The Project scope / project boundaries, for example, upgrade the VMware vSphere infrastructure at the organisations central European offices only.
  • Project goals, what is it the organisation wants to achieve? For example reduce physical server footprint by 25% before the end of the financial year.
  • Service Level Agreements (SLA), Service Level Objectives (SLO), Recovery Time Objectives (RTO), Recovery Point Objectives (RPO) : [Maximum Tolerable Downtime MTD].
  • Key Performance Indicators (KPI), relating to application response times.
  • Any requirements, both functional and non-functional i.e regulatory compliance – HIPAA, SOX, PCI etc. Understand the impact on the design required to meet HIPAA compliancy (a US standard, but acknowledged under EU-ISO/IEC 13335-1:2004 information protection guidelines), which states that data, communication must be encrypted (HTTPs, SSL, IPSEC, SSH). A functional requirement specifies something the design must do for example support 5000 virtual machines, whereas a non-functional requirement specifies how the system should behave, For example: Workloads deemed as business critical must not be subject to resource starvation (CPU, Mem, Network, Disk) and must be protected using appropriate mechanisms.
  • Constraints:  Limit design choices based on data consolidation from the information gathering exercise. An example could be that you need to use the organisations existing NFS storage solution. A functional requirement may be that the intended workload you need to virtualize is MS Exchange. Currently virtualising MS Exchange on NFS is not supported – if the customer had a requirement to virtualise MS Exchange but only had an NFS-based storage solution, the proposal would lead to an unsupported configuration. Replacing the storage solution may not be feasible and out of scope due to financial reasons.
  • Risks: Put simply are defined by the probability of a threat, the vulnerability of an asset to that threat, and the impact it would have if it occurred. Risks throughout the project must therefore be recorded and mitigated, regardless of which aspect of the project they apply to.  An example risk, a project aimed at a datacentre that doesn’t have enough capacity to meet the anticipated infrastructure requirements. The datacentre facilities team is working on adding additional power but due to planning issues may not be able to meet the expected deadlines set by the customer. This risk would therefore need to be mitigated and documented to minimise / remove the chance of it occurring.
  • Assumptions: The identification or classification of a design feature without validation. For example: In a multi-site proposal the bandwidth requirements for datastore replication is sufficient to support the stated recovery time objectives. If the site link has existing responsibilities how will the inclusion of additional replication traffic affect existing operations? During the design phase you may identify additional assumptions each of which must be documented and validated before proceeding.

 2. Current state, Future state and Gap Analysis:

  • Identifying the current state can be done by conducting an audit of the existing infrastructure, obtaining infrastructure diagrams, system documentation, holding workshops with SME’s and application owners.
  • A future state analysis is performed after the current state analysis and typically outlines where the organization will be at the end of the projects lifecycle.
  • A gap analysis outlines how the project will move from the current state to the future state and more importantly, what is needed by the organization to get there.

3. Conceptual, Logical & Physical Design Process:

  • A conceptual design identifies how the solution is intended to achieve its goals either through text, graphical block diagrams or both.
  • A logical design must focus on the relationships between the infrastructure components – typically this does not contain any vendor names, physical details such as amount of storage or compute capacity available.
  • A physical design shows a detailed description of what solutions have been implemented in order to achieve the project goals. For example: How the intended host design would mitigate against a single point of failure.
  • Get stakeholder approval on design decisions before moving to the implementation phase. Throughout the design process you should continually evaluate design decisions against the goal requirements and the infrastructure qualities (Availability, Manageability, Performance, Recoverability, Security).
    • Availability: Typically concerned with uptime and calculated as a percentage based on the organisations service level agreements (SLA). The key point is mitigating against single point of failure across all components. Your aim is to build resiliency into your design. Availability is calculated as a percentage or 9s value : [Availability % = ((minutes in a year – average annual downtime in minutes) / minutes in a year) × 100].
    • Manageability: Concerned with the operating expenditure of the proposed solution or object. How well will the solution scale, manage, implement, upgrade, patch etc..
    • Performance: How will the system deliver required performance metrics, typically aligned to the organisations KPIs and focus on workload requirements: response times, latency etc..
    • Recoverability: RTO/RPO | MTD : Recovery Time Objective: Time frames associated with service recovery. Recovery Point Objective: How much data loss is acceptable? Maximum Tolerable Downtime: A value derived from the business which defines the total amount of time that a business process can be disrupted without causing any unacceptable consequences.
    • Security: Compliance, access control. How best can you you protect the asset, workload, from intruders or DOS attacks. More importantly what are the consequences/risks of your design decisions.

4. Migration and Implementation Planning:

  • Identify low risk virtualisation targets and proceed with migrating these within the organisation first. This is beneficial in achieving early ROI, build confidence and assist other operational aspects of future workload migrations.
  • Work with application owners to create milestones and migration schedules.
  • Arrange downtime outside of peak operating hours, ensure you have upto date and fully documented rollback and recovery procedures.
  • Do not simply accept and adopt best practises; understand why they are required and their impact on the design.

Additional Guidelines: 

  • Create service dependency mappings: These are used to identify the impact of something unexpected and how best to protect the workload in the event of disaster. DNS for example plays an important role in any infrastructure – if this was provided through MS Active Directory in an all virtualised environment, what impact would the failure of this have on your applications, end users, external customers? How can you best mitigate the risks of this failing?
  • Plan for performance then capacity: If you base your design decisions on capacity you may find that as the infrastructure grows you start experiencing performance related issues.  This is primarily attributed to poor storage design, having insufficient drives to meet the combined workload I/O requirements.
  • Analyse workload performance and include capacity planning percentage to account for growth.
  • What are the types of workloads to be virtualised – Oracle, SQL, Java etc.  Ensure you understand and follow best practices for virtualized environments – reviewing and challenging where appropriate. Oracle for example has very strict guidelines on what they deem as cluster boundaries and can impact your Oracle licensing agreement.
  • Don’t assume something cannot be virtualised due to an assumed issue.
  • Benchmarking applications before they are virtualised can be valuable in determining a configuration issue post virtualisation.
  • When virtualising new applications check with the application vendor regarding any virtualisation recommendations. Be mindful of oversubscribing resources to workloads that won’t necessarily benefit from it. “Right sizing” virtual machines is an important part of your virtualisation project. This can be challenging as application vendors set specific requirements around CPU and memory.
    • For existing applications be aware of oversized virtual machines and adjust resources based on actual usage.
  • What mechanisms will you use to guarantee predicable levels of performance during periods of contention? See vSphere NIOC, SIOC.
  • VARS/Partners may be able to provide the necessary tools to assess current workloads, examples of which include VMware Capacity Planner (can capture performance information for Windows/Linux), IOStat, Windows Perfmon, vscsiStats,  vRealize Operations Manager…

5. Functional Testing / Quality Assurance :

This is a very important part of your design as it allows you to validate your configuration decisions. Also ensuring configurational aspects of the design are implemented as documented. This stage is also used to ensure the design meets both functional and non-fuctional requirements. Essentially the process maps the expected outcome against actual results.

  • Functional Testing is concerned with exercising core component function. For example, can the VM/Workload run on the proposed infrastructure.
  • Non-functional testing is concerned with exercising application functionality using a combination of invalid inputs, some unexpected operating conditions and by some other “out-of-bound” scenarios. This test is designed to evaluate the readiness of a system according to several criteria not covered by functional testing. Test examples include; vSphere HA, FT, vMotion, Performance
, security…

6. Continuous Improvement:

The ITIL framework is aimed at maximising the ability of IT to provide services that are cost effective and meet the expectations and requirements of the organisation and customers. This is therefore supported by streamlining service delivery and supporting processes by developing and documenting repeatable procedures. The ITIL Framework CSI (Continual Service Improvement) provides a simple seven-step process to follow.

Stage 1: Define what you should measure
Stage 2: Define what you currently measure
Stage 3: Gather the data
Stage 4: Processing of the data
Stage 5: Analysis of the data
Stage 6: Presentation of the information
Stage 7: Implementation of corrective action

  • Workloads rarely remain static. The virtualised environment will need constant assessment to ensure service levels are met and KPIs are being achieved. You may have to adjust memory and CPU as application requirements increase or decrease. Monitoring is an important part in the process and can help you identify areas which need attention. Use built-in alarms to identify latency in storage and vCPU ready times, which can be easily set to alert you to an issue.
  • Establish a patching procedure  (Host, vApps, VMs, Appliances, 3rd party extensible devices).
  • Use vSphere Update Manager to upgrade hosts, vmtools, virtual appliances. This goes deeper than just the hypervisor – ensure storage devices, switches, HBA, firmware are kept up-to-date and inline with vendor guidelines.
  • Support proactive performance adjustments and tuning, analyse issues : determine the root cause, plan corrective action, remediate then re-assess.
  • Document troubleshooting procedures.
  • Use automation to reduce operational overheads.
  • Maintain a database of configuration items (these are components that make up the infrastructure), their status, lifecycle, support plan, relationships and which department assumes responsibility for them when something goes wrong.

7. Monitoring Performance Availability and Capacity:

  • Ensure the optimal and cost effective use of the IT infrastructure to meet the current and future business needs. Match resources to workloads that require a specific level of service. Locate business critical workloads on datastores backed by tier 1 replicated volumes on infrastructure that mitigates against single point of failure.
  • Make use of built-in tools for infrastructure monitoring and have a process for managing / monitoring service levels.
  • Monitor not only the virtual machines but the underlying infrastructure, using built-in tools already mentioned above, to monitor latency.
  • Performance and capacity reports should include, hosts / clusters, datastores and resource pools.
  • Monitor and report on usage trends at all levels, compute, storage and networking.
  • Scripts for monitoring environment health (see Alan Renouf’s vCheck script).
  • A comprehensive capacity plan uses information gathered from day-to-day tuning of VMware performance, current demand, modeling and application sizing (future demand).

Additional Service Management Tasks:

  • Integrate the virtual infrastructure into your configuration and change management procedures.
  • Ensure staff are trained to support the infrastructure – investment here is key in ensuring a) staff are not frustrated supporting an environment they don’t understand and b) the business gets the most out of their investment.
  • Develop and schedule maintenance plans to ensure the environment can be updated and is running optimally.
  • Plan and perform daily, weekly, monthly maintenance tasks.  For example, search for unconsolidated snapshots, review VMFS volumes for space in use and available capacity (anything less than 10% available space should be reviewed). Check logical drive space on hosts. Check that any temporary vms can they be turned off or deleted. Monthly maintenance tasks, create a capacity report for the environment and distribute to IT and management. Update your VM templates, review the vmware website for patches, vulnerabilities and bug fixes.

Reference Documentation:

Conceptual Logical Physical It is Simple, by John A Zackman
Leveraging ITIL to Manage Your Virtual Environment, by Laurent Mandorla, Fredrik Hallgårde, BearingPoint, Inc.
Performance Best Practises for VMware vSphere
ITIL v3 Framework, Service Management Guide
Control Objectives for Information and Related Technology (COBIT) framework by ISACA
Oracle Databases on VMware vSphere Best Practise Guide
VMware vSphere Monitoring Performance Guide

Troubleshooting Storage Performance

Properly assessing the workload requirements is key to good storage design. This process should be reviewed continuously as requirements change and the infrastructure expands – I/O characteristics are rarely static. In relation to performance a good storage design should account for growth in workload and throughput. In this post we will cover some of the potential areas affecting performance and look at common solutions.

Principle Areas Affecting SAN Performance

  • Failure, to correctly assess and understand the workload characteristics and requirements (IO, Bandwidth, IO Profiles – Random vs Sequential)
  • Subsystem bottlenecks such as choice of physical disks (whether you have enough back-end IOPs to support workloads).
    • Drive response times, this is defined as the time it takes for a disk to execute an I/O request.
      • Nearline (NL) provide less I/O performance than 10/15K SAS disks.
      • Response time = (queue length + 1) x the service time to complete task.
    • Slow array  response times might be caused by a number of issues some include:
      • SAN fabric fan-in ratios / increasing contention causing throughput bottlenecks.
      • Front-end cache saturation (memory storing I/Os before they are de-staged to the backend drives).
      • Heavy workloads
  • Large workload I/O sizes can saturate interfaces impacting performance.
    • To identify VM I/O sizes use the vscsiStats tool, Cormac Hogan explains it nicely.
  • Poor change control mechanisms and workload awareness (Beware of unnecessary operations being performed on the array. New SAN – “Wouldn’t it be cool if we run IOmeter @100% random ops to see what this thing can really do!” – not if its in production.
  • I/O path saturation, incorrectly configured path selection policies.
  • Poor choice of RAID configuration for the supporting workloads. There is a penalty for every write operation performed, this varies depending on RAID policy (see below). Note that RAID rebuilds can impact performance and depending on the size of the disk can take a long time to complete.
RAID I/O Penalty
1 2
5 4
6 6
10 2
RAID-DP (NetApp) 2

Fabric Switching, IP & FC Impact on Performance FC Fabric Check physical switches for:

  • CRC errors, indicating code violations within FC frames.
    • Potential causes are: Faulty SFP’s or damaged cables. The impact is additional CPU overhead caused by retransmissions.
  • Loss of Sync – indicative of multiple code violations (character corruption), due to incompatible speeds between initiators and targets (don’t forget your ISL’s).
    • Loss of Sync errors, can also be caused by faulty SFP’s or HBA’s.
  • Class 3 discards, no acknowledgement from the receiver, FC frames can be discarded due to port congestion (check ISL oversubscription).
  • Exchange completion times, An ‘exchange’ is made up of [frames] & [sequences]. Frames include information about source and destination ID’s, originating exchange and sequence ID’s. Often a true measure of performance in switched fabric environments is how long each exchange takes to complete (specifically reads/write operations). Its important to note that latency here, cascades up the stack ultimately impacting the application.
    •  Issues affecting exchange rates: HBA’s (check firmware), physical server, number of hops, disk speeds, interfaces, configuration issues, I/O transaction sizes (Ref I/O sizes : see Brocade doc on buffer credit management).
  • Check Fan-in/Fan-out ratios, ensure there is enough bandwidth available – see Class 3 discards (above).

IP Fabric

  • The switch should be able to process and forward a continuous stream of data at full 
wire speed (1 Gbps) on all ports simultaneously, also known as ‘non-blocking’.
  • Adequate port buffering, this is working space for inbound (ingress) and outbound (egress). There are two different modes: 1) Shared port buffering, 2) Dedicated port buffering. Shared port buffering dynamically allocates memory to to ports as needed, where dedicated port buffering has a fixed memory amount per port.
  • Best practise: Disable spanning tree on initiator and target ports. If you have to use STP enable rapid spanning tree to reduce the convergence time and set port to immediately forward frames.
  • Check switch vendor recommendations regarding flow control (preventing buffer overflow, reducing retransmissions).
  • Check switch vendor recommendations for storm control.
  • Stacking / Trunking, check ISL is not oversubscribed, be aware of backplane oversubscription.
  • Both IP and FC storage traffic can be susceptible to congestion, however when an iSCSI path is overloaded, the TCP/IP protocol drops packets and requires them to be resent. This is typically caused by oversubscription of paths or low port buffer.
  • Minimise the amount of hops between initiators and targets, traffic should not be routed and sit on the same subnet.

Lastly, if your switches have the capability use performance counters to identify times of peak workloads, you may be able to cross reference this information with other latency alarms setup across your datacentre (See VMware LogInsight).

Calculating functional IOPs of the array

This is important as it will give you a good idea of the performance capabilities taking into account the RAID write penalty.

First we need to understand the Raw IOPs of the array, this is calculated as drive unit IOPs x n (total amount of drives in the array).

Second we need understand the I/O cross section – Read% vs Write%. This information can be obtained from your SAN management tools or looking at throughput for reads & writes.

Formula: Total Reads (KBps) + Total Writes (KBps) = Total Throughput in KBps

340,000KBps  + 100,000KBps  = 440,000 KBps 340,000 / 440,000 = 77.27 % Read 100,000 / 440,000 = 22.72 % Write

We can then use the this information to determine the functional IOPs of the array, this value is key in assessing whether or not the SAN is up to the job.

Functional IOPs = (Raw IOPs x Read%) + ((Raw IOPs x Write%) / RAID Write Penalty) 

Calculating Throughput Requirements

MBps = (Peak VM Workload IOPS * KB per IO) /1024 (use the VMware vscsiStats tool to output workload IO sizes)

MBps = (2000 * 8) /1024 = 15.625MBps

In the above example where the I/O workload requirement is 2000 IOPS, we would need 16MBps (128 Mbp/s @ 8KB per IO) of throughput to satisfy that requirement.

Note: esxtop can be used to determine SAN performance problems impacting hosts. Here are a couple of latency counters that should be monitored closely:

Value  Description
CMDS/s This is the total amount of commands per second and includes IOPS (Input/Output Operations Per Second) and other SCSI commands such as SCSI reservations, locks, vendor string requests, unit attention commands etc. being sent to or coming from the device or virtual machine being monitored. In most cases CMDS/s = IOPS unless there are a lot of metadata operations (such as SCSI reservations)
DAVG/cmd This is the average response time in milliseconds per command being sent to the deviceWarning threshold = 25High numbers (greater than 15-20ms) represent a slow or over worked array.
KAVG/cmd This is the amount of time the command spends in the Vmkernel.Warning threshold = 2 High numbers (greater than 2ms) represent either an overworked array or an overworked host.
GAVG/cmd This is the response time as it is perceived by the guest operating system. This number is calculated with the formula: DAVG + KAVG = GAVGWarning threshold = 2

Calculating I/O Size and I/O Profile : Random vs Sequential

See VMware vscsiStats tool / Dtrace for linux, SQLIO Windows.

ESX host monitoring

Have a look at the following VMware KB – Using esxtop to identify storage performance issues for ESX / ESXi (multiple versions) (1008205)

Virtual Machine Monitoring

  • Setup latency alarms within VMware vCentre to monitor virtual machine total disk latency.
    • The default is 75ms however, this should be adjusted depending on your KPI’s (<=25ms).

Host SCSI Latency

  • To reduce latency on the host, ensure that the sum of active commands from all virtual machines does not consistently exceed the LUN queue depth.
  • Either increase the LUN queue depth or move virtual machines to another LUN.
    • Observe vendor best practices for adjusting queue length on HBAs.
  • Work with the storage teams to ensure that each LUN is comprised of multiple physical disks.
  • To reduce latency on the array, reduce the maximum number of outstanding I/O commands to the shared LUN.
    • Distribute workloads across datastores.
    • Reduce VM to datastore ratios / reduce logical contention.
  • The maximum number of outstanding I/O commands that the array is capable of handling varies by array configuration, see my post on storage design considerations: Link

SCSI Reservations A SCSI reservation is a lock operation on the LUN preventing I/O from other operations. Consistent lock operations add latency measured in milliseconds.

  • Operations performed which can create locking metadata:
    • Creating a VMFS datastore
    • Expanding a VMFS datastore onto additional extents
    • Powering on a virtual machine
    • Acquiring a lock on a file
    • Creating or deleting a file
    • Creating a template
    • Deploying a virtual machine from a template
    • Creating a new virtual machine
    • Migrating a virtual machine with vMotion
    • Growing a file, for example, a snapshot file or a thin provisioned virtual disk
    • For the zeroed thick type of virtual disk the reservation is required only when zeroing the blocks.

In vSphere 4.x VMware released VAAI – VMware API for Array Integration, these primitives included an ATS – Atomic Test & Set which reduces locking.

  • For compatibility, check the VMware HCL/SAN Vendor –

From the hosts the following commands can be used to check the internal status ESX4.x

# esxcfg-scsidevs -l | egrep &amp;quot;Display Name:|VAAI Status:&amp;quot;

ESX5.x

# esxcli storage core device vaai status get

Note: You may need to install the binaries on the ESX host, if these have not been included. The binaries come in the form of VIBs, a reboot of the host will be required after installation.

VAAI changes are logged in the VMKernel logs at /var/log/vmkernel (for ESXi 5.0 vmkernel.log) or /var/log/messages).

VMware also provide a great KB for troubleshooting SCSI reversions conflicts on VMware Infrastructure VMware KB: 1005009 VMware KB enabling and troubleshooting VAAI : KB 1021976

Possible Solutions

Correctly plan and identify the workload characteristics and requirements:

  • Analyse total throughput, identify I/O profiles (read versus write percentages), I/O types (sequential vs random) and particularly I/O size.
  • Understand the workload characteristics, In most cases database workloads are inherently more I/O intensive than say the same number of web servers.
  • To reduce contention and latency, avoid oversubscribing by:
    • Spreading workloads across different LUNs (use VMware SDRS to automate balance workload across datastores).
    • Identify front-end bottlenecks, balance I/O to different storage processors and storage processor ports.
    • Use different FC or Ethernet paths.
    • Use multiple ESXi storage HBAs and HBA Ports.
    • Set appropriate pathing policies based on storage vendor best practises.
    • Sometimes poor storage performance can be attributed to bad host and VM management, check VM disk alignment, identify if VM’s are swapping consistently to your SAN , attributing to unnecessary I/O across your storage fabric, use local server-side storage to home VM SWAP if possible.
      • In the case of the latter reduce memory overcommit or increase physical RAM/Hosts.
  • Set appropriate queue depth values on HBA adapters, follow vendor recommendations. Observe impact to consolidation ratios specifically the number of VMs in a VMFS datastore. Setting queue depths too high can have a negative impact on performance (very large queue depths 64+, tend to just mask issues and are indicative of a larger performance problem).
  • Distributing workloads with a large number of transactions per second, you can also reduce latency by reducing the number of hops in the storage path.
  • Increase I/O front-end ports if possible (some SAN devices support the installation of additional HBA cards).
  • Check if target and initiator ports have negotiated correctly (check port properties on SAN fabric switches).
  • For NFS/iSCSI – Use, good quality switches and preferably dedicated stacks (1GbE – Dell PowerConnect 6200’s, Cisco 3750’s), if you are using 10GbE use NIOC to prevent other I/O operations from impacting your storage I/O.
  • Not directly related to performance but ensure proper L2 isolation between iSCSI traffic and general network traffic.
  • Calculate I/O cost to support the workload, factoring RAID write penalty.
    • Increase the number of drives to meet requirements.
  • Upgrade controller/SP cache, however in the event cache is saturated you will be reliant on back-end performance of the underlying drives, front-end cache is not a fix for poor back-end storage design.
    • To prevent noisy neighbours using all available front-end cache look at array side tools to create I/O polices for different workloads, prioritise those that are business critical.
    • VMware Storage IO Control (SIOC) can be used when latency impacts VM workloads by prioritising IO (only invoked during periods of contention).
  • Investigate potential use of SSD’s to absorb intensive I/O operations (Server side cache may also help – See PernixData FVP for VMs).
  • Investigate sub-lun tiering mechanisms to move hot blocks of data to drives with faster performance characteristics and less used blocks to slower storage (EMC FAST, HP 3PAR AO).
  • Use array multipathing policies, either native or thirdparty such as EMC powerpath. These policies can help by distributing I/O across all available storage paths in a more effective manner.
  • I/O Analyzer (VMware fling) can measure storage performance in a virtual environment and to help diagnose storage performance concerns. I/O Analyzer, supplied as an easy-to-deploy virtual appliance.
  • VMware Log Insight can aggregate and perform deep analysis of system logs identifying trends in many metrics in a fully customisable package. This is particularly useful when investigating storage related problems. http://www.vmware.com/products/vcenter-log-insight/

Recommended Reading

VMware KB / Troubleshooting Storage Performance Issues with VMware Products

Brocade FOS Admin Guide / Buffer Credit Management 

VMware Blog / Troubleshooting Storage Performance Queues

IBM Redbooks / Storage Area Networking