An assortment of indigestible things

Restoring datastore performance graphs after upgrading to vCenter 5.0U1

Datastore performance data graphs just say 'No data available'


My recent upgrade from ESXi and vCenter 4.1U3 to 5.0U1 went so smoothly, I should have known that not everything was as it should have been. My battle scars from decades of fighting buggy software (in other words, software) were tingling, and it wasn’t too long before I found the problem. All my datastores’ non-realtime performance data graphs—which I rely on for troubleshooting slow VMs—were blank. ‘No data available‘ was all I got. A call to the refreshingly excellent VMware support folk resulted in a pointer to this KB article. It’s a bit involved, especially if you don’t already have PowerCLI installed, so here’s a quick walk-through.

The reason for all this

The KB article says that

In vSphere 5.0 Update 1 and later, by default the Storage DRS and SIOC counters are active at the higher statistics collection level 3. This effectively disables non-realtime datastore performance charts. In order to re-enable these performance charts, you can choose to move those counters back to collection level 1.

but goes on to warn that

moving Storage DRS and SOIC counters to a lower collection level will negatively impact performance, because more statistics are persisted, requiring more I/O and database operations.

First, install PowerCLI…

…for which you will need a Windows box. That’s right: while the ordinary CLI has both Windows and Linux installers, PowerCLI needs .NET and PowerShell. Oh well, luckily I have a dogsbody Windows VM that I can use for such things.

Finding a download for PowerCLI isn’t easy either. Searching for ‘PowerCLI’ in ‘All Downloads’ on VMware’s download website gives no results (cheers, very helpful), at least at the time of writing. You have to go into vSphere downloads and click ‘Drivers & Tools’, then open up ‘Automation Tools and SDKs’; it’s the link at the bottom. If you can’t be arsed with all that, here’s a direct link to the PowerCLI download page. You’ll need to be registered on VMware’s website to download it.

When you run the downloaded installer, it’ll first install the VIX API. If you’ve never used Powershell for anything on this machine, you’ll then get a warning about the execution policy not being set to ‘RemoteSigned’.

By default, the policy is ‘Restricted‘, meaning that Powershell only works interactively. Setting the policy to ‘RemoteSigned‘ means that ‘downloaded scripts must be signed by a trusted publisher before they can be run‘. If you’re happy to make that change (the installer says it’ll be OK if you don’t, but I didn’t want any more trouble), you’ll need to run a Powershell session as administrator, and enter the command

Set-ExecutionPolicy RemoteSigned

If it throws an error about not being able to write to the registry, you’re not running as administrator: remember to start a command prompt by right-clicking and selecting ‘Run as Administrator‘.

After that, the installation should complete without incident.

Running VMware’s little hack

There’s no getting away from it: this is a little hack. The KB article asks that you download a file called LevelMappingUtility.zip (it’s near the bottom of the article under ‘Attachments’). This contains two files: a Powershell script, and a CSV file with some variable-value pairs in it. You need to extract both files into a directoryfolder (I used c:\tmp), start a PowerCLI session from the new entry in your start menu, and do something like this (and be patient, it takes ages to respond):

Import-Module c:\tmp\VMware.VimAutomation.PowerCliExtensions.CounterLevelMapping.psm1
Connect-VIServer <ip address of vCenter server>,<same ip address again>

At this point (or at least a minute or so after this point) a window will pop up asking you for your vCenter credentials. You might get a message whinging about certificate problems, then you’ll be prompted for the credentials again. Finally you’ll be asked if you want to use multiple default servers; I just said ‘no’ as I only have one vCenter instance.

If you’ve got this far, you can now run the command that actually makes the changes:

Import-Csv -path c:\tmp\sdrs-sioc-reenable.csv | Set-PxCounterLevelMapping -Server <ip address of vCenter server>,<same ip address again>

After a while, you should get a couple of dozen ‘Updating’ messages to show it all went well.

If that was a bit hard to follow, here’s a screenshot of the first couple of commands:

and the last one:

My graphs came back about half an hour after I finished this procedure. I’ve been monitoring my vCenter instance closely but I haven’t seen any evidence of performance problems. However, I don’t use Storage DRS, which is said to increase the risk of vCenter database overload. It’s probably worth keeping an eye on things for a few days after you make this change.

Previous

Getting ESXi 5.0 to use a remote syslog server

Next

Using ZoneMinder with a cheap CCTV camera

6 Comments

  1. soul

    ok, thanks, but…

    what if the same thing happen’s to ESXi 4.0 environment?

    How can I check performance on datastores there.

    I am not sure that this ever worked for me, so maybe there is something…
    WIll look into, but I thought I should ask anyway

    GL

    • flup

      If you’re using vCenter 4.0 then presumably you also have a VMware support contract, so the best thing to do would be to log an SR and see what they have to say for themselves.

  2. Jonathan

    This looks to still be an issue in version 5.1 Build 941893. My performance charts were empty as well. I followed these instructions and data started to appear. Thank you for this walk-through.

  3. Eliezer

    Hi,

    Does the file sdrs-sioc-reenable.csv needs to be manually edited with the info of my environment?

    Because the file I downloaded came with this info:

    datastore.datastoreReadOIO.latest,1,1,example.server.com
    datastore.datastoreWriteIops.latest,1,1,example.server.com
    datastore.datastoreWriteOIO.latest,1,1,example.server.com
    datastore.siocActiveTimePercentage.average,1,1,example.server.com

    So, if I want to change the level to 3, I need to change those 1,1 to 3,3 and also put the fqdn of my vcenter server?

    Best Regards!

  4. Randy

    Thanks, this was a big help for me!

  5. Jim Lentz

    This was an issue even on a fresh VCenter 5.5 install for me. Your procedure worked in the 3 environments we have, Essentials, Essentials + and Enterprise. Thanks!

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Powered by WordPress & Theme by Anders Norén