Getting useful information out of ESXi with SNMP queries

This technique can be used to find out if any SNMP-enabled device has something interesting to share, but here I concentrate on VMware ESXi 4.1. No idea what the deal is with earlier versions.

There are quite a few blog posts out there describing how to enable the SNMP daemon on ESXi (this rather well-written one is the one I used).  However, the only information I’ve found says that the only data available is in the standard MIBs supported by most devices, giving information similar to netstat.  If that’s all there was, it ain’t particularly useful.

The other day I was sitting up in bed at stupid o’clock patching my ESXi servers, and I was skimming over the VMware KB articles describing the patches in case there was a footnote like ‘this patch will delete all of your VMs and revoke your driving licence’.  Then I saw this in the KB article for patch ESXi410-201107401-BG:

If the embedded SNMP agent of an ESXi host is enabled and SNMP queries are sent using the VMWARE-VMINFO-MIB.mib file to virtual machines that are being migrated, cloned, created, or destroyed, the ESXi host might stop responding.

SNMP queries to virtual machines?  Well I never.  Time to do a bit more digging.  (Hopefully this trail of breadcrumbs will avoid teaching your proverbial to suck proverbials…)

First I tried to find out what the SNMP daemon thought it could do:

$ snmpwalk -v 1 -c community hostname | head -2
SNMPv2-MIB::sysDescr.0 = STRING: VMware ESX 4.1.0 build-433742 VMware, Inc. x86_64
SNMPv2-MIB::sysObjectID.0 = OID: SNMPv2-SMI::enterprises.6876.4.1

Well… the sysObjectID gives us an enterprise number of 6876… so off I went to look it up at ByteSphere’s brilliant MIB download site. Unsurprisingly, it turned out to be VMware’s OID. There are about 14 MIBs there, so I just downloaded them all and put them in /usr/share/snmp/mibs on the machine I was querying from (this is the right place on an Ubuntu box with the snmp package installed).

Then I tried my command again, telling snmpwalk to read all available MIBs:

$ snmpwalk -v 1 -c community -m ALL hostname | head -2
SNMPv2-MIB::sysDescr.0 = STRING: VMware ESX 4.1.0 build-433742 VMware, Inc. x86_64
SNMPv2-MIB::sysObjectID.0 = OID: VMWARE-PRODUCTS-MIB::vmwESX

OK, so we’re definitely looking at something in VMware’s own tree. However, I tried walking the tree from this point and got

$ snmpwalk -v 1 -c community -m ALL hostname VMWARE-PRODUCTS-MIB::vmwESX
End of MIB

Hmm. OK, well maybe that’s not quite right, so let’s look at the numeric OID and work back up the tree until we (hopefully) get something:

$ snmpget -v 1 -c community -O n hostname SNMPv2-MIB::sysObjectID.0
.1.3.6.1.2.1.1.2.0 = OID: .1.3.6.1.4.1.6876.4.1

For those not on intimate terms with SNMP, the first number is simply the numeric representation for sysObjectID.0. Think of it as an IP address, and the symbolic name as the DNS record. What I’m going to do is drop each element from the end of the OID until the SNMP daemon gives me something (or I run out of numbers!).

$ snmpwalk -v 1 -c community -m ALL hostname .1.3.6.1.4.1.6876.4.1
End of MIB
$ snmpwalk -v 1 -c community -m ALL hostname .1.3.6.1.4.1.6876.4
End of MIB
$ snmpwalk -v 1 -c community -m ALL hostname .1.3.6.1.4.1.6876
VMWARE-ROOT-MIB::vmwSystem.1.0 = STRING: "VMware ESXi"
VMWARE-ROOT-MIB::vmwSystem.2.0 = STRING: "4.1.0"
VMWARE-ROOT-MIB::vmwSystem.4.0 = STRING: "433742"
...lots more...

Ah-hah! So the sysObjectID value wasn’t quite right, but we’ve found another cache of information. For each VM we can see:

VMWARE-VMINFO-MIB::vmwVmDisplayName.16 = STRING: WhateverMyVMisCalled
VMWARE-VMINFO-MIB::vmwVmConfigFile.16 = STRING: /vmfs/volumes/......
VMWARE-VMINFO-MIB::vmwVmGuestOS.16 = STRING: ubuntuGuest
VMWARE-VMINFO-MIB::vmwVmMemSize.16 = INTEGER: 512 megabytes
VMWARE-VMINFO-MIB::vmwVmState.16 = STRING: poweredOn
VMWARE-VMINFO-MIB::vmwVmVMID.16 = INTEGER: 16
VMWARE-VMINFO-MIB::vmwVmGuestState.16 = STRING: running
VMWARE-VMINFO-MIB::vmwVmCpus.16 = INTEGER: 1
...lots of stuff about disk controllers and NICs, including MAC addresses...
VMWARE-VMINFO-MIB::vmwCdromName.16.3002 = STRING: W: VM configured to use client device
VMWARE-VMINFO-MIB::vmwCdromConnected.16.3002 = STRING: false

In particular it’s worth monitoring vmwVmGuestOS in case it says ‘E: tools not installed’ (in which case vmmemctl isn’t there, which is a bad thing). vmwCdromName is also worth watching, because if it’s mapped to a file only accessible to that particular ESXi server, your VM might not be able to vMotion.

There’s some stuff under VMWARE-ROOT-MIB::vmwResources about devices configured on the server itself as opposed to the individual VMs, but that seems of little use for monitoring purposes. At least for my monitoring purposes.

Of course there’s nothing to say that the SNMP daemon won’t serve lots more useful information if only I knew what OID to ask for… but here at least I’ve found some useful stuff. As I said at the start, the technique of following the sysObjectID will work for any SNMP-enabled device. Plenty of manufacturers serve lots of interesting data but don’t document it.

One final point (at the risk of stating the bleedin’ obvious): ESXi does not have a host-based firewall, so please make sure that you block incoming traffic on port 161/udp at your firewall. I haven’t tried SNMP SET commands but if supported they could quite easily ruin your day.