An assortment of indigestible things

Getting useful information out of ESXi with SNMP queries

This technique can be used to find out if any SNMP-enabled device has something interesting to share, but here I concentrate on VMware ESXi 4.1. No idea what the deal is with earlier versions.

There are quite a few blog posts out there describing how to enable the SNMP daemon on ESXi (this rather well-written one is the one I used).  However, the only information I’ve found says that the only data available is in the standard MIBs supported by most devices, giving information similar to netstat.  If that’s all there was, it ain’t particularly useful.

The other day I was sitting up in bed at stupid o’clock patching my ESXi servers, and I was skimming over the VMware KB articles describing the patches in case there was a footnote like ‘this patch will delete all of your VMs and revoke your driving licence’.  Then I saw this in the KB article for patch ESXi410-201107401-BG:

If the embedded SNMP agent of an ESXi host is enabled and SNMP queries are sent using the VMWARE-VMINFO-MIB.mib file to virtual machines that are being migrated, cloned, created, or destroyed, the ESXi host might stop responding.

SNMP queries to virtual machines?  Well I never.  Time to do a bit more digging.  (Hopefully this trail of breadcrumbs will avoid teaching your proverbial to suck proverbials…)

First I tried to find out what the SNMP daemon thought it could do:

$ snmpwalk -v 1 -c community hostname | head -2
SNMPv2-MIB::sysDescr.0 = STRING: VMware ESX 4.1.0 build-433742 VMware, Inc. x86_64
SNMPv2-MIB::sysObjectID.0 = OID: SNMPv2-SMI::enterprises.6876.4.1

Well… the sysObjectID gives us an enterprise number of 6876… so off I went to look it up at ByteSphere’s brilliant MIB download site. Unsurprisingly, it turned out to be VMware’s OID. There are about 14 MIBs there, so I just downloaded them all and put them in /usr/share/snmp/mibs on the machine I was querying from (this is the right place on an Ubuntu box with the snmp package installed).

Then I tried my command again, telling snmpwalk to read all available MIBs:

$ snmpwalk -v 1 -c community -m ALL hostname | head -2
SNMPv2-MIB::sysDescr.0 = STRING: VMware ESX 4.1.0 build-433742 VMware, Inc. x86_64
SNMPv2-MIB::sysObjectID.0 = OID: VMWARE-PRODUCTS-MIB::vmwESX

OK, so we’re definitely looking at something in VMware’s own tree. However, I tried walking the tree from this point and got

$ snmpwalk -v 1 -c community -m ALL hostname VMWARE-PRODUCTS-MIB::vmwESX
End of MIB

Hmm. OK, well maybe that’s not quite right, so let’s look at the numeric OID and work back up the tree until we (hopefully) get something:

$ snmpget -v 1 -c community -O n hostname SNMPv2-MIB::sysObjectID.0
.1.3.6.1.2.1.1.2.0 = OID: .1.3.6.1.4.1.6876.4.1

For those not on intimate terms with SNMP, the first number is simply the numeric representation for sysObjectID.0. Think of it as an IP address, and the symbolic name as the DNS record. What I’m going to do is drop each element from the end of the OID until the SNMP daemon gives me something (or I run out of numbers!).

$ snmpwalk -v 1 -c community -m ALL hostname .1.3.6.1.4.1.6876.4.1
End of MIB
$ snmpwalk -v 1 -c community -m ALL hostname .1.3.6.1.4.1.6876.4
End of MIB
$ snmpwalk -v 1 -c community -m ALL hostname .1.3.6.1.4.1.6876
VMWARE-ROOT-MIB::vmwSystem.1.0 = STRING: "VMware ESXi"
VMWARE-ROOT-MIB::vmwSystem.2.0 = STRING: "4.1.0"
VMWARE-ROOT-MIB::vmwSystem.4.0 = STRING: "433742"
...lots more...

Ah-hah! So the sysObjectID value wasn’t quite right, but we’ve found another cache of information. For each VM we can see:

VMWARE-VMINFO-MIB::vmwVmDisplayName.16 = STRING: WhateverMyVMisCalled
VMWARE-VMINFO-MIB::vmwVmConfigFile.16 = STRING: /vmfs/volumes/......
VMWARE-VMINFO-MIB::vmwVmGuestOS.16 = STRING: ubuntuGuest
VMWARE-VMINFO-MIB::vmwVmMemSize.16 = INTEGER: 512 megabytes
VMWARE-VMINFO-MIB::vmwVmState.16 = STRING: poweredOn
VMWARE-VMINFO-MIB::vmwVmVMID.16 = INTEGER: 16
VMWARE-VMINFO-MIB::vmwVmGuestState.16 = STRING: running
VMWARE-VMINFO-MIB::vmwVmCpus.16 = INTEGER: 1
...lots of stuff about disk controllers and NICs, including MAC addresses...
VMWARE-VMINFO-MIB::vmwCdromName.16.3002 = STRING: W: VM configured to use client device
VMWARE-VMINFO-MIB::vmwCdromConnected.16.3002 = STRING: false

In particular it’s worth monitoring vmwVmGuestOS in case it says ‘E: tools not installed’ (in which case vmmemctl isn’t there, which is a bad thing). vmwCdromName is also worth watching, because if it’s mapped to a file only accessible to that particular ESXi server, your VM might not be able to vMotion.

There’s some stuff under VMWARE-ROOT-MIB::vmwResources about devices configured on the server itself as opposed to the individual VMs, but that seems of little use for monitoring purposes. At least for my monitoring purposes.

Of course there’s nothing to say that the SNMP daemon won’t serve lots more useful information if only I knew what OID to ask for… but here at least I’ve found some useful stuff. As I said at the start, the technique of following the sysObjectID will work for any SNMP-enabled device. Plenty of manufacturers serve lots of interesting data but don’t document it.

One final point (at the risk of stating the bleedin’ obvious): ESXi does not have a host-based firewall, so please make sure that you block incoming traffic on port 161/udp at your firewall. I haven’t tried SNMP SET commands but if supported they could quite easily ruin your day.

Previous

Welcome

Next

Managing temporary ‘throwaway’ email addresses on your own mailserver

10 Comments

  1. Rajesh

    Good Article!!!!!!!!

    Can you guide on how to set SNMP checks for nagios/check_mk to monitor all data displayed by : snmpwalk -v 1 -c community -m ALL hostname .1.3.6.1.4.1.6876

    • flup

      I’m afraid that I don’t use check_mk; if I was doing this I’d write a perl script to do it. Hopefully someone else will be able to help you though.

  2. Here’s a simpler way to find out about what a given ESXi version’s snmp agent supports. First download the mib module file set from:
    http://communities.vmware.com/community/developer/forums/managementapi
    This link is updated to always point to the latest edition. SNMP MIB modules are backward compatible so like installing pc drivers, you just need the latest set for any number of ESXi or ESX systems at various versions.

    Next examine the VMWARE-AGENTCAP-MIB. This file reports by ESXi version what the SNMP agent will report on. You can also poll the SNMPv2-MIB: sysORTable:
    snmpwalk -mall -v2c -c public 192.0.2.1 sysORTable
    The SNMPv2-MIB (RFC 3418) reports sysObjectId value as defined in VMWARE-PRODUCTS-MIB. The VMWARE-SYSTEM-MIB reports ESXi version and build details.
    Those are useful to identify a given system and the former is used by most mgmt systems to determine what the system is.
    Once you have all the MIB modules extracted from the zip archive, get a copy of the libsmi tool. Run the command
    smidump -f tree VMWARE-ROOT-MIB.mib VMWARE-VMINFO-MIB.mib
    (Notice you have to list any files that have IMPORT X-MIB statements at the top of the file before the one you want to dump)
    This will give you a whole new way of viewing arcane SMIv2 format mib modules.
    The ‘smidump -f identifiers’ is especially useful to do oid to name mappings.

    >ESXi does not have a host-based firewall
    It does in 5.x releases, esxcli network firewall is the ESX Shell command.
    >I haven’t tried SNMP SET commands but if supported
    No SNMP Set support, ESXi agent is read-only.

    • flup

      Many thanks for your helpful comments. I hadn’t tried ESXi 5 when I wrote this post (was it out yet?), but I do like the introduction of a host-based firewall.

  3. allen

    Just an FYI : The right OID name to use for walking the tree is simply ‘vmware’ (defined in VMWARE-ROOT-MIB), e.g. “snmpwalk -Os -v2c -m ALL -c public [[hostname]] vmware

    This will pull all the OIDs that start at VMWARE-ROOT. No need to use numeric IDs, you were just using the wrong OID name.

  4. Scott Bemister

    Hi I was wondering if you could help me figure out what a trap means coming from my ESXi host. I use OpManager by Adventnet as my monitoring system and it is receiving the following trap (which i believe comes from VMINFO mib).

    Enterprise: vmwESX Generic Type: 6 Specific Type: 2 Variable Bindings: vmwVmID: 2464 , vmwVmConfigFilePath: /vmfs/volumes/4b8f876a-b66b5681-6a40-0026b95d1645/DEV02/DEV02.vmx , vmwVmDisplayName.2464: DEV02

    As you can see, i can parse the trap but, it doesn’t actually tell me what is happening. Which means I don’t really know what to do with this trap when I get it.

    Any help you may be willing to provide would be most appreciated.

    • flup

      Doesn’t mean anything to me I’m afraid, but I’ll leave it here in case anyone else can help.

  5. Michael MacFaden

    To answer Scott Bemister’s question here’s what I’d do in the event I can’t log an SR with VMware GSS to get a quick answer:

    1) This OID indicates VMware 1.3.6.1.4.1.6876 and any oid starting with it is VMware origin
    To find any companies enterprise root, use this authoritative list:
    http://www.iana.org/assignments/enterprise-numbers/enterprise-numbers

    Next if on the web there are a number of oid to name lookup tools
    I like to use http://mibdepot.com/index.shtml to find the answer for any arbitrary OID, be it from VMware or not.

    For VMware in particular, having the current OID listing: http://kb.vmware.com/kb/2054359
    provides a great table of contents. From there download the vmware mib modules from:
    http://kb.vmware.com/kb/1013445 # both KBs are linked together making traversal pretty simple

    So from the oid listing we would search for notifications ending with “2”

    VMWARE-VMINFO-MIB vmwVmPoweredOff notification 1.3.6.1.4.1.6876.4.1.0.2

    Then grab the VMARE-VMINFO-MIB.mib file and it shows this notificaiton is defined as:

    vmwVmPoweredOff NOTIFICATION-TYPE
    OBJECTS { vmwVmID, vmwVmConfigFilePath, vmwVmDisplayName }
    STATUS current
    DESCRIPTION
    “This trap is sent when a virtual machine is powered off. The origin of this event can be several:
    for instance may be operator initiated, vmx process terminating abnormally. NOTE: vms powered down due
    to VMotion are not reported. Upon receiving this notification client applications should
    poll the vmwVmTable to obtain current status.”
    ::= { vmwESXNotifications 2 }

  6. Is there a way to find if VMware Tools are installed on a VM? or to know VMware Tools version?
    I found things in the documentation of the MIBs that say (this is for the “vmwVmMAC”, but it is repeated for one or two more).

    “Reports the configured virtual hardware MAC address. If VMware Tools is not running, the value is zero or empty.”

    but nothing to explicitly query if VMware Tools. Any ideas?

    Thanks in advance 🙂

  7. Carol Ostos

    Has anyone experience this? When I run the snmpwalk against vmware, I get
    No log handling enabled – turning on stderr logging
    Did not find ‘vmDisplayName’ in module VMWARE-VMINFO-MIB (/usr/share/snmp/mibs/VMWARE-TRAPS-MIB.txt)

    Any clues?

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Powered by WordPress & Theme by Anders Norén