Revision as of 12:39, 16 April 2012

Introduction

Tarus wrote up a useful guide on using the Net-SNMP extend function to perform custom "script-like" collections while retaining the efficiency of SNMP. Here is a wikified version to capture the method for OpenNMS documentation.

The blog post is a good read and provides greater detail and context.

Example 1

Mail queue stat collection.

"mailq" command on mail server, with results ending in:

-- 23 Kbytes in 6 Requests.

parse the "6"

mailq | tail -n 1 | awk '{if (NF > 4) {print $5} else {print 0}}'

make it a script

# cat /root/bin/mailqstats.sh
#!/bin/bash
mailq | tail -n 1 | awk '{if (NF > 4) {print $5} else {print 0}}'

add an extend config into /etc/snmp/snmpd.conf

extend mailqstats /root/bin/mailqstats.sh

restart snmpd and use snmpwalk to see results of extend addition

snmpwalk -v1 -c public mail.example.com .1.3.6.1.4.1.8072.1.3.2

NET-SNMP-EXTEND-MIB::nsExtendNumEntries.0 = INTEGER: 1
NET-SNMP-EXTEND-MIB::nsExtendCommand."mailqstats" = STRING: /root/bin/mailqstats.sh
NET-SNMP-EXTEND-MIB::nsExtendArgs."mailqstats" = STRING:
NET-SNMP-EXTEND-MIB::nsExtendInput."mailqstats" = STRING:
NET-SNMP-EXTEND-MIB::nsExtendCacheTime."mailqstats" = INTEGER: 5
NET-SNMP-EXTEND-MIB::nsExtendExecType."mailqstats" = INTEGER: exec(1)
NET-SNMP-EXTEND-MIB::nsExtendRunType."mailqstats" = INTEGER: run-on-read(1)
NET-SNMP-EXTEND-MIB::nsExtendStorage."mailqstats" = INTEGER: permanent(4)
NET-SNMP-EXTEND-MIB::nsExtendStatus."mailqstats" = INTEGER: active(1)
NET-SNMP-EXTEND-MIB::nsExtendOutput1Line."mailqstats" = STRING: 6
NET-SNMP-EXTEND-MIB::nsExtendOutputFull."mailqstats" = STRING: 6
NET-SNMP-EXTEND-MIB::nsExtendOutNumLines."mailqstats" = INTEGER: 1
NET-SNMP-EXTEND-MIB::nsExtendResult."mailqstats" = INTEGER: 0
NET-SNMP-EXTEND-MIB::nsExtendOutLine."mailqstats".1 = STRING: 6

the OID of the result uses extend function name, "mailqstats," converted into ASCII values “109.97.105.108.113.115.116.97.116.115″

Collection

configure data collection in datacollection-config.xml

<group name="mailq-stats" ifType="ignore">
   <mibObj oid=".1.3.6.1.4.1.8072.1.3.2.4.1.2.10.109.97.105.108.113.115.116.97.116.115"
      instance="1" alias="mailqsize" type="octetstring" />
</group>

Note the type of “octetstring”. If you look at the type of this OID from the walk above, you’ll see it is “string”. RRDtool and JRobin can’t store string data, thus it needs to be converted to a number. Setting the type to “octetstring” causes this to happen (it is converted to a gauge). If it was left as a string, OpenNMS would collect it but store it only once in the strings.properties file for the node. It would not be able to graph it.

include the collection in a system definition

<systemDef name="Net-SNMP (UCD)">
  <sysoidMask>.1.3.6.1.4.1.2021.250.</sysoidMask>
  <collect>
     <includeGroup>mib2-host-resources-system</includeGroup>
     <includeGroup>mib2-host-resources-memory</includeGroup>
     <includeGroup>mib2-host-resources-storage</includeGroup>
     <includeGroup>net-snmp-disk</includeGroup>
     <includeGroup>ucd-loadavg</includeGroup>
     <includeGroup>ucd-memory</includeGroup>
     <includeGroup>ucd-sysstat</includeGroup>
     <includeGroup>ucd-sysstat-raw</includeGroup>
     <includeGroup>ucd-sysstat-raw-more</includeGroup>
     <includeGroup>mailq-stats</includeGroup>
   </collect>
</systemDef>

add graph definition to the appropriate snmp-graph.properties file

report.netsnmp.mailq.name=Current Mail Queue Size
report.netsnmp.mailq.columns=mailqsize
report.netsnmp.mailq.type=nodeSnmp
report.netsnmp.mailq.command=--title="Currently Queued Messages" \
 DEF:queue={rrd1}:mailqsize:AVERAGE \
 LINE2:queue#0000A0:"Size    " \
 GPRINT:queue:AVERAGE:"Avg  \\: %8.2lf " \
 GPRINT:queue:MIN:"Min  \\: %8.2lf " \
 GPRINT:queue:MAX:"Max  \\: %8.2lf \\n"

do not forget to add the “netsnmp.mailq” report to the “reports=” line at the top of the file.

restart OpenNMS

graph displays under Node-level Performance Data

(Optional) To monitor mailqstats as a service, the following config (or provisiond equivalent) is used.

Discovery

capsd-configuration.xml-

<protocol-plugin
   protocol="Mailq"
   class-name="org.opennms.netmgt.capsd.plugins.SnmpPlugin" scan="on">
   <property
      key="vbname"
      value=".1.3.6.1.4.1.8072.1.3.2.4.1.2.10.109.97.105.108.113.115.116.97.116.115.1" />
   <property key="timeout" value="2000" />
   <property key="retry" value="1" />
</protocol-plugin>

Polling

poller-configuration.xml-

<service name="Mailq" interval="300000" user-defined="false" status="on">
   <parameter key="retry" value="1"/>
   <parameter key="timeout" value="3000"/>
   <parameter key="port" value="161"/>
   <parameter key="oid"
       value=".1.3.6.1.4.1.8072.1.3.2.4.1.2.10.109.97.105.108.113.115.116.97.116.115.1"/>
   <parameter key="operator" value="<"/>
   <parameter key="operand" value="300"/>
</service>

<monitor service="Mailq" class-name="org.opennms.netmgt.poller.monitors.SnmpMonitor"/>

Note: this configuration monitors the service as "down" when the queue count is less than 300.

Note: the "operator" value is represented as "& l t ;" (spaces added) (HTML for "<")

Example 2

Mail queues stat collection using qshape.

I amended the method shown by Tarus above with a method for collecting the Postfix Queue values from Cacti. Postfix Queues As I wanted to see the state of four queues. Deferred, Active, Incoming and Hold, and where most problems we have are related to the Active queue becoming congested.

Create a script

#!/bin/bash
#getmailq.sh - original written by Valec 2006. Steal and share.
#Get postfix queue lengths for Deferred, Active, Incoming and Hold
#Place in /usr/local/sbin/
#extend mailqstats /usr/local/sbin/getmailq.sh 
#(Default oid for NET-SNMP-EXTEND-MIB is .1.3.6.1.4.1.8072.1.3.2)
QUEUES="incoming active deferred hold"
for i in $QUEUES; do
       COUNT=`qshape $i | grep TOTAL | awk '{print $2}'`
       printf "$COUNT \n"
done

Add the following to /snmpd.conf

extend mailqstats /usr/local/sbin/getmailq.sh

restart snmpd and use snmpwalk to see results of extend addition

snmpwalk -v2c -cpublic mail.example.com .1.3.6.1.4.1.8072.1.3.2

NET-SNMP-EXTEND-MIB::nsExtendNumEntries.0 = INTEGER: 1
NET-SNMP-EXTEND-MIB::nsExtendCommand."mailqstats" = STRING: /usr/local/sbin/getmailq.sh
NET-SNMP-EXTEND-MIB::nsExtendArgs."mailqstats" = STRING:
NET-SNMP-EXTEND-MIB::nsExtendInput."mailqstats" = STRING:
NET-SNMP-EXTEND-MIB::nsExtendCacheTime."mailqstats" = INTEGER: 5
NET-SNMP-EXTEND-MIB::nsExtendExecType."mailqstats" = INTEGER: exec(1)
NET-SNMP-EXTEND-MIB::nsExtendRunType."mailqstats" = INTEGER: run-on-read(1)
NET-SNMP-EXTEND-MIB::nsExtendStorage."mailqstats" = INTEGER: permanent(4)
NET-SNMP-EXTEND-MIB::nsExtendStatus."mailqstats" = INTEGER: active(1)
NET-SNMP-EXTEND-MIB::nsExtendOutput1Line."mailqstats" = STRING: 0
NET-SNMP-EXTEND-MIB::nsExtendOutputFull."mailqstats" = STRING: 0
4
213
0
NET-SNMP-EXTEND-MIB::nsExtendOutNumLines."mailqstats" = INTEGER: 4
NET-SNMP-EXTEND-MIB::nsExtendResult."mailqstats" = INTEGER: 0
NET-SNMP-EXTEND-MIB::nsExtendOutLine."mailqstats".1 = STRING: 0
NET-SNMP-EXTEND-MIB::nsExtendOutLine."mailqstats".2 = STRING: 4
NET-SNMP-EXTEND-MIB::nsExtendOutLine."mailqstats".3 = STRING: 213
NET-SNMP-EXTEND-MIB::nsExtendOutLine."mailqstats".4 = STRING: 0

Collection

configure data collection in datacollection-config.xml

  <group name="mailq-stats" ifType="ignore">
       <mibObj oid=".1.3.6.1.4.1.8072.1.3.2.4.1.2.10.109.97.105.108.113.115.116.97.116.115"
          instance="1" alias="incoming" type="octetstring" />
       <mibObj oid=".1.3.6.1.4.1.8072.1.3.2.4.1.2.10.109.97.105.108.113.115.116.97.116.115" 
          instance="2" alias="active" type="octetstring" />
       <mibObj oid=".1.3.6.1.4.1.8072.1.3.2.4.1.2.10.109.97.105.108.113.115.116.97.116.115" 
          instance="3" alias="deferred" type="octetstring" />
       <mibObj oid=".1.3.6.1.4.1.8072.1.3.2.4.1.2.10.109.97.105.108.113.115.116.97.116.115"
          instance="4" alias="hold" type="octetstring" />
   </group>

include the collection in a system definition, in my case

     <systemDef name="Net-SNMP">
       <sysoidMask>.1.3.6.1.4.1.8072.3.</sysoidMask>
       <collect>
         <includeGroup>mib2-host-resources-system</includeGroup>
         <includeGroup>mib2-host-resources-memory</includeGroup>
         .
         .
         .
         <includeGroup>zxtmpool</includeGroup>
       <includeGroup>mailq-stats</includeGroup>
       </collect>
     </systemDef>

add graph definition to the appropriate snmp-graph.properties file

# Custom NET-SNMP Script extend: MailQ from Postfix
report.netsnmp.mailq.name=Current Mail Queue's Sizes
report.netsnmp.mailq.columns=incoming,active,deferred,hold
report.netsnmp.mailq.type=nodeSnmp
report.netsnmp.mailq.command=--title="E-Mail: Current Queued Postfix Messages" \
 --vertical-label="Size" \
 DEF:in={rrd1}:incoming:AVERAGE \
 DEF:act={rrd2}:active:AVERAGE \
 DEF:def={rrd3}:deferred:AVERAGE \
 DEF:hld={rrd4}:hold:AVERAGE \
 LINE2:in#544e52:"Incoming " \
 GPRINT:in:AVERAGE:"Current  \\: %8.0lf" \
 GPRINT:in:MAX:"     Max  \\: %8.0lf\\n" \
 LINE2:act#0066FF:"Active   " \
 GPRINT:act:AVERAGE:"Current  \\: %8.0lf" \
 GPRINT:act:MAX:"     Max  \\: %8.0lf\\n" \
 LINE2:def#00aa00:"Deferred " \
 GPRINT:def:AVERAGE:"Current  \\: %8.0lf" \
 GPRINT:def:MAX:"     Max  \\: %8.0lf\\n" \
 LINE2:hld#ff0000:"Hold     " \
 GPRINT:hld:AVERAGE:"Current  \\: %8.0lf" \
 GPRINT:hld:MAX:"     Max  \\: %8.0lf\\n"

Not forgetting to put netsnmp.mailq to reports=

The results should be a graph like this