SNMP Modular Input: Why does data collection randomly stop with error "unknownEngineID snmp_stanza:snmp://xxxx"?

[samlll42]

Using Splunk 6.2.1 and latest snmp_ta (1.2.7)

SNMP data collection stops working randomly and shows the error below in splunkd.log (for each of the stanzas configured)

02-25-2015 11:04:24.837 -0800 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/snmp_ta/bin/snmp.py" unknownEngineID snmp_stanza:snmp://xxxx

It can be easily reproduced by stopping the snmp daemon that it is querying for a few minutes and starting it again.

The easiest way I found to get it back up is to do a /en-US/debug/refresh. Then errors stop and SNMP data collection works again.

Using the following stanza in inputs.conf:

[snmp://XXX]
communitystring = xxxx
destination = xxxx
do_bulk_get = 1
ipv6 = 0
max_repetitions = 25
mib_names = xxx (custom MIB)
non_repeaters = 0
object_names = 1.3.6.1.4.1.7102.1971
snmp_mode = attributes
snmp_version = 3
sourcetype = xxxxx_snmp_ta
split_bulk_output = 1
v3_authProtocol = usmHMACMD5AuthProtocol
v3_privKey =
v3_privProtocol = usmDESPrivProtocol
v3_securityName = xxxxx
v3_authKey = xxxxx
snmpinterval = 300

It is gathering data from Linux Snmpd (net-snmp)with a custom MIB provided by a 3rd party vendor.

Anyone had the same issue? any idea on how to resolve this?

Thanks

[Super_Knulps]

Please someone answer to this.
Up.

Thank you very much in advance.

[samlll42]

Unfortunately wasn't able to find out the cause of this problem had to give up on snmp_ta and switch to a custom scripted input with snmpbulkwalk... 😞

[jadengoho]

could you tell me the process on how you do the custom script ? If possible can you post the script itself here ?

[samlll42]

More details... Having same behavior with CLI:

/opt/splunk/bin/splunk cmd splunkd print-modinput-config snmp snmp://mobile | /opt/splunk/bin/splunk cmd python /opt/splunk/etc/apps/snmp_ta/bin/snmp.py

Everything runs fine:
[...]
SNMPv2-SMI::enterprises."8072.1.2.1.1.4.0.8.1.3.6.1.2.1.1.9.127" = "mibII/sysORTable" xxx
SNMPv2-SMI::enterprises."8072.1.2.1.1.4.0.8.1.3.6.1.2.1.2.1.127" = "if number" xxx
xxx

[Stopping the snmpd for a few seconds]

ERROR No SNMP response received before timeout snmp_stanza:snmp://xxx

[restarting the snmpd and collections no longer works]

ERROR unknownEngineID snmp_stanza:snmp://xxx
ERROR unknownEngineID snmp_stanza:snmp://xxx
ERROR unknownEngineID snmp_stanza:snmp://xxx
ERROR unknownEngineID snmp_stanza:snmp://xxx

After that, have to reload inputs (or do a /debug/refresh), which restarts the process and it works again.

INFO ExecProcessor - New scheduled exec process: python /opt/splunk/etc/apps/snmp_ta/bin/snmp.py

The same behavior (without interruption of snmpd service) can be expected systematically after a few hours.

Any suggestion?