Can't receive or index snmp traps from either HP Blade Center VirtualConnect switches and Onboard Administrators

[chjamey]

Hello. We are evaluating Splunk to see if it will meet all of our monitoring needs. I recently installed the snmp modular input on a windows server 2012 R2 server.

I'm able to poll attributes via OID but I can't seem to index any traps when I choose to "receive traps" instead of poll. I'm able to snmp walk the network .

I am testing with an HP Onboard Administrator. Having it send traps to the splunk server. It is on the same subnet. I'm able to snmp walk the OA from the splunk server.

When I send a test trap from the OA to the splunk server, I get nothing. I read about having to add custom MIBs for certain devices and I don't see the HP blade center mibs in the mib egg. I've tried many different configs and googled this a ton.

I downloaded the HP VirtualConnect mib from hp and was going to try to convert the mib to a py and add it to the solution but I can't find the build-pysnmp-mib or smidump.

I appreciate any help or direction on this.

Jamey

[vincenteous]

Hi chjamey,

What user is currently running the splunk service? Administrator/root user or another user? From my experience, splunk cannot listen to trap because of port privilege problem. Any user who is not a root/Administrator user doesn't have the privilege to use any port below 1024. In this case, it's port 162 for SNMP trap. What I did for the workaround solution was to re-route the listener port to something bigger than 1024 (For example: 162 -> 9162) and restart the splunk service. After that, I created the listener from Web UI, specify index and so forth, and it's done.

Hope this helps. If not, hope this will be a hint to another solution.

Best Regards,

Vincent

[chjamey]

Thanks a lot for the information Vincent.

The splunkd service is running as an Active Directory service account that is in local admins group on the splunk enterprise, server 2012 R2 box.

I'd rather not have to change the listening port on the splunk server because I'd have to change the destination port in all the switches\OAs.

Will keep digging. I am looking at having net-snmp possibly right to a file and then having splunk index the files. Not as clean but if it works and isn't a lot of performance overhead, then it may be a solution.

Thanks again!
Jamey

[Damien_Dallimor]

Any errors in the logs ?

Splunk search : index=_internal ExecProcessor error snmp.py

Trap Listener Host attribute set correctly ? (ie: matches the host the trap sender is sending to)

SNMP Version set correctly ?

[chjamey]

Any ideas Damien? I'm trying to get this POC up as soon as possible.

Thanks again,
Jamey

[chjamey]

Just to add to this... to test the server receiving trap destinations, I disabled splunk and installed a third party trap destination receiver on the splunk server, kiwi, and verified that I am able to receive the traps just fine.

[chjamey]

Thanks for your response Damien. The following error was shown over and over...

02-18-2015 13:15:05.315 -0500 ERROR ExecProcessor - message from "python "C:\Program Files\Splunk\etc\apps\snmp_ta\bin\snmp.py"" Failed to register transport and run dispatcher: bind() for (u'SplunkServer01', 162) failed: [Errno 10013] An attempt was made to access a socket in a way forbidden by its access permissions snmp_stanza:snmp://BCI 9 Traps

The user Splunk is running under is admin on the splunk server. Any ideas?

Thanks,
Jamey

[Damien_Dallimor]

According to the error , you don't have permission on the OS to open port 162.

[chjamey]

Yeah I know that is what it is saying but when I disable splunk, enable the other snmp collector service(Kiwi syslog and using the same AD account as the Splunk service) for testing, reboot the Splunk server, I'm able to receive the snmp messages just fine. I'll keep working on it and report back.

Thanks for your reply.

Jamey

[chjamey]

Forgot to put in the other info you asked for....

snmp version is good(2c, which matches the version on the OA) and the Trap Listener Host(splunk server) is correct as well.