topic Re: Trying to get SNMP data into Splunk... in All Apps and Add-ons

Trying to get SNMP data into Splunk...

cpt12tech — Tue, 29 Sep 2020 09:07:35 GMT

I'm having problems getting any SNMP data into Splunk using the SNMP Modular Input. I've set up a Polling Input that is polling a Windows 2008 Server. So far no data has shown up in Splunk.

I've checked for errors using:
index=_internal ExecProcessor error snmp.py
and don't see any results.

[snmp://Archive]
communitystring = public
destination = 192.168.1.101
do_bulk_get = 1
do_get_subtree = 1
host = Archive
ipv6 = 0
snmp_mode = attributes
snmp_version = 2C
sourcetype = snmp_ta
split_bulk_output = 0
trap_rdns = 0
v3_authProtocol = usmHMACMD5AuthProtocol
v3_privProtocol = usmDESPrivProtocol

Re: Trying to get SNMP data into Splunk...

jkat54 — Wed, 16 Mar 2016 23:29:31 GMT

Are your snmp ports open? Defaults to 161 on the source (called destination in the a snmp app) and 162 on the Splunk server.

If your splunk server is linux and you're following best practices, you will not be able to open port 162 because only root can open ports below 1025.

Re: Trying to get SNMP data into Splunk...

cpt12tech — Wed, 16 Mar 2016 23:40:40 GMT

Thanks for the reply. I'm running Splunk on a Windows server. I'm monitoring the same source with another SNMP management app on a different server and am able to connect and poll the data. The firewall is turned off on the Windows server running Splunk and the Windows server I'm attempting to poll.

Re: Trying to get SNMP data into Splunk...

jkat54 — Thu, 17 Mar 2016 00:01:53 GMT

Any firewalls between the two on the network?

Use the following command to test:

start -> run -> cmd [enter/click ok]
telnet [ipaddress] [portnumber]

If you get a blinking cursor on a blank page, the port is open to the ipaddress you provided. Else you'll get a timeout error. If you do get the blinking cursor press ctrl + ] to escape out of the telnet session. Try from both servers to the other using the correct ports and ips.

Telnet might not be installed on the server, if not run this at command prompt

pkgmgr /iu:"TelnetClient"

Re: Trying to get SNMP data into Splunk...

pmeyerson — Thu, 17 Mar 2016 14:52:17 GMT

I can give a couple of things that helped me with the issues I had with SNMP Modular Input. I've found the process pretty cumbersome and required a lot of troubleshooting, at least from Windows.

check the splunkd log, errors will be reported there: \var\log\splunk\splunkd.log
if possible, you can install wireshark on the splunk server, and filter for UDP, this will allow you to see the SNMP packets leaving the splunk server and reply, if any, being received.
I was using Symantec Endpoint Protection 12.1.2100.2093. Even with the "Proactive Threat Protection" and "Network Threat Protection" disabled via the Symantec GUI, I found that outbound UDP was being denied. I installed on a testbed system without Symantec and the generic network error message in the Splunk log was fixed.
I also found that troubleshooting was a lot easier when going for a standard OID, like sysContact. This ruled out issues with getting the oid number to poll correct.

As an aside, if anyone has issues compiling their MIB files into python, you can actually drop an empty .py file into the custom mib directory, specify that filename in the SNMP Modular Input data input setup, and you will be able to poll. You just don't see the translation of oid values to key name. If the MIB file is specified in your input setup but not present polling will abort for that input.

Does your remote device report any errors like bad community name received or login failure, etc? It seems odd that you are specifying snmp_version = 2C but also including v3 auth settings?

Re: Trying to get SNMP data into Splunk...

cpt12tech — Thu, 17 Mar 2016 21:15:30 GMT

Tried the telnet, but there was no response. I searched around the internet and it seems that Telnet doesn't support UDP. I downloaded nmap, and scanned port 161 and it reports as open.

Really great tool.
https://nmap.org

Re: Trying to get SNMP data into Splunk...

cpt12tech — Tue, 29 Sep 2020 09:07:54 GMT

Installed Wireshark, another great tool, and I do see SNMP data coming into the Splunk server, but don't see any SNMP messages leaving the server. I've set up trap notifications on the server's I'm monitoring and assuming those are the SNMP messages being received. Polling should be sending out messages? Searched in the Splunkd log (index=_internal sourcetype=splunkd snmp) and did find some messages.

I set up two trap receivers in Splunk as well. One with localhost as the trap destination, and the other with the IP I'm using to send the SNMP messages to.

03-17-2016 15:44:53.661 -0600 ERROR ExecProcessor - message from "python "C:\Program Files\Splunk\etc\apps\snmp_ta\bin\snmp.py"" Failed to register transport and run dispatcher: bind() for ('localhost', 162) failed: [Errno 10013] An attempt was made to access a socket in a way forbidden by its access permissions snmp_stanza:snmp://Kaleido-Trap

03-17-2016 15:24:32.816 -0600 ERROR ExecProcessor - message from "python "C:\Program Files\Splunk\etc\apps\snmp_ta\bin\snmp.py"" Failed to register transport and run dispatcher: bind() for (u'192.168.1.102', 162) failed: [Errno 10013] An attempt was made to access a socket in a way forbidden by its access permissions snmp_stanza:snmp://Archive-Trap

I've noticed all the trap messages are coming in on port 161. For windows servers I'm using the Microsoft SNMP service. For other devices I have I use whatever they provide. Will the SNMP Modular Input receive from port 161?

I didn't specify an MIB, just wanted to see if I could get any data into the Splunk database. I haven't been able to compile custom MIB's using python and get command not found when trying to make the egg.

How did you find out that the Symantec Endpoint Protection was blocking the UDP port?

Re: Trying to get SNMP data into Splunk...

pmeyerson — Thu, 17 Mar 2016 22:59:23 GMT

Hi cpt12tech.

When you poll SNMP from splunk, you should see SNMP packets leaving your server get-request and snmp packets returning get-response I believe they show as. I can find a packet capture when I am at office tomorrow. Of particular note, drill into those SNMP packets and verify the UDP port number, etc is as expected.

You are running splunk as a service? Are you running it as LocalSystem or some other user account?
Does your organization push any Microsoft Group Policy, or other firewall/antivirus policy that would block outgoing UDP connections from the machine? Another thing to check would be to try disabling any SNMP service running directly on the server you have splunk installed on, it almost sounds like they may be conflicting?

SNMP Modular Input can listen on any UDP port - specify the port number when you are in the settings page for the Data Input.

This: http://www.alvestrand.no/objectid/1.3.6.1.2.1.1.html is a good set of OIDs to use for test polls. I don't think you have to specify the MIB file name, just pick one or more of those OIDs. They are standard MIBS and I believe included with SNMP Modular Input.

As to SEP - I have an unfortunate long history from previous work with SEP not behaving as it should. I took a chance and repeated my setup on a windows VM that did not have it installed - I put Microsoft Security Essentials on it instead. (use at your own risk). I also had problems with SEP client blocking command line API calls needed to setup splunk clustering.

HTH

Re: Trying to get SNMP data into Splunk...

jkat54 — Fri, 18 Mar 2016 00:26:39 GMT

Oh... Duh!!! Sorry. Netcat supports udp but it's Linux based... I'll be in the dunce corner

Re: Trying to get SNMP data into Splunk...

jkat54 — Fri, 18 Mar 2016 00:28:59 GMT

Your error indicates that port 162 can't be bound to on your Splunk server.

[Errno 10013] An attempt was made to access a socket in a way forbidden by its access permissions

My best guess is that you have SNMP Agent or other SNMP service running on the Splunk server and it has already bound to the port. Start -> run -> services.msc [return/click ok]. Look for SNMP services. You can also netstat -ano at command prompt to see if the port is already in use when splunkd is stopped.

To fix, disable whatever is bound to the port.

Other possibility includes splunkd running as underprivileged user (a best practice), and the user needs a Group Policy setting to allow it to bind to 162.

Oh and the easiest solution. Edit the SNMP modular input settings and use TRAP port 1162 or any other port instead of the default 162. or just disable the TRAP by U checking the box

Re: Trying to get SNMP data into Splunk...

monteirolopes — Tue, 31 May 2016 13:43:08 GMT

Thank you jkat54!
It works!

Re: Trying to get SNMP data into Splunk...

cpt12tech — Wed, 13 Jul 2016 21:23:21 GMT

Getting back into this to see if I can get it to work. Thanks for the help jkat.

Ran the command 'nstat -ano' and there was a service listening on 127.0.0.1. Stopped Splunkd and the listener went away, also went away when I disabled snmp_ta. I wanted to try changing the snmp ip that was listening on 161. Tried going into the app properties to check my settings but I don't see the GUI? Just a standard app GUI. Did something change or am I looking in the wrong place.

Also tried deleting snmp_ta app and reinstalling, but don't see the config gui.

Re: Trying to get SNMP data into Splunk...

pmeyerson — Wed, 13 Jul 2016 21:40:26 GMT

There isn't a seperate GUI component for SNMP Modular Input. You change the settings by adjusting the data inputs: Settings -> Data Inputs -> SNMP. The settings are defined per Input you define.

Re: Trying to get SNMP data into Splunk...

cpt12tech — Thu, 14 Jul 2016 18:32:15 GMT

Ahh.. thank you. That started me off on a rabbit trail. I get a 500 internal error when I go to data inputs... while trying to solve that issue, now the Splunk web service is refusing to start and I cannot connect to Splunk via a web browser. Oui.

Re: Trying to get SNMP data into Splunk...

cpt12tech — Thu, 14 Jul 2016 23:37:43 GMT

got the web browser working by typing in
splunk enable webserver
but haven't figured out why I get an internal server error when I go to the data inputs link...

Re: Trying to get SNMP data into Splunk...

cpt12tech — Wed, 22 Mar 2017 23:46:21 GMT

This was a bug in the current Splunk version. Waited for Splunk update, data inputs are working again. Finally getting back to SNMP implementation.

Re: Trying to get SNMP data into Splunk...

cpt12tech — Thu, 23 Mar 2017 00:20:19 GMT

I'm now receiving traps into Splunk.

I ran into an issue while working on SNMP where I couldn't see the Data Inputs in Splunk 6.4.2.
https://answers.splunk.com/answers/427846/clicking-on-data-inputs-shows-500-internal-server.html
This has been fixed in 6.4.3.

Just now getting back to working on getting SNMP traps into Splunk. I'm not sure what fixed the problem, however the server I'm running Splunk on has three NICs attached to 3 different subnets. The traps in the Data Input were set to localhost. Using wireshark I could see what address the SNMP traps were being addressed to, added that IP to my SNMP trap in the Data Inputs and data began to flow into splunk. Haven't gotten polling working, but that's a different issue.

Pretty sure I tried those different IP addresses in the past, but not 100% sure.

Many thanks to jkat54 & pmeyerson for their help!