All Apps and Add-ons
Highlighted

Trying to get SNMP data into Splunk...

Contributor

I'm having problems getting any SNMP data into Splunk using the SNMP Modular Input. I've set up a Polling Input that is polling a Windows 2008 Server. So far no data has shown up in Splunk.

I've checked for errors using:
index=_internal ExecProcessor error snmp.py
and don't see any results.

[snmp://Archive]
communitystring = public
destination = 192.168.1.101
dobulkget = 1
dogetsubtree = 1
host = Archive
ipv6 = 0
snmpmode = attributes
snmp
version = 2C
sourcetype = snmpta
split
bulkoutput = 0
trap
rdns = 0
v3authProtocol = usmHMACMD5AuthProtocol
v3
privProtocol = usmDESPrivProtocol

0 Karma
Highlighted

Re: Trying to get SNMP data into Splunk...

SplunkTrust
SplunkTrust

Are your snmp ports open? Defaults to 161 on the source (called destination in the a snmp app) and 162 on the Splunk server.

If your splunk server is linux and you're following best practices, you will not be able to open port 162 because only root can open ports below 1025.

Highlighted

Re: Trying to get SNMP data into Splunk...

Contributor

Thanks for the reply. I'm running Splunk on a Windows server. I'm monitoring the same source with another SNMP management app on a different server and am able to connect and poll the data. The firewall is turned off on the Windows server running Splunk and the Windows server I'm attempting to poll.

0 Karma
Highlighted

Re: Trying to get SNMP data into Splunk...

SplunkTrust
SplunkTrust

Any firewalls between the two on the network?

Use the following command to test:

start -> run -> cmd [enter/click ok]
telnet [ipaddress] [portnumber]

If you get a blinking cursor on a blank page, the port is open to the ipaddress you provided. Else you'll get a timeout error. If you do get the blinking cursor press ctrl + ] to escape out of the telnet session. Try from both servers to the other using the correct ports and ips.

Telnet might not be installed on the server, if not run this at command prompt

pkgmgr /iu:"TelnetClient"

0 Karma
Highlighted

Re: Trying to get SNMP data into Splunk...

Contributor

Tried the telnet, but there was no response. I searched around the internet and it seems that Telnet doesn't support UDP. I downloaded nmap, and scanned port 161 and it reports as open.

Really great tool.
https://nmap.org

0 Karma
Highlighted

Re: Trying to get SNMP data into Splunk...

SplunkTrust
SplunkTrust

Oh... Duh!!! Sorry. Netcat supports udp but it's Linux based... I'll be in the dunce corner

0 Karma
Highlighted

Re: Trying to get SNMP data into Splunk...

Path Finder

I can give a couple of things that helped me with the issues I had with SNMP Modular Input. I've found the process pretty cumbersome and required a lot of troubleshooting, at least from Windows.

  • check the splunkd log, errors will be reported there: \var\log\splunk\splunkd.log
  • if possible, you can install wireshark on the splunk server, and filter for UDP, this will allow you to see the SNMP packets leaving the splunk server and reply, if any, being received.
  • I was using Symantec Endpoint Protection 12.1.2100.2093. Even with the "Proactive Threat Protection" and "Network Threat Protection" disabled via the Symantec GUI, I found that outbound UDP was being denied. I installed on a testbed system without Symantec and the generic network error message in the Splunk log was fixed.
  • I also found that troubleshooting was a lot easier when going for a standard OID, like sysContact. This ruled out issues with getting the oid number to poll correct.

As an aside, if anyone has issues compiling their MIB files into python, you can actually drop an empty .py file into the custom mib directory, specify that filename in the SNMP Modular Input data input setup, and you will be able to poll. You just don't see the translation of oid values to key name. If the MIB file is specified in your input setup but not present polling will abort for that input.

Does your remote device report any errors like bad community name received or login failure, etc? It seems odd that you are specifying snmp_version = 2C but also including v3 auth settings?

Highlighted

Re: Trying to get SNMP data into Splunk...

Contributor

Installed Wireshark, another great tool, and I do see SNMP data coming into the Splunk server, but don't see any SNMP messages leaving the server. I've set up trap notifications on the server's I'm monitoring and assuming those are the SNMP messages being received. Polling should be sending out messages? Searched in the Splunkd log (index=_internal sourcetype=splunkd snmp) and did find some messages.

I set up two trap receivers in Splunk as well. One with localhost as the trap destination, and the other with the IP I'm using to send the SNMP messages to.

03-17-2016 15:44:53.661 -0600 ERROR ExecProcessor - message from "python "C:\Program Files\Splunk\etc\apps\snmpta\bin\snmp.py"" Failed to register transport and run dispatcher: bind() for ('localhost', 162) failed: [Errno 10013] An attempt was made to access a socket in a way forbidden by its access permissions snmpstanza:snmp://Kaleido-Trap

03-17-2016 15:24:32.816 -0600 ERROR ExecProcessor - message from "python "C:\Program Files\Splunk\etc\apps\snmpta\bin\snmp.py"" Failed to register transport and run dispatcher: bind() for (u'192.168.1.102', 162) failed: [Errno 10013] An attempt was made to access a socket in a way forbidden by its access permissions snmpstanza:snmp://Archive-Trap

I've noticed all the trap messages are coming in on port 161. For windows servers I'm using the Microsoft SNMP service. For other devices I have I use whatever they provide. Will the SNMP Modular Input receive from port 161?

I didn't specify an MIB, just wanted to see if I could get any data into the Splunk database. I haven't been able to compile custom MIB's using python and get command not found when trying to make the egg.

How did you find out that the Symantec Endpoint Protection was blocking the UDP port?

0 Karma
Highlighted

Re: Trying to get SNMP data into Splunk...

Path Finder

Hi cpt12tech.

When you poll SNMP from splunk, you should see SNMP packets leaving your server get-request and snmp packets returning get-response I believe they show as. I can find a packet capture when I am at office tomorrow. Of particular note, drill into those SNMP packets and verify the UDP port number, etc is as expected.

You are running splunk as a service? Are you running it as LocalSystem or some other user account?
Does your organization push any Microsoft Group Policy, or other firewall/antivirus policy that would block outgoing UDP connections from the machine? Another thing to check would be to try disabling any SNMP service running directly on the server you have splunk installed on, it almost sounds like they may be conflicting?

SNMP Modular Input can listen on any UDP port - specify the port number when you are in the settings page for the Data Input.

This: http://www.alvestrand.no/objectid/1.3.6.1.2.1.1.html is a good set of OIDs to use for test polls. I don't think you have to specify the MIB file name, just pick one or more of those OIDs. They are standard MIBS and I believe included with SNMP Modular Input.

As to SEP - I have an unfortunate long history from previous work with SEP not behaving as it should. I took a chance and repeated my setup on a windows VM that did not have it installed - I put Microsoft Security Essentials on it instead. (use at your own risk). I also had problems with SEP client blocking command line API calls needed to setup splunk clustering.

HTH

Highlighted

Re: Trying to get SNMP data into Splunk...

SplunkTrust
SplunkTrust

Your error indicates that port 162 can't be bound to on your Splunk server.

[Errno 10013] An attempt was made to access a socket in a way forbidden by its access permissions

My best guess is that you have SNMP Agent or other SNMP service running on the Splunk server and it has already bound to the port. Start -> run -> services.msc [return/click ok]. Look for SNMP services. You can also netstat -ano at command prompt to see if the port is already in use when splunkd is stopped.

To fix, disable whatever is bound to the port.

Other possibility includes splunkd running as underprivileged user (a best practice), and the user needs a Group Policy setting to allow it to bind to 162.

Oh and the easiest solution. Edit the SNMP modular input settings and use TRAP port 1162 or any other port instead of the default 162. or just disable the TRAP by U checking the box alt text

Speak Up for Splunk Careers!

We want to better understand the impact Splunk experience and expertise has has on individuals' careers, and help highlight the growing demand for Splunk skills.