How to send snmp traps from my Linux machine to a Splunk indexer server on threshold values?

[richaGindodia]

I've been trying to do this for the past three days.

I want my Linux machine to send snmp traps to splunk server on threshhold values
Here is what I am doing
1. Installing snmp on my machine
2. Installing snmp and snmptrapd on splunk server
3. on my machine listing trapsink as splunk servers public ip
4. giving the same ip in snmptrap command

I followed the same steps to get the trap working on my network systems, and I was able to send traps from my machine to other machines on same network. I read the documentation (http://docs.splunk.com/Documentation/Splunk/6.2.2/Data/SendSNMPeventstoSplunk ) on how to send snmp traps to Splunk server, but they have the same steps. What do i do? I am not interested in the add-on for snmp that splunk provides because I don't want splunk to poll my system. I want to do it the other way round. Please help.

[arjunganta]

Thanks brother, I have same question in my mind. I have referred so many websites.But I did not get any proper answer. At last question have me answer.

[dwaddle · Accepted solution]

Let's start out by separating your concerns and working out a logical sequence of first principles to build on.

1. Make Splunk able to receive a trap and index it.
2. Make a server able to send ANY trap successfully to Splunk (and see it get indexed)
3. Make a server able to send traps based on a threshold

For #1, you can choose to use the native snmptrapd on the host running Splunk, or you can choose to use the modular input. Given the options, I would personally choose to run snmptrapd for the following reasons.

• It separates the concerns of listening on the socket from indexing the data.
• Limits the amount of code that must run as root to support the "standard" UDP port of 162. That is, Splunk does not have to run as root, and there is no iptables port redirection magic required in order to map packets arriving at 162 to a higher port.
  
• It makes debugging easier, because I have a file that traps should land in -- regardless of whether Splunk is properly configured to process that file or not.
• It minimizes the possibility of missing SNMP traps due to Splunk restarts. Similar to syslog, the snmptrapd daemon will need to be restarted far less often than Splunk and it will restart orders of magnitude faster.
• MIBS are suddenly orders of magnitude easier to deal with, because we're not fooling with the obtuse (or is it abstruse?) MIB format that pySNMP uses.

So given the decision to use the native snmptrapd then our list of steps gets refined a little.

1. Configure snmptrapd to receive traps and write them to a logfile
2. Make a server able to send ANY trap successfully to snmptrapd
3. Configure Splunk to be able to monitor snmptrapd's log file and see it get indexed
4. Make a server able to send traps based on a threshold

---

STEP THE FIRST

I'll use my CentOS 6.2 box as a guinea pig here. We start by making sure Net-SNMP is installed.

sudo yum install -y net-snmp net-snmp-utils

Now let's configure it. In /etc/snmp/snmptrapd.conf:

# Example configuration file for snmptrapd
#
# No traps are handled by default, you must edit this file!
#
# authCommunity   log,execute,net public
# traphandle SNMPv2-MIB::coldStart    /usr/bin/bin/my_great_script cold

authCommunity log public

And in /etc/sysconfig/snmptrapd:

# snmptrapd command line options
# OPTIONS="-Lsd -p /var/run/snmptrapd.pid"
OPTIONS="-A -Lf /var/log/snmptrapd.log -p /var/run/snmptrapd.pid"

We can now do a service snmptrapd start and see the daemon start, and see it log a little intro line to the snmptrapd.log file:

[root@localhost log]# tail -f snmptrapd.log 
Created directory: /var/lib/net-snmp/mib_indexes
NET-SNMP version 5.5

---

STEP THE SECOND

Now we do a simple test to see if another host (my Mac) can send an arbitrary trap and have it show up in the log.

snmptrap -v 1 -c public 192.168.96.190 .1.3.6.1.6.3 "" 0 0 coldStart.0

And hey, look, I see it in the log on the CentOS box:

2015-08-09 09:50:12 0.0.0.0(via UDP: [192.168.96.1]:64763->[192.168.96.190]) TRAP, SNMP v1, community public
    SNMPv2-SMI::snmpModules Cold Start Trap (0) Uptime: 0:00:00.00

If your trap didn't make it, then you should be looking at things like firewalls and so forth in order to debug the issue.

---

STEP THE THIRD

So, this should be as simple as adding a Data input in the Splunk UI, or editing inputs.conf directly. I'll go with inputs.conf because that's how I roll.

[monitor:///var/log/snmptrapd.log]
sourcetype=snmptrapd

And I restart Splunk. These events are now visible in Splunk.

---

STEP THE FOURTH

This is where it actually gets interesting. But, it is also the "least Splunkadelic" part of the question. For most SNMP agents, the idea of "sending based on a threshold" is not a common concept. So, how you go about this part may vary from agent to agent. Since I have NET-SNMP, and it is thoroughly documented in the world, I'm going to lean heavily on a couple of references.

http://serverfault.com/questions/248332/set-up-snmp-trap-for-disk-usage
http://www.net-snmp.org/wiki/index.php/TUT:DisMan_Monitoring

In NET-SNMP land, the general idea of a "threshold trap" is done using the DisMan Event MIB. Following the steps in the net-snmp wiki link, I added this to /etc/snmp/snmpd.conf:

createUser    lemonitor SHA lepassword  AES
 rouser        lemonitor
 iquerySecName lemonitor

trap2sink 127.0.0.1 public

monitor -r 5 machineTooBusy hrProcessorLoad > 2

And then ran a service snmpd restart. Because I set the threshold so low, I almost immediately got a trap:

2015-08-09 10:10:28 localhost [UDP: [127.0.0.1]:37402->[127.0.0.1]]:
DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (6520) 0:01:05.20  SNMPv2-MIB::snmpTrapOID.0 = OID: DISMAN-EVENT-MIB::mteTriggerFired  DISMAN-EVENT-MIB::mteHotTrigger.0 = STRING: machineTooBusy  DISMAN-EVENT-MIB::mteHotTargetName.0 = STRING:  DISMAN-EVENT-MIB::mteHotContextName.0 = STRING:     DISMAN-EVENT-MIB::mteHotOID.0 = OID: HOST-RESOURCES-MIB::hrProcessorLoad.196608 DISMAN-EVENT-MIB::mteHotValue.0 = INTEGER: 51

---

And there you have it.

[arjunganta]

You rocked it. I have been struggling to configure SNMP traps for 4 months. But you gave me right answer. It is working fine for me. Thank u thanks thank u alot.

[arunsunny]

Hey,

Can we write multiple Options in one /etc/sysconfig/snmptrapd ?

Also,

For Example -> I have a requirement where I need to receive SNMP traps from two different sources and write those traps messages into two different log files? Please suggest what is the best way to do this?

Thanks,
Arun

[jnhth]

 @arunsunny  Did you find a config for send traps to diff logs files?

[stanley1191]

Very useful. Thanks!!

[woodcock]

Bravo!!!!!

[woodcock]

The Splunk SNMP Modular Input can receive TRAPs, too (maybe it couldn't when you first looked at it but it definitely can now):

https://splunkbase.splunk.com/app/1537/