topic Re: Extracting fields from snmp traps in Splunk Search

Extracting fields from snmp traps

swannie — Mon, 22 Aug 2016 17:51:52 GMT

Hi all,

I'm trying to extract key/value data from SNMP trap data logged to my splunk server. I have snmptrapd running in the background and logging to a file, which splunk is monitoring. All that is working great. The data that makes it into splunk looks like this:

2016-08-22 12:42:21 10.192.2.110 [UDP: [10.192.2.110]:32771->[10.10.15.76]]:
sysUpTimeInstance = 80:20:41:00.00  snmpTrapOID.0 = ciscoLwappDot11ClientMIBNotifs.2    cldcClientMacAddress.'......' = d4:b:1a:92:fc:8b    cldcClientWlanProfileName.'......' = SSID   cldcClientEntry.'......'.139 = 10.64.97.246 cldcApMacAddress.'......' = 58:f3:9c:c8:55:20   cldcClientEntry.'......'.139 = 0    cldcClientEntry.'......'.139 = 396

I'm trying to use the kv command like this: " ... | kv pairdelim="\t" kvdelim=" = " "which grabs some of the fields, but I think the problem is that there's extra snmp variable data that I need to get rid of. With the example above I get a key of "cldcClientEntry_________139" and value of "10.64.97.246" but what I want to get is "cldcClientEntry" and value of "10.64.97.246" ... so I think I need some way to stop matching the key as soon as it hits a period, but I'm not quite sure where/how to do that. Any suggestions?

Thanks,
Brian

Re: Extracting fields from snmp traps

javiergn — Mon, 22 Aug 2016 18:16:31 GMT

Try this instead:

your base search
| rex mode=sed "s/(\.\'[\.]+\')?(\.\d+)? =/ =/g"
| kv pairdelim="\t" kvdelim=" = "

It worked fine for me based on your sample above (and using several spaces instead of tabs because of the copy and paste):

| makeresults
| fields - _time
| eval _raw = "2016-08-22 12:42:21 10.192.2.110 [UDP: [10.192.2.110]:32771->[10.10.15.76]]:
 sysUpTimeInstance = 80:20:41:00.00    snmpTrapOID.0 = ciscoLwappDot11ClientMIBNotifs.2    cldcClientMacAddress.'......' = d4:b:1a:92:fc:8b    cldcClientWlanProfileName.'......' = SSID    cldcClientEntry.'......'.139 = 10.64.97.246    cldcApMacAddress.'......' = 58:f3:9c:c8:55:20    cldcClientEntry.'......'.139 = 0    cldcClientEntry.'......'.139 = 396"
| rex mode=sed "s/(\.\'[\.]+\')?(\.\d+)? =/ =/g"
| kv pairdelim="    " kvdelim=" = "

Output:

Re: Extracting fields from snmp traps

s2_splunk — Mon, 22 Aug 2016 18:17:15 GMT

Hi Brian,
from your sample event, it appears that the unwanted string is always .'.....', sometimes followed by .nnn so the simplest is probably removing that from the source events during indexing using SEDCMD for your source (or sourcetype) in props.conf, like so:

[source::yourlogfile]
SEDCMD-removeUnwantedStuff = s/\.\'[.'1234567890]+//g

Not that this will alter the events before they get indexed, so it will only apply to new events coming in.
If you don't want to do that, you can use the rex command at search time, like so:

yoursearch | rex mode=sed ddd "s/\.\'[.'1234567890]+//g"

I hope this helps.

Re: Extracting fields from snmp traps

swannie — Mon, 22 Aug 2016 19:15:59 GMT

Ahh... I didn't think about pre-processing those fields. I'll have to try it on a few different log entries and see what I need to tweak. Thanks for your assistance!

Re: Extracting fields from snmp traps

swannie — Mon, 22 Aug 2016 19:17:10 GMT

Thanks for the idea, but I might have gave a bad example. The text following the OID is variable and different (it happened to be the same in this log entry because it was all non-printable characters).

Re: Extracting fields from snmp traps

swannie — Mon, 22 Aug 2016 20:00:48 GMT

Okay, I'm having a problem wrapping my head around the SED command, and the interwebs aren't helping. Here's another log example:

2016-08-22 14:57:08 10.192.2.110 [UDP: [10.192.2.110]:32771->[10.10.15.76]]:
sysUpTimeInstance = 80:22:55:48.00  snmpTrapOID = ciscoLwappDot11ClientMIBNotifs.2  cldcClientMacAddress.'.*..q.' = e8:2a:ea:c6:71:d6   cldcClientWlanProfileName.'.*..q.' = RAMBLERS   cldcClientEntry.'..*..q' = 10.64.30.36  cldcApMacAddress.'.*..q.' = f0:7f:6:3e:20:30    cldcClientEntry.'..*..q' = 0    cldcClientEntry.'..*..q' = 330

Basically, I want to take everything from the first period up to (but not including) the equal sign and trash it. I tried to modify the original SED script you gave me, but it's not working as expected. Any thoughts?

Re: Extracting fields from snmp traps

swannie — Mon, 22 Aug 2016 20:31:24 GMT

I think I got it .. here's what I settled on:

rex mode=sed "s/\.([^\s]+) =/ =/g"

Re: Extracting fields from snmp traps

s2_splunk — Mon, 22 Aug 2016 22:13:03 GMT

Fair enough. Looks like you figured it out how to adjust the RegEx in SED to match any character from the first dot to the next equals sign.
You may want to consider removing that extraneous stuff before indexing the data, unless you need those characters for any other search. You could configure SEDCMD in your props.conf to achieve that.
Either way works fine, schema on the fly at work! 🙂