topic Re: Using kv or extract ain't working in Splunk Search

Using kv or extract ain't working

joshua_hart — Fri, 19 Jul 2013 18:39:43 GMT

I have a McAfee Firewall Appliance log (Sidewinder for those of us familiar with the tool) that comes to Splunk by way of syslog and I'm trying to extract k/v from the log. Here's an example:

2013-07-19T12:31:41-04:00 Firewall_host auditd: 2013-07-19 16:31:41 +0000 f_ntp_daemon a_server t_netprobe p_major pid: 24912 logid: 0 cmd: 'ntpd' hostname: Firewall_host.fq.dn event: probe attempt srcip: 0.0.0.0 srcport: 123 srczone: internal protocol: 17 dstip: 0.0.0.0 dstport: 123 attackip: 0.0.0.0 attackzone: internal reason: Received a connection attempt destined for a service that the current policy does not support.

I can't figure out what to use as the pairdelim and kvdelim values. I've tried pairdelim=" ", kvdelim=":"; pairdelim="p_major ", kvdelim=":"; pairdelim="p_major\s", kvdelim=":\s"--and every variation in between. Usually I'll place a '| table, srcip' on the end to see if it extracts the fields, but no such luck.

-Josh

Re: Using kv or extract ain't working

tgow — Fri, 19 Jul 2013 19:42:04 GMT

The key-value extraction works best when every field is set to the key=value pair. For instance the server name is not in a kv pair so this might be why it is not working. I would recommend that you just build the fields using rex, erex or the field extractor.

Here is a quick shot at the regular expression to build the fields:

sourcetype=sidewinder | rex field=_raw "\s+srcip:\s+(?\S+)\s+" | table _time, srcip

Re: Using kv or extract ain't working

bmacias84 — Fri, 19 Jul 2013 21:47:40 GMT

Unfortunatly since there is nto constant delime you will have to use regex, I recommend doing so in a transform.

Re: Using kv or extract ain't working

joshua_hart — Sat, 10 Aug 2013 00:09:59 GMT

I'll give that a try when I get in to work on Monday. Thanks!

Re: Using kv or extract ain't working

joshua_hart — Mon, 12 Aug 2013 13:10:17 GMT

I was able to extract all the fields using rex expressions. Now, how do I translate that into a permanent thing? I tried adding EXTRACT expressions to a props.conf file that is present on my HF, and SH systems, but when I search for that sourcetype, none of the fields are present in the field list. I'm using the same regexes in props.conf as I am in the rex search. Search is below:

sourcetype=sidewinder | rex field=_raw "\s+reason:\s+(?<reason>.+)$" | rex field=_raw "\spid:\s+(?<pid>\S+)\s" | rex field=_raw "\slogid:\s+(?<logid>\S+)\s" | rex field=_raw "\scmd:\s+(?<cmd>\S+)\s" | rex field=_raw "\shostname:\s+(?<hostname>\S+)\s" | rex field=_raw "\sevent:\s+(?<event>\S+)\s" | rex field=_raw "\ssrcip:\s+(?<srcip>\S+)\s" | rex field=_raw "\ssrcport:\s+(?<srcport>\S+)\s" | rex field=_raw "\ssrczone:\s+(?<srczone>\S+)\s" | rex field=_raw "\sdstip:\s+(?<dstip>\S+)\s" | rex field=_raw "\sdstport:\s+(?<dstport>\S+)\s" | rex field=_raw "\sdstzone:\s+(?<dstzone>\S+)\s" | rex field=_raw "\sattackip:\s+(?<attackip>\S+)\s" | rex field=_raw "\sattackzone:\s+(?<attackzone>\S+)\s"