topic Extract variable number of fields in Splunk Search

Extract variable number of fields

grist — Tue, 20 Sep 2011 06:18:01 GMT

I have a Smarts Audit Log that I am trying to do a search time field extraction for. Most of the lines are fairly regularly formatted using tabs (\t) as a seperator.

The problem I have is that while most of the lines have 8 fields, there are some that only have 5 and I'm not sure how to deal with that so they can all live happily in the same report. I've read a few of the suggested fixes but they seem to rely on there being a particular description field to tailor different regexes to different line formats. Mine just come up short if the fields aren't there.

Any suggestions or pointers gratefully accepted. 🙂

Re: Extract variable number of fields

MarioM — Tue, 20 Sep 2011 06:21:01 GMT

if you could post an example of your log that would be helpful

Re: Extract variable number of fields

Simeon — Tue, 20 Sep 2011 20:11:32 GMT

You can have multiple extractions occur against different types of events for a specific sourcetype. This is not uncommon. The key here is that you must find a way to differentiate the lines that only have 5 fields. If it is formatted differently, that will be pretty straightforward. If it is formatted with only delimiters, then you can create a regex that operates on 5 values instead of 8. Posting a data sample will allow others to help further.

Re: Extract variable number of fields

grist — Wed, 21 Sep 2011 00:41:40 GMT

The only difference between the lines is the number of fields. They are all tab separated so it's pretty easy to split them with a regex.

I'll have a go with 2 regexes. I'm thinking I need to so something like what's described in the first answer to http://splunk-base.splunk.com/answers/23274/parsing-variable-fields-in-a-log-file but I'm not 100% on how to do it so all the results will show in the same search.