How to filter or extract fields before indexing time

[crazyeva]

I have got very large orginal data, with events strictly formatted as "f1,f2,f3,..."
most of the fields are meaningless: "0,f2,0,0,0,0,f7,0,f9,..." i only want f2,f7,f9
Can I filter fields before indexing, drop unnecessary data, avoid reach license limit?

[Ayn · Accepted solution]

Yes, as long as you can formulate a regular expression that defines how Splunk should include or exclude data. You can either to nullQueue routing (= drop events altogether) or

Docs on how to use each: nullQueue routing - http://docs.splunk.com/Documentation/Splunk/6.0/Forwarding/Routeandfilterdatad
SEDCMD - http://docs.splunk.com/Documentation/Splunk/latest/admin/Propsconf - see specification on SEDCMD at the middle of the page.

[kristian_kolb · Accepted solution]

source file

2013-11-01 11:11:11 f1 f2 f3 f4 f5 f6
2013-11-02 13:15:11 d1 d2 d3 d4 d5 d6
2013-11-02 14:23:22 e1 e2 e3 e4 e5 e6
2013-11-03 12:23:21 g1 g2 g3 g4 g5 g6

props.conf

[your_sourcetype]
TRANSFORMS-blah = keep235

transforms.conf

[keep235]
DEST_KEY = _raw
REGEX = ^(\S+)\s+(\S+)\s+(\S+)\s+(\S+)\s+(\S+)\s+(\S+)\s+(\S+)\s+(\S+)
FORMAT = $1 $2 $4 $5 $7

Result:

03/11/2013 12:23:21.000  2013-11-03 12:23:21 g2 g3 g5
02/11/2013 14:23:22.000  2013-11-02 14:23:22 e2 e3 e5

^Splunk parsed timestamp  ^event timestamp     ^less columns/fields

Hope this helps,

/K

[crazyeva]

Thank you, that's sweet!

[Ayn · Accepted solution]

Yes, as long as you can formulate a regular expression that defines how Splunk should include or exclude data. You can either to nullQueue routing (= drop events altogether) or

Docs on how to use each: nullQueue routing - http://docs.splunk.com/Documentation/Splunk/6.0/Forwarding/Routeandfilterdatad
SEDCMD - http://docs.splunk.com/Documentation/Splunk/latest/admin/Propsconf - see specification on SEDCMD at the middle of the page.

[crazyeva]

THANK YOU!
I have read some of related answers, and thought SEDCMD was just able to drop the content after a tag, nullQueue was "at event level"
Than i have to dig into "sed" tool. is there a awkcmd?