How to configure props.conf to remove any line NOT containing a certain string?

[essklau]

Hi.

All I want is the props.conf equivalent of this delete action from sed:

'/pattern/!d' 

That is it... just delete ANY line NOT containing "pattern".

Any takers?

[aweitzman · Accepted solution]

You can't use ^ as negation here. It's not a character class.

It might be easier to write two transforms, one that discards everything and one that keeps the ones you want:

props.conf
[your sourcetype]
TRANSFORMS-separate = discardall, keepsome


Order is important here. discardall comes first because all are applied in order, last one wins.

transforms.conf
[discardall]
REGEX=.*
DEST_KEY=queue
FORMAT=nullQueue


[keepsome]
REGEX=StringPattern
DEST_KEY=queue
FORMAT=indexQueue


[aweitzman]

Based on your comment, I promoted my comment so you could mark it as your answer.

[essklau]

settled to:

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = pattern
DEST_KEY = queue
FORMAT = indexQueue

pretty much weitzman's answer. Thanks!

[strive]

transforms.conf

[strip_pattern_lines]
REGEX = StringPattern
DEST_KEY = queue
FORMAT = nullQueue

props.conf

[your sourcetype]
TRANSFORMS-tonullqueue = strip_pattern_lines

Dont forget to restart the splunk after making above changes.

Update:

transforms.conf

[strip_pattern_lines]
REGEX = ^(?:[^ ]* ){4}(?!(PATTERN))
DEST_KEY = queue
FORMAT = nullQueue

props.conf

[your_sourcetype]
TRANSFORMS-tonullqueue = strip_pattern_lines

[strive]

I have edited my answer. Test regex once. I am not good at regex.

[essklau]

Sure:

2014-08-27 veryseriousinfo {zippity:boop.bop} hola23: PATTERN: Welcome to the jungle
2014-08-27 abunchofsilliness {bangarang:yes:arang} flip11: The news in Uganda is grim
2014-08-27 happygoluckyfool {drinkyourovaltine} lamp34: thisdoesnotmatter

I only want to index the lines containing "PATTERN"

[strive]

Agree. Thats why i have asked for sample log lines, so that we can suggest right configurations

[aweitzman]

(Just an FYI - This probably won't work on multiline events either, as @bmacias84 points out.)

[strive]

Can you post your sample log lines which you want to send to null Queue

[essklau]

I tried

[strip_pattern_lines]
REGEX = ^StringPattern
DEST_KEY = queue
FORMAT = nullQueue

with no success yet.

[bmacias84]

I don't believe that this works on multiline events.

[strive]

When you say delete, you do not want to index such lines. Is that right?

For this you need transforms.conf and props.conf