topic Re: How to configure props.conf to remove any line NOT containing a certain string? in Getting Data In

How to configure props.conf to remove any line NOT containing a certain string?

essklau — Fri, 05 Sep 2014 16:34:32 GMT

Hi.

All I want is the props.conf equivalent of this delete action from sed:

'/pattern/!d'

That is it... just delete ANY line NOT containing "pattern".

Any takers?

Re: How to configure props.conf to remove any line NOT containing a certain string?

strive — Fri, 05 Sep 2014 17:29:28 GMT

When you say delete, you do not want to index such lines. Is that right?

For this you need transforms.conf and props.conf

Re: How to configure props.conf to remove any line NOT containing a certain string?

strive — Fri, 05 Sep 2014 17:33:24 GMT

transforms.conf

[strip_pattern_lines]
REGEX = StringPattern
DEST_KEY = queue
FORMAT = nullQueue

props.conf

[your sourcetype]
TRANSFORMS-tonullqueue = strip_pattern_lines

Dont forget to restart the splunk after making above changes.

Update:

transforms.conf

[strip_pattern_lines]
REGEX = ^(?:[^ ]* ){4}(?!(PATTERN))
DEST_KEY = queue
FORMAT = nullQueue

props.conf

[your_sourcetype]
TRANSFORMS-tonullqueue = strip_pattern_lines

Re: How to configure props.conf to remove any line NOT containing a certain string?

bmacias84 — Fri, 05 Sep 2014 17:55:03 GMT

I don't believe that this works on multiline events.

Re: How to configure props.conf to remove any line NOT containing a certain string?

essklau — Mon, 28 Sep 2020 17:30:19 GMT

I tried

[strip_pattern_lines]
REGEX = ^StringPattern
DEST_KEY = queue
FORMAT = nullQueue

with no success yet.

Re: How to configure props.conf to remove any line NOT containing a certain string?

strive — Fri, 05 Sep 2014 19:13:10 GMT

Can you post your sample log lines which you want to send to null Queue

Re: How to configure props.conf to remove any line NOT containing a certain string?

aweitzman — Fri, 05 Sep 2014 19:17:00 GMT

You can't use ^ as negation here. It's not a character class.

It might be easier to write two transforms, one that discards everything and one that keeps the ones you want:

props.conf
[your sourcetype] TRANSFORMS-separate = discardall, keepsome

Order is important here. discardall comes first because all are applied in order, last one wins.

transforms.conf
[discardall] REGEX=.* DEST_KEY=queue FORMAT=nullQueue

[keepsome] REGEX=StringPattern DEST_KEY=queue FORMAT=indexQueue

Re: How to configure props.conf to remove any line NOT containing a certain string?

aweitzman — Fri, 05 Sep 2014 19:18:43 GMT

(Just an FYI - This probably won't work on multiline events either, as @bmacias84 points out.)

Re: How to configure props.conf to remove any line NOT containing a certain string?

strive — Fri, 05 Sep 2014 19:24:19 GMT

Agree. Thats why i have asked for sample log lines, so that we can suggest right configurations

Re: How to configure props.conf to remove any line NOT containing a certain string?

essklau — Fri, 05 Sep 2014 20:07:29 GMT

Sure:

2014-08-27 veryseriousinfo {zippity:boop.bop} hola23: PATTERN: Welcome to the jungle
2014-08-27 abunchofsilliness {bangarang:yes:arang} flip11: The news in Uganda is grim
2014-08-27 happygoluckyfool {drinkyourovaltine} lamp34: thisdoesnotmatter

I only want to index the lines containing "PATTERN"

Re: How to configure props.conf to remove any line NOT containing a certain string?

strive — Fri, 05 Sep 2014 20:20:38 GMT

I have edited my answer. Test regex once. I am not good at regex.

Re: How to configure props.conf to remove any line NOT containing a certain string?

essklau — Mon, 08 Sep 2014 13:29:12 GMT

settled to:

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = pattern
DEST_KEY = queue
FORMAT = indexQueue

pretty much weitzman's answer. Thanks!

Re: How to configure props.conf to remove any line NOT containing a certain string?

aweitzman — Mon, 08 Sep 2014 13:37:47 GMT

Based on your comment, I promoted my comment so you could mark it as your answer.