How to configure props.conf and transforms.conf to only whitelist events that start with a timestamp and have a LogLevel?

[Norling80]

Hi

We have a very volatile log source which we today control by sending unwanted events to the nullQueue. This is good, but not 100% waterproof and reliable due to the nature of the log source.

Is there a way to achieve the same behavior, but instead whitelist events? Basically, we only want to index events that starts with a timestamp and has a LogLevel, examples below:

2015-10-21 12:10:05,786 +0200 INFO .... "the rest of the event..."
2015-10-21 12:10:05,786 +0200 WARNING .... "the rest of the event..."
2015-10-21 12:10:05,786 +0200 ERROR .... "the rest of the event..."
2015-10-21 12:10:05,786 +0200 SEVERE .... "the rest of the event..."

cheers
Magnus

[woodcock]

Yes, you do this by first sending ALL events to nullQueue and then pull back just the events that you like. NOTE: I did not test this RegEx.

In props.conf:

[YourSourceTypeHere]
TRANSFORMS-set= setnull,setparsing

In transforms.conf:

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = ^\d{4}-\d{2}-\d{2} \d{1-2}:\d{2}:\d{2},\d{3}\s+(?:[+\-0-9]*\s+)?(INFO|WARNING|ERROR|SEVERE)
DEST_KEY = queue
FORMAT = indexQueue