topic Re: Filter out event before indexing using REGEX in Security

Filter out event before indexing using REGEX

damucka — Tue, 08 Jan 2019 12:29:40 GMT

Hello,

I would like to filter out the log entries (before indexing) which are created in the debugging mode. They can be identified by the letter "d" in my logfile:

[293672]{-1}[-1/-1] 2019-01-08 11:24:29.210542 d PITRestart       LogReplayCoordinator.cpp(02658) : replayStepFinished(), logPos=0x1d2e2bab65b8
Line 1
Line 2
Line N

In the above there is a header line, which has always a similar structure and the 3 additional lines of the event. The header line includes always the pattern:

11:24:29.210542 d

where the letter can be "i" for info, "e" for error, "d" for debug, etc.
Now, I would like to filter out all the events (header line plus belonging lines) where the loglevel is "d".
How would I achieve this?

On the example of the below:

[259451]{-1}[-1/-1] 2019-01-08 11:24:29.213984 a STATS_CTRL       Schedule.cpp(00106) : Enter void StatisticsService::ScheduleReloader::updateExecutionQueue(StatisticsService::ExecutionQueue&)
Arg this = 0x00007e382849b0a8
[293672]{-1}[-1/-1] 2019-01-08 11:24:29.210542 d PITRestart       LogReplayCoordinator.cpp(02658) : replayStepFinished(), logPos=0x1d2e2bab65b8
Line 1
Line 2
Line N
[265685]{242705}[1340/-1] 2019-01-08 11:24:29.144534 e StatementResourc StatementResourceTracking.cc(00217) : statistics collection is not finished: stmt=0x00007e8643473400, stmtid=1042411823155799

I would like to get rid of the second event and have only the following after indexing:

[259451]{-1}[-1/-1] 2019-01-08 11:24:29.213984 a STATS_CTRL       Schedule.cpp(00106) : Enter void StatisticsService::ScheduleReloader::updateExecutionQueue(StatisticsService::ExecutionQueue&)
Arg this = 0x00007e382849b0a8
[265685]{242705}[1340/-1] 2019-01-08 11:24:29.144534 e StatementResourc StatementResourceTracking.cc(00217) : statistics collection is not finished: stmt=0x00007e8643473400, stmtid=1042411823155799

Could you help me with the props.conf, transforms.conf and especially the corresponding REGEX for that?

Kind Regards,
Kamil

Re: Filter out event before indexing using REGEX

harsmarvania57 — Tue, 08 Jan 2019 13:02:58 GMT

Hi @damucka,

Please try below config on Indexer/Heavy Forwarder whichever comes first from Universal Forwarder.

props.conf

[yoursourcetype]
TRANSFORMS-eliminatedebug = setnull

transforms.conf

[setnull]
REGEX=(?m)\d{4}\-\d{2}\-\d{2}\s\d{2}\:\d{2}\:\d{2}\.\d{6}\sd
DEST_KEY=queue
FORMAT=nullQueue

EDIT: If Line 1, Line 2 ... Line N are events with Debug line then you can try below transforms.conf

[setnull]
REGEX=(?s)\d{4}\-\d{2}\-\d{2}\s\d{2}\:\d{2}\:\d{2}\.\d{6}\sd.*\n
DEST_KEY=queue
FORMAT=nullQueue

Re: Filter out event before indexing using REGEX

inventsekar — Tue, 08 Jan 2019 13:11:39 GMT

can we simply use "210542 d"
REGEX=210542\sd

Re: Filter out event before indexing using REGEX

lakshman239 — Tue, 08 Jan 2019 13:14:40 GMT

You would also need to ensure that the line with debug, "d" and the following line 1, 2 ... should be configured/treated as multi-line, so as to remove all of them together with nullQueue

Re: Filter out event before indexing using REGEX

harsmarvania57 — Tue, 08 Jan 2019 13:15:04 GMT

Hi @inventsekar,

I guess datetime value which is provided is for sample only so 210542\sd will not work because it is subsecond and every event will have different values for subsecond.

Re: Filter out event before indexing using REGEX

harsmarvania57 — Tue, 08 Jan 2019 13:16:30 GMT

I am guessing that 1,2 ... those are same type of events and not multiline values.

Re: Filter out event before indexing using REGEX

damucka — Tue, 08 Jan 2019 13:30:07 GMT

At the moment we have 3 events there, each starting with the header line. Not sure about the multiline values ... I did nothing with the configuration, Splunk recognizes it itself. So, to make it clear - I would like to get rid of the second event.

Re: Filter out event before indexing using REGEX

harsmarvania57 — Tue, 08 Jan 2019 13:32:13 GMT

I have provided 2 different regex for transforms.conf, you can try that and let us know if it will not work.

Re: Filter out event before indexing using REGEX

damucka — Tue, 08 Jan 2019 13:56:44 GMT

Thank you. I will test and let you know.
Is it possible to match several sourcetypes in props.conf to the same entry in transforms.conf? Like below:

[yoursourcetype1]
TRANSFORMS-eliminatedebug = setnull

[yoursourcetype2]
TRANSFORMS-eliminatedebug = setnull

[yoursourcetype3]
TRANSFORMS-eliminatedebug = setnull

Re: Filter out event before indexing using REGEX

harsmarvania57 — Tue, 08 Jan 2019 14:06:21 GMT

Yes you can