Security

Filter out event before indexing using REGEX

damucka
Builder

Hello,

I would like to filter out the log entries (before indexing) which are created in the debugging mode. They can be identified by the letter "d" in my logfile:

[293672]{-1}[-1/-1] 2019-01-08 11:24:29.210542 d PITRestart       LogReplayCoordinator.cpp(02658) : replayStepFinished(), logPos=0x1d2e2bab65b8
Line 1
Line 2
Line N

In the above there is a header line, which has always a similar structure and the 3 additional lines of the event. The header line includes always the pattern:

11:24:29.210542 d

where the letter can be "i" for info, "e" for error, "d" for debug, etc.
Now, I would like to filter out all the events (header line plus belonging lines) where the loglevel is "d".
How would I achieve this?

On the example of the below:

[259451]{-1}[-1/-1] 2019-01-08 11:24:29.213984 a STATS_CTRL       Schedule.cpp(00106) : Enter void StatisticsService::ScheduleReloader::updateExecutionQueue(StatisticsService::ExecutionQueue&)
Arg this = 0x00007e382849b0a8
[293672]{-1}[-1/-1] 2019-01-08 11:24:29.210542 d PITRestart       LogReplayCoordinator.cpp(02658) : replayStepFinished(), logPos=0x1d2e2bab65b8
Line 1
Line 2
Line N
[265685]{242705}[1340/-1] 2019-01-08 11:24:29.144534 e StatementResourc StatementResourceTracking.cc(00217) : statistics collection is not finished: stmt=0x00007e8643473400, stmtid=1042411823155799

I would like to get rid of the second event and have only the following after indexing:

[259451]{-1}[-1/-1] 2019-01-08 11:24:29.213984 a STATS_CTRL       Schedule.cpp(00106) : Enter void StatisticsService::ScheduleReloader::updateExecutionQueue(StatisticsService::ExecutionQueue&)
Arg this = 0x00007e382849b0a8
[265685]{242705}[1340/-1] 2019-01-08 11:24:29.144534 e StatementResourc StatementResourceTracking.cc(00217) : statistics collection is not finished: stmt=0x00007e8643473400, stmtid=1042411823155799

Could you help me with the props.conf, transforms.conf and especially the corresponding REGEX for that?

Kind Regards,
Kamil

Tags (1)
0 Karma
1 Solution

harsmarvania57
SplunkTrust
SplunkTrust

Hi @damucka,

Please try below config on Indexer/Heavy Forwarder whichever comes first from Universal Forwarder.

props.conf

[yoursourcetype]
TRANSFORMS-eliminatedebug = setnull

transforms.conf

[setnull]
REGEX=(?m)\d{4}\-\d{2}\-\d{2}\s\d{2}\:\d{2}\:\d{2}\.\d{6}\sd
DEST_KEY=queue
FORMAT=nullQueue

EDIT: If Line 1, Line 2 ... Line N are events with Debug line then you can try below transforms.conf

[setnull]
REGEX=(?s)\d{4}\-\d{2}\-\d{2}\s\d{2}\:\d{2}\:\d{2}\.\d{6}\sd.*\n
DEST_KEY=queue
FORMAT=nullQueue

View solution in original post

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

Hi @damucka,

Please try below config on Indexer/Heavy Forwarder whichever comes first from Universal Forwarder.

props.conf

[yoursourcetype]
TRANSFORMS-eliminatedebug = setnull

transforms.conf

[setnull]
REGEX=(?m)\d{4}\-\d{2}\-\d{2}\s\d{2}\:\d{2}\:\d{2}\.\d{6}\sd
DEST_KEY=queue
FORMAT=nullQueue

EDIT: If Line 1, Line 2 ... Line N are events with Debug line then you can try below transforms.conf

[setnull]
REGEX=(?s)\d{4}\-\d{2}\-\d{2}\s\d{2}\:\d{2}\:\d{2}\.\d{6}\sd.*\n
DEST_KEY=queue
FORMAT=nullQueue
0 Karma

lakshman239
SplunkTrust
SplunkTrust

You would also need to ensure that the line with debug, "d" and the following line 1, 2 ... should be configured/treated as multi-line, so as to remove all of them together with nullQueue

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

I am guessing that 1,2 ... those are same type of events and not multiline values.

0 Karma

damucka
Builder

At the moment we have 3 events there, each starting with the header line. Not sure about the multiline values ... I did nothing with the configuration, Splunk recognizes it itself. So, to make it clear - I would like to get rid of the second event.

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

I have provided 2 different regex for transforms.conf, you can try that and let us know if it will not work.

0 Karma

damucka
Builder

Thank you. I will test and let you know.
Is it possible to match several sourcetypes in props.conf to the same entry in transforms.conf? Like below:

[yoursourcetype1]
TRANSFORMS-eliminatedebug = setnull

[yoursourcetype2]
TRANSFORMS-eliminatedebug = setnull

[yoursourcetype3]
TRANSFORMS-eliminatedebug = setnull

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

Yes you can

0 Karma

inventsekar
Ultra Champion

can we simply use "210542 d"
REGEX=210542\sd

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

Hi @inventsekar,

I guess datetime value which is provided is for sample only so 210542\sd will not work because it is subsecond and every event will have different values for subsecond.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...