Splunk Search

REGEX format to remove unwanted log data using specific text with quotes and spaces

srunyon
New Member

I have a log that sends ( eventtype=dlp level=notice vd="PERIM" filteridx=0 filtertype=none filtercat=none severity=medium ) without the ( or ) and would like to know the regex format to use in transforms.conf to send it to the nullQueue. Any help would be greatly appreciated.

Tags (2)
0 Karma

DMohn
Motivator

If you really need to capture this exact event, you can use the following config:

props.conf:

 [your_sourcetype_stanza]
 TRANSFORMS-null1 = eliminate_events

transforms.conf

 [eliminate_events]
 REGEX=^\seventtype=dlp\slevel=notice\svd=\"PERIM\"\sfilteridx=0\sfiltertype=none\sfiltercat=none\sseverity=medium\s$
 DEST_KEY=queue
 FORMAT=nullQueue

You could still check if you may shorten the regex somehow, or make it mire flexible, if the events to be sent to nullQueue vary slightly.

0 Karma

srunyon
New Member

I see how the regex should work. I tried using both a [] and [source::] definitions but I am still getting the events. I can only guess that I am either placing the profs.conf and transforms.conf in the wrong location.

0 Karma

somesoni2
Revered Legend

So this is your actual raw data from the log file?/

eventtype=dlp level=notice vd="PERIM" filteridx=0 filtertype=none filtercat=none severity=medium 
0 Karma

srunyon
New Member

Yes, including the beginning and ending spaces.

0 Karma

somesoni2
Revered Legend

This Splunk doc should give you all the configurations that you need to configure for this event filtering.

http://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Routeandfilterdatad#Discard_specific_e...

The regex could be ^\s*eventtype=dlp. Again, we're not sure if this will filter other events as well, so please provide some more samples of both the logs, one you want to keep and one you want to discard.

0 Karma

srunyon
New Member

To answer you point, I included only the part of the message that is consistent within the data that I do not want to index. This should ensure that I do not discard any valid messages.

0 Karma

somesoni2
Revered Legend

Ok.. so based on where they (the pattern) appear in your data, use the appropriate REGEX. If your actual events starts with the pattern you posted, @DMohn's answer should do the trick. If they do not , remove the caret sign "^" from REGEX.

0 Karma
Get Updates on the Splunk Community!

Thanks for the Memories! Splunk University, .conf24, and Community Connections

Thank you to everyone in the Splunk Community who joined us for .conf24 – starting with Splunk University and ...

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...