topic Re: REGEX format to remove unwanted log data using specific text with quotes and spaces in Splunk Search

REGEX format to remove unwanted log data using specific text with quotes and spaces

srunyon — Wed, 09 Mar 2016 16:41:55 GMT

I have a log that sends ( eventtype=dlp level=notice vd="PERIM" filteridx=0 filtertype=none filtercat=none severity=medium ) without the ( or ) and would like to know the regex format to use in transforms.conf to send it to the nullQueue. Any help would be greatly appreciated.

Re: REGEX format to remove unwanted log data using specific text with quotes and spaces

somesoni2 — Wed, 09 Mar 2016 16:51:44 GMT

So this is your actual raw data from the log file?/

eventtype=dlp level=notice vd="PERIM" filteridx=0 filtertype=none filtercat=none severity=medium

Re: REGEX format to remove unwanted log data using specific text with quotes and spaces

srunyon — Wed, 09 Mar 2016 17:58:47 GMT

Yes, including the beginning and ending spaces.

Re: REGEX format to remove unwanted log data using specific text with quotes and spaces

somesoni2 — Wed, 09 Mar 2016 18:11:31 GMT

This Splunk doc should give you all the configurations that you need to configure for this event filtering.

http://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Routeandfilterdatad#Discard_specific_events_and_keep_the_rest

The regex could be ^\s*eventtype=dlp. Again, we're not sure if this will filter other events as well, so please provide some more samples of both the logs, one you want to keep and one you want to discard.

Re: REGEX format to remove unwanted log data using specific text with quotes and spaces

DMohn — Wed, 09 Mar 2016 18:35:20 GMT

If you really need to capture this exact event, you can use the following config:

props.conf:

 [your_sourcetype_stanza]
 TRANSFORMS-null1 = eliminate_events

transforms.conf

 [eliminate_events]
 REGEX=^\seventtype=dlp\slevel=notice\svd=\"PERIM\"\sfilteridx=0\sfiltertype=none\sfiltercat=none\sseverity=medium\s$
 DEST_KEY=queue
 FORMAT=nullQueue

You could still check if you may shorten the regex somehow, or make it mire flexible, if the events to be sent to nullQueue vary slightly.

Re: REGEX format to remove unwanted log data using specific text with quotes and spaces

srunyon — Thu, 10 Mar 2016 20:17:59 GMT

To answer you point, I included only the part of the message that is consistent within the data that I do not want to index. This should ensure that I do not discard any valid messages.

Re: REGEX format to remove unwanted log data using specific text with quotes and spaces

somesoni2 — Thu, 10 Mar 2016 20:58:47 GMT

Ok.. so based on where they (the pattern) appear in your data, use the appropriate REGEX. If your actual events starts with the pattern you posted, @DMohn's answer should do the trick. If they do not , remove the caret sign "^" from REGEX.

Re: REGEX format to remove unwanted log data using specific text with quotes and spaces

srunyon — Fri, 11 Mar 2016 14:58:11 GMT

I see how the regex should work. I tried using both a [] and [source::] definitions but I am still getting the events. I can only guess that I am either placing the profs.conf and transforms.conf in the wrong location.