topic Re: Transform to exclude lines in a log and keep the rest in Getting Data In

Transform to exclude lines in a log and keep the rest

dfratson — Wed, 03 Jul 2019 09:58:54 GMT

Hi,
I need to import a log file in to Splunk however I want to exclude a certain type of entry. The entry to be ignored begins with:

2019-07-03 10:46:10.259  WARN 8944 ---

The only change to this is the date / time stamp. The log entries I want to index begin with:

2019-07-03 10:46:10.208  INFO 8944 ---

and

2019-07-02 13:23:03.194 ERROR 8944 ---

I have tried to implement this using props and transform however it continues to index everything. I've tried a few different options for this and nothing is working as required.

Currently I have the following:

props.conf

[my_log]
MAX_EVENTS = 100000
TRANSFORMS-null = setnull,setparsing

transforms.conf

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = (?i)(ERROR|INFO)
DEST_KEY = queue
FORMAT = indexQueue

Can anyone point me in the right direction?

Thanks in advance.

Re: Transform to exclude lines in a log and keep the rest

arekdabrowski — Thu, 09 Apr 2020 13:57:42 GMT

Hi @dfratson.
You've handled this exercise. I have the same problem. It looks trivially simple, but in practice, SplunkForwarder sends me the entries I deleted with

[setnull]
 REGEX = .
 DEST_KEY = queue
 FORMAT = nullQueue

Re: Transform to exclude lines in a log and keep the rest

rmmiller — Fri, 10 Apr 2020 16:03:04 GMT

If you are trying this with just your Universal Forwarder, that's the problem. You can't do this with just the UF.

See these pages:
https://answers.splunk.com/answers/111257/universal-forwarder-nullqueue.html
http://wiki.splunk.com/Where_do_I_configure_my_Splunk_settings

If you are trying this with a heavy forwarder or indexer, try a different approach.
Your configuration is trying to keep only INFO and ERROR events while filtering out the rest. Is WARN the only event type you're trying to filter out? If so, have you tried flipping this around so you're just excluding WARN?

props.conf

[mylog]
TRANSFORMS-null= setnull

transforms.conf

[setnull]
REGEX = (?i)\bWARN\b
DEST_KEY = queue
FORMAT = nullQueue

Reference: https://docs.splunk.com/Documentation/Splunk/8.0.3/Forwarding/Routeandfilterdatad

Hope that helps!
rmmiller

Re: Transform to exclude lines in a log and keep the rest

arekdabrowski — Tue, 14 Apr 2020 10:08:05 GMT

You inspired me, thank you.
I solved the problem.
It turned out that I'm doing the transformation (transforms.conf) to SplunkForwarder, I moved the transformation to Indexer and now it works.

Re: Transform to exclude lines in a log and keep the rest

rmmiller — Tue, 14 Apr 2020 19:29:35 GMT

This is the best answer -- you figured it out yourself! Thanks for the points, and good luck on your Splunk travels! --rmmiller