Transform to exclude lines in a log and keep the rest

[dfratson]

Hi,
I need to import a log file in to Splunk however I want to exclude a certain type of entry. The entry to be ignored begins with:

2019-07-03 10:46:10.259  WARN 8944 ---

The only change to this is the date / time stamp. The log entries I want to index begin with:

2019-07-03 10:46:10.208  INFO 8944 ---

and

2019-07-02 13:23:03.194 ERROR 8944 ---

I have tried to implement this using props and transform however it continues to index everything. I've tried a few different options for this and nothing is working as required.

Currently I have the following:

props.conf

[my_log]
MAX_EVENTS = 100000
TRANSFORMS-null = setnull,setparsing

transforms.conf

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = (?i)(ERROR|INFO)
DEST_KEY = queue
FORMAT = indexQueue

Can anyone point me in the right direction?

Thanks in advance.

[arekdabrowski]

You inspired me, thank you.
I solved the problem.
It turned out that I'm doing the transformation (transforms.conf) to SplunkForwarder, I moved the transformation to Indexer and now it works.

[rmmiller]

This is the best answer -- you figured it out yourself! Thanks for the points, and good luck on your Splunk travels! --rmmiller

[rmmiller]

If you are trying this with just your Universal Forwarder, that's the problem. You can't do this with just the UF.

See these pages:
https://answers.splunk.com/answers/111257/universal-forwarder-nullqueue.html
http://wiki.splunk.com/Where_do_I_configure_my_Splunk_settings

---

If you are trying this with a heavy forwarder or indexer, try a different approach.
Your configuration is trying to keep only INFO and ERROR events while filtering out the rest. Is WARN the only event type you're trying to filter out? If so, have you tried flipping this around so you're just excluding WARN?

props.conf

[mylog]
TRANSFORMS-null= setnull

transforms.conf

[setnull]
REGEX = (?i)\bWARN\b
DEST_KEY = queue
FORMAT = nullQueue

Reference: https://docs.splunk.com/Documentation/Splunk/8.0.3/Forwarding/Routeandfilterdatad

Hope that helps!
rmmiller

[arekdabrowski]

Hi @dfratson.
You've handled this exercise. I have the same problem. It looks trivially simple, but in practice, SplunkForwarder sends me the entries I deleted with

[setnull]
 REGEX = .
 DEST_KEY = queue
 FORMAT = nullQueue