Getting Data In

Transform to exclude lines in a log and keep the rest

dfratson
New Member

Hi,
I need to import a log file in to Splunk however I want to exclude a certain type of entry. The entry to be ignored begins with:

2019-07-03 10:46:10.259  WARN 8944 ---

The only change to this is the date / time stamp. The log entries I want to index begin with:

2019-07-03 10:46:10.208  INFO 8944 ---

and

2019-07-02 13:23:03.194 ERROR 8944 ---

I have tried to implement this using props and transform however it continues to index everything. I've tried a few different options for this and nothing is working as required.

Currently I have the following:

props.conf

[my_log]
MAX_EVENTS = 100000
TRANSFORMS-null = setnull,setparsing

transforms.conf

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = (?i)(ERROR|INFO)
DEST_KEY = queue
FORMAT = indexQueue

Can anyone point me in the right direction?

Thanks in advance.

0 Karma

arekdabrowski
Explorer

You inspired me, thank you.
I solved the problem.
It turned out that I'm doing the transformation (transforms.conf) to SplunkForwarder, I moved the transformation to Indexer and now it works.

rmmiller
Contributor

This is the best answer -- you figured it out yourself! Thanks for the points, and good luck on your Splunk travels! --rmmiller

0 Karma

rmmiller
Contributor

If you are trying this with just your Universal Forwarder, that's the problem. You can't do this with just the UF.

See these pages:
https://answers.splunk.com/answers/111257/universal-forwarder-nullqueue.html
http://wiki.splunk.com/Where_do_I_configure_my_Splunk_settings


If you are trying this with a heavy forwarder or indexer, try a different approach.
Your configuration is trying to keep only INFO and ERROR events while filtering out the rest. Is WARN the only event type you're trying to filter out? If so, have you tried flipping this around so you're just excluding WARN?

props.conf

[mylog]
TRANSFORMS-null= setnull

transforms.conf

[setnull]
REGEX = (?i)\bWARN\b
DEST_KEY = queue
FORMAT = nullQueue

Reference: https://docs.splunk.com/Documentation/Splunk/8.0.3/Forwarding/Routeandfilterdatad

Hope that helps!
rmmiller

0 Karma

arekdabrowski
Explorer

Hi @dfratson.
You've handled this exercise. I have the same problem. It looks trivially simple, but in practice, SplunkForwarder sends me the entries I deleted with

[setnull]
 REGEX = .
 DEST_KEY = queue
 FORMAT = nullQueue
0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!