Getting Data In

How do I remove the lines with INFO or WARN from logging to Splunk?

nls7010
Path Finder

I want to NOT ingest the events that have INFO or WARN in them. Can I use the following in the Props.conf without anything in the transforms.conf? And, of course will it work?

[type1-logs]

SEDCMD-removeINFO =s/INFO .+//g

[type2-logs]

SEDCMD-removeWARN =s/WARN .+//g
0 Karma

damann
Communicator

You can use the nullqueue for this purpose:

You need a TRANSFORMS-removeINFO and TRANSFORMS-removeWARN in props.conf:

[type1-logs]
TRANSFORMS-removeINFO = setnullinfo

[type2-logs]
TRANSFORMS-removeWARN = setnullwarn

and in your transforms.conf you need to set the DEST_KEY to queue and FORMAT to nullQueue:

[setnullinfo]
REGEX = INFO\w
DEST_KEY = queue
FORMAT = nullQueue

[setnullwarn]
REGEX = WARN\w
DEST_KEY = queue
FORMAT = nullQueue

Look out for more informations about routing and filtering data in the docs:
https://docs.splunk.com/Documentation/Splunk/7.2.4/Forwarding/Routeandfilterdatad#Filter_event_data_...

0 Karma

nickhills
Ultra Champion

in transforms.conf you need something like this:

[drop_info_warn]
 REGEX = (INFO|WARN)
 DEST_KEY = queue
 FORMAT = nullQueue

then in props.conf you need

[type1-logs]
TRANSFORMS-drop = drop_info_warn

[type2-logs]
TRANSFORMS-drop = drop_info_warn

I am making the assumption that type1-logs, and type2-logs are two different sourcetypes you wish to exclude these events from. If its only one type of log, then you only need to include one such stanza.

To add some context to SEDCMD - this allows you to amend/replace text as the data is indexed, it won't allow you to 'drop' it.
This means that you could mark all the INFO/WARN lines as blank, but you're still going to index them and:
a.) This is almost certainly NOT what you want
b.) You will still be paying to index them (licence usage/disk etc)

By using props AND transforms, you prevent Splunk ever processing (indexing) events which contain these log levels, and I suspect this is what you really want.

If my comment helps, please give it a thumbs up!
0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Event Series: Telemetry Pipeline Management

Balancing Scale and Spend: Gaining Control Over High-Volume Metrics in Splunk Observability Cloud As ...

Kick the Tires Before You Commit: A Hands-On Tour of the Splunk Observability Cloud ...

Evaluating an enterprise observability platform usually goes like this: fill out a form, get a free trial with ...

Deep insights, no barriers: Splunk Observability Cloud Free Edition

As software delivery cycles continue to accelerate, observability shouldn’t be a luxury — it should be a ...