How to configure nullQueue to filter out repetitive lines from a log file before indexing?

[rturecek]

Splunk 6.1 Linux indexers feeding server with master license.

I am trying to filter out repetitive lines from a log file before they are indexed. Need to configure the 3 conf files: inputs, props and transform.

The server where the log file is located(different from indexer server where conf files are located): mmd5
mmd5 path/log: /var/log/*/CheckPointReconciler.log*
Log line I want to filter out to nullQueue ( filter on 'Reading')

 2015-12-30 2:02:12.736  14181:4 INFO    job_id none     main    Reading checkpoint directory /mm/feeder/chkpt

[MuS · Accepted solution]

Hi rturecek,

This example discards the example line in your events by sending them to nullQueue:

In props.conf, set the TRANSFORMS- attribute:

[source::/var/log/*/CheckPointReconciler.log*]
TRANSFORMS-001_CheckPointReconciler_NullQueue = CheckPointReconciler_NullQueue

Create a corresponding stanza in transforms.conf. Set DEST_KEY to queue and FORMAT to nullQueue:

[CheckPointReconciler_NullQueue]
REGEX = [\d-\s:\.]+INFO\s+[^\s]+\snone\s+main\s+Reading\scheckpoint\sdirectory
DEST_KEY = queue
FORMAT = nullQueue

You can also set the sourcetype instead of the source in props.conf. Do this either on a heavyweight forwarder or the indexer and remember to restart Splunk afterwards.

Hope this helps ...

cheers, MuS

[rturecek]

Thank you, The * in the path had to defined a little more precisely but now works.