Getting Data In

How to configure nullQueue to filter out repetitive lines from a log file before indexing?

New Member

Splunk 6.1 Linux indexers feeding server with master license.

I am trying to filter out repetitive lines from a log file before they are indexed. Need to configure the 3 conf files: inputs, props and transform.

The server where the log file is located(different from indexer server where conf files are located): mmd5
mmd5 path/log: /var/log/*/CheckPointReconciler.log*
Log line I want to filter out to nullQueue ( filter on 'Reading')

 2015-12-30 2:02:12.736  14181:4 INFO    job_id none     main    Reading checkpoint directory /mm/feeder/chkpt
0 Karma
1 Solution

SplunkTrust
SplunkTrust

Hi rturecek,

This example discards the example line in your events by sending them to nullQueue:

In props.conf, set the TRANSFORMS- attribute:

[source::/var/log/*/CheckPointReconciler.log*]
TRANSFORMS-001_CheckPointReconciler_NullQueue = CheckPointReconciler_NullQueue

Create a corresponding stanza in transforms.conf. Set DEST_KEY to queue and FORMAT to nullQueue:

[CheckPointReconciler_NullQueue]
REGEX = [\d-\s:\.]+INFO\s+[^\s]+\snone\s+main\s+Reading\scheckpoint\sdirectory
DEST_KEY = queue
FORMAT = nullQueue

You can also set the sourcetype instead of the source in props.conf. Do this either on a heavyweight forwarder or the indexer and remember to restart Splunk afterwards.

Hope this helps ...

cheers, MuS

View solution in original post

SplunkTrust
SplunkTrust

Hi rturecek,

This example discards the example line in your events by sending them to nullQueue:

In props.conf, set the TRANSFORMS- attribute:

[source::/var/log/*/CheckPointReconciler.log*]
TRANSFORMS-001_CheckPointReconciler_NullQueue = CheckPointReconciler_NullQueue

Create a corresponding stanza in transforms.conf. Set DEST_KEY to queue and FORMAT to nullQueue:

[CheckPointReconciler_NullQueue]
REGEX = [\d-\s:\.]+INFO\s+[^\s]+\snone\s+main\s+Reading\scheckpoint\sdirectory
DEST_KEY = queue
FORMAT = nullQueue

You can also set the sourcetype instead of the source in props.conf. Do this either on a heavyweight forwarder or the indexer and remember to restart Splunk afterwards.

Hope this helps ...

cheers, MuS

View solution in original post

New Member

Thank you, The * in the path had to defined a little more precisely but now works.

0 Karma