topic Re: Configuring nullQueue With Monitor and Sourcetype in Getting Data In

Configuring nullQueue With Monitor and Sourcetype

confma812 — Fri, 25 Apr 2014 20:50:41 GMT

I've seen many questions regarding dropping information from logs by sending them to a nullQueue, however this is always with a source:: and where I differ is that I've applied a sourcetype to different folders, based on log organization.

My goal is to nullify any line with INFO in it, in various formats, using the regex .*(INFO|\(INFO\)|INFO\:).*

etc/apps/Splunk_TA_windows/local/inputs.conf

[monitor://C:\Program\Prog\logs\debug\*.log]

disabled = false

sourcetype = data_logs_debug


[monitor://C:\Program\Prog\logs\tomcat\*.log]

disabled = false

sourcetype = data_logs_tomcat


[monitor://C:\Program\Prog\logs\.*log]

disabled = false

sourcetype = data_logs_operational

etc/apps/Splunk_TA_windows/local/props.conf

Note: I have also done source:: instead of sourcetype with no luck.

[data_logs_debug]

TRANSFORMS-null=eliminate_info


[data_logs_tomcat]

TRANSFORMS-null=eliminate_info


[data_logs_operational]

TRANSFORMS-null=eliminate_info

etc/apps/Splunk_TA_windows/local/transforms.conf

[eliminate_info]

REGEX = .*(INFO|\(INFO\)|INFO\:).*

DEST_KEY = queue

FORMAT = nullQueue

Re: Configuring nullQueue With Monitor and Sourcetype

martin_mueller — Fri, 25 Apr 2014 23:23:36 GMT

Am I right in guessing that all three of those .conf files reside on a Universal Forwarder?

If so, move or copy props.conf and transforms.conf to the indexer(s).

A note about your REGEX, that will be horribly inefficient. Replace .*(INFO|\(INFO\)|INFO\:).* with INFO - it'll match exactly the same events, but avoid insane amounts of backtracking caused by the .* at the front. That matches everything anyway and its matches aren't captured, so leaving that off doesn't change anything. Leaving off your other two alternatives \(INFO\) and INFO\: doesn't change anything either, because INFO already matches them.

Re: Configuring nullQueue With Monitor and Sourcetype

confma812 — Mon, 28 Apr 2014 14:34:17 GMT

Yes, it is using the Universal Forwarder. I assume since it's UF and not a Heavy Forwarder it can't drop at the forwarder level, but has to at the indexer level?

A true point the regex, as I may just filter it down to (INFO) and INFO: since I'm looking for a particular format in my logs.

And say once I configure that transform on the indexer, will the indexer start removing that indexed data, or will it only apply for newly indexed data?

Re: Configuring nullQueue With Monitor and Sourcetype

martin_mueller — Mon, 28 Apr 2014 14:53:12 GMT

Yup, a UF does not parse/inspect its data. As a result, it cannot route or filter based on event content - usually that's done on the indexers.

Re: Configuring nullQueue With Monitor and Sourcetype

confma812 — Mon, 28 Apr 2014 16:44:02 GMT

Thank you. Good point on the regex - as I tested the efficiency of Splunk with and without the un-greedy approach (.*) and without it improved indexing speed tenfold.

Re: Configuring nullQueue With Monitor and Sourcetype

martin_mueller — Mon, 28 Apr 2014 16:54:07 GMT

.* is greedy, .*? is non-greedy. Both are useless here though because they match everything and their matched string isn't used in the FORMAT.