topic Re: route unwanted logs to a null queue in Getting Data In

route unwanted logs to a null queue

firasarabo — Sat, 12 Feb 2011 07:07:59 GMT

Hi,

I want to prevent DEBUG logging from bieng indexed by the splunk indexers. we use light weight forwarders on both linux and window boxes, hte indexer is in a linux box.

so here is what I tried. the two files below are in the indexers since we use a light weight forwarder

1- create propes.conf in %SPLUNK_HOME%/etc/system/local/props.conf

[source::....log(.\d+)?]
TRANSFORMS-debug_log = debug_log_transform

2- create transforms.conf in %SPLUNK_HOME/etc/system/local/transforms.conf

[debug_log_transform]
REGEX = \d+\.\d+\.\d+\s\d+\.\d+\.\d+\.\d+\sDEBUG(.*)$
DEST_KEY = queue
FORMAT = nullQueue

doing the above in splunk indexer is not working for me, am I doing some thing wrong here?

the sample logs I need to exclude is:

2011-02-11 23:04:05,448 DEBUG [com.nphase.magicbus.autobinding.cxf.transport.incantation.IncantationConduit] - ...done

Thanks, Firas

Re: route unwanted logs to a null queue

Ron_Naken — Sat, 12 Feb 2011 07:13:24 GMT

It's difficult to read with the formatting of your question, but off-hand, it looks like there are a couple issues that might cause your transform to fail:

try [source::....log...] in your props.conf (check your spelling, it shows as propes.conf in the question.
your REGEX looks incorrect, I would try REGEX=DEBUG\s\[

P.S. You might want to edit your question and highlight your "code" sections that aren't formatting properly and click the "101010" button on the editor bar.

Re: route unwanted logs to a null queue

firasarabo — Sat, 12 Feb 2011 07:57:18 GMT

you are right, the formatting was not good, I edited by adding few "new lines" hope that cleared things. i'll also try your suggestions

Re: route unwanted logs to a null queue

firasarabo — Sat, 12 Feb 2011 08:06:18 GMT

I tried your suggestions but still not working for me.

Re: route unwanted logs to a null queue

Ron_Naken — Sat, 12 Feb 2011 08:40:14 GMT

This will work for sure, unless you have a typo or configuration issue. Just copy-and-paste these:

$SPLUNK_HOME/etc/system/local/props.conf:

[source::....log...]
TRANSFORMS-debug_log = debug_log_transform

$SPLUNK_HOME/etc/system/local/transforms.conf:

[debug_log_transform]
REGEX=DEBUG\s\[
DEST_KEY = queue
FORMAT = nullQueue

Re: route unwanted logs to a null queue

Ron_Naken — Sat, 12 Feb 2011 08:41:53 GMT

I had a typo in the source, but you should be able to copy the examples from the samples I pasted.

Re: route unwanted logs to a null queue

firasarabo — Sat, 12 Feb 2011 09:07:21 GMT

Ron, that didn't work either.

Re: route unwanted logs to a null queue

Ron_Naken — Sat, 12 Feb 2011 09:12:41 GMT

I tried this in the lab and it works for me, using the log entry you posted. This will work for any file whose filename contains .log anywhere in the path/filename, unless you have another props.conf/transforms.conf that is overriding these settings.

Re: route unwanted logs to a null queue

Ron_Naken — Sat, 12 Feb 2011 09:13:25 GMT

By the way, check all your apps ($SPLUNK_HOME/etc/apps/) for props.conf/transforms.conf settings that might be overriding these.

Re: route unwanted logs to a null queue

Ron_Naken — Sat, 12 Feb 2011 09:14:16 GMT

Note: You might also have a transform that is applied to the sourcetype or host that is affecting these settings.

Re: route unwanted logs to a null queue

firasarabo — Mon, 14 Feb 2011 07:50:22 GMT

I found many props.conf under $SPLUNK_HOME/etc/apps/ and I am not suer which one is really used by splunk. just to be on the same page, I am looking only on the indexer and not teh forwarders, let me know if you meant to look on the forwarders. teh list of props.conf is:

./apps/learned/local/props.conf
./apps/sample_app/default/props.conf
./apps/unix/default/props.conf
./apps/search/default/props.conf
./apps/SplunkLightForwarder/default/props.conf

list of transorms.conf:

./apps/unix/default/transforms.conf

Re: route unwanted logs to a null queue

Ron_Naken — Mon, 14 Feb 2011 10:30:33 GMT

If there's a props/transforms that is overriding yours, it's likely going to be in one of the /local folder, not the /default folders. I would make this change on the forwarders, so the irrelevant data is never sent to the indexer.

Re: route unwanted logs to a null queue

firasarabo — Tue, 15 Feb 2011 01:39:34 GMT

Ron,

it seems that your solution is working for me on one environment but not the other. on the one that is working I am not seeing DEBUG logs as used to, I'll need to monitor it for a bit and confirm.

I do have a question though. so if I understand it correctly all DEBUG logging is going to a nullQueue and will not be indexed therefore it will not affect our license limit?

Thanks, Firas

Re: route unwanted logs to a null queue

Ron_Naken — Tue, 15 Feb 2011 02:02:50 GMT

Correct. This transform throws the data away before it is indexed, so it won't count towards your license.

Re: route unwanted logs to a null queue

spock_yh — Tue, 11 Oct 2011 13:00:26 GMT

Firas, did you ever manage to solve this? I'm facing a similar situation.