Getting Data In

route unwanted logs to a null queue

firasarabo
Path Finder

Hi,

I want to prevent DEBUG logging from bieng indexed by the splunk indexers. we use light weight forwarders on both linux and window boxes, hte indexer is in a linux box.

so here is what I tried. the two files below are in the indexers since we use a light weight forwarder

1- create propes.conf in %SPLUNK_HOME%/etc/system/local/props.conf

[source::....log(.\d+)?]
TRANSFORMS-debug_log = debug_log_transform

2- create transforms.conf in %SPLUNK_HOME/etc/system/local/transforms.conf

[debug_log_transform]
REGEX = \d+\.\d+\.\d+\s\d+\.\d+\.\d+\.\d+\sDEBUG(.*)$
DEST_KEY = queue
FORMAT = nullQueue

doing the above in splunk indexer is not working for me, am I doing some thing wrong here?

the sample logs I need to exclude is:

2011-02-11 23:04:05,448 DEBUG [com.nphase.magicbus.autobinding.cxf.transport.incantation.IncantationConduit] - ...done

Thanks, Firas

Tags (1)

firasarabo
Path Finder

Ron,

it seems that your solution is working for me on one environment but not the other. on the one that is working I am not seeing DEBUG logs as used to, I'll need to monitor it for a bit and confirm.

I do have a question though. so if I understand it correctly all DEBUG logging is going to a nullQueue and will not be indexed therefore it will not affect our license limit?

Thanks, Firas

0 Karma

spock_yh
Path Finder

Firas, did you ever manage to solve this? I'm facing a similar situation.

0 Karma

Ron_Naken
Splunk Employee
Splunk Employee

Correct. This transform throws the data away before it is indexed, so it won't count towards your license.

0 Karma

Ron_Naken
Splunk Employee
Splunk Employee

This will work for sure, unless you have a typo or configuration issue. Just copy-and-paste these:

$SPLUNK_HOME/etc/system/local/props.conf:

[source::....log...]
TRANSFORMS-debug_log = debug_log_transform

$SPLUNK_HOME/etc/system/local/transforms.conf:

[debug_log_transform]
REGEX=DEBUG\s\[
DEST_KEY = queue
FORMAT = nullQueue

Ron_Naken
Splunk Employee
Splunk Employee

If there's a props/transforms that is overriding yours, it's likely going to be in one of the /local folder, not the /default folders. I would make this change on the forwarders, so the irrelevant data is never sent to the indexer.

firasarabo
Path Finder

I found many props.conf under $SPLUNK_HOME/etc/apps/ and I am not suer which one is really used by splunk. just to be on the same page, I am looking only on the indexer and not teh forwarders, let me know if you meant to look on the forwarders. teh list of props.conf is:

./apps/learned/local/props.conf
./apps/sample_app/default/props.conf
./apps/unix/default/props.conf
./apps/search/default/props.conf
./apps/SplunkLightForwarder/default/props.conf

list of transorms.conf:

./apps/unix/default/transforms.conf

Ron_Naken
Splunk Employee
Splunk Employee

Note: You might also have a transform that is applied to the sourcetype or host that is affecting these settings.

0 Karma

Ron_Naken
Splunk Employee
Splunk Employee

By the way, check all your apps ($SPLUNK_HOME/etc/apps/) for props.conf/transforms.conf settings that might be overriding these.

Ron_Naken
Splunk Employee
Splunk Employee

I tried this in the lab and it works for me, using the log entry you posted. This will work for any file whose filename contains .log anywhere in the path/filename, unless you have another props.conf/transforms.conf that is overriding these settings.

0 Karma

firasarabo
Path Finder

Ron, that didn't work either.

0 Karma

Ron_Naken
Splunk Employee
Splunk Employee

It's difficult to read with the formatting of your question, but off-hand, it looks like there are a couple issues that might cause your transform to fail:

  1. try [source::....log...] in your props.conf (check your spelling, it shows as propes.conf in the question.
  2. your REGEX looks incorrect, I would try REGEX=DEBUG\s\[

P.S. You might want to edit your question and highlight your "code" sections that aren't formatting properly and click the "101010" button on the editor bar.

Ron_Naken
Splunk Employee
Splunk Employee

I had a typo in the source, but you should be able to copy the examples from the samples I pasted.

0 Karma

firasarabo
Path Finder

I tried your suggestions but still not working for me.

0 Karma

firasarabo
Path Finder

you are right, the formatting was not good, I edited by adding few "new lines" hope that cleared things. i'll also try your suggestions

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!