Getting Data In
Highlighted

route unwanted logs to a null queue

Path Finder

Hi,

I want to prevent DEBUG logging from bieng indexed by the splunk indexers. we use light weight forwarders on both linux and window boxes, hte indexer is in a linux box.

so here is what I tried. the two files below are in the indexers since we use a light weight forwarder

1- create propes.conf in %SPLUNK_HOME%/etc/system/local/props.conf

[source::....log(.\d+)?]
TRANSFORMS-debug_log = debug_log_transform

2- create transforms.conf in %SPLUNK_HOME/etc/system/local/transforms.conf

[debug_log_transform]
REGEX = \d+\.\d+\.\d+\s\d+\.\d+\.\d+\.\d+\sDEBUG(.*)$
DEST_KEY = queue
FORMAT = nullQueue

doing the above in splunk indexer is not working for me, am I doing some thing wrong here?

the sample logs I need to exclude is:

2011-02-11 23:04:05,448 DEBUG [com.nphase.magicbus.autobinding.cxf.transport.incantation.IncantationConduit] - ...done

Thanks, Firas

Tags (1)
Highlighted

Re: route unwanted logs to a null queue

Builder

It's difficult to read with the formatting of your question, but off-hand, it looks like there are a couple issues that might cause your transform to fail:

  1. try [source::....log...] in your props.conf (check your spelling, it shows as propes.conf in the question.
  2. your REGEX looks incorrect, I would try REGEX=DEBUG\s\[

P.S. You might want to edit your question and highlight your "code" sections that aren't formatting properly and click the "101010" button on the editor bar.

Highlighted

Re: route unwanted logs to a null queue

Path Finder

you are right, the formatting was not good, I edited by adding few "new lines" hope that cleared things. i'll also try your suggestions

0 Karma
Highlighted

Re: route unwanted logs to a null queue

Path Finder

I tried your suggestions but still not working for me.

0 Karma
Highlighted

Re: route unwanted logs to a null queue

Builder

I had a typo in the source, but you should be able to copy the examples from the samples I pasted.

0 Karma
Highlighted

Re: route unwanted logs to a null queue

Builder

This will work for sure, unless you have a typo or configuration issue. Just copy-and-paste these:

$SPLUNK_HOME/etc/system/local/props.conf:

[source::....log...]
TRANSFORMS-debug_log = debug_log_transform

$SPLUNK_HOME/etc/system/local/transforms.conf:

[debug_log_transform]
REGEX=DEBUG\s\[
DEST_KEY = queue
FORMAT = nullQueue
Highlighted

Re: route unwanted logs to a null queue

Path Finder

Ron, that didn't work either.

0 Karma
Highlighted

Re: route unwanted logs to a null queue

Builder

I tried this in the lab and it works for me, using the log entry you posted. This will work for any file whose filename contains .log anywhere in the path/filename, unless you have another props.conf/transforms.conf that is overriding these settings.

0 Karma
Highlighted

Re: route unwanted logs to a null queue

Builder

By the way, check all your apps ($SPLUNK_HOME/etc/apps/) for props.conf/transforms.conf settings that might be overriding these.

Highlighted

Re: route unwanted logs to a null queue

Builder

Note: You might also have a transform that is applied to the sourcetype or host that is affecting these settings.

0 Karma
Speak Up for Splunk Careers!

We want to better understand the impact Splunk experience and expertise has has on individuals' careers, and help highlight the growing demand for Splunk skills.