I want to prevent DEBUG logging from bieng indexed by the splunk indexers. we use light weight forwarders on both linux and window boxes, hte indexer is in a linux box.
so here is what I tried. the two files below are in the indexers since we use a light weight forwarder
1- create propes.conf in %SPLUNK_HOME%/etc/system/local/props.conf
[source::....log(.\d+)?] TRANSFORMS-debug_log = debug_log_transform
2- create transforms.conf in %SPLUNK_HOME/etc/system/local/transforms.conf
[debug_log_transform] REGEX = \d+\.\d+\.\d+\s\d+\.\d+\.\d+\.\d+\sDEBUG(.*)$ DEST_KEY = queue FORMAT = nullQueue
doing the above in splunk indexer is not working for me, am I doing some thing wrong here?
the sample logs I need to exclude is:
2011-02-11 23:04:05,448 DEBUG [com.nphase.magicbus.autobinding.cxf.transport.incantation.IncantationConduit] - ...done
it seems that your solution is working for me on one environment but not the other. on the one that is working I am not seeing DEBUG logs as used to, I'll need to monitor it for a bit and confirm.
I do have a question though. so if I understand it correctly all DEBUG logging is going to a nullQueue and will not be indexed therefore it will not affect our license limit?
This will work for sure, unless you have a typo or configuration issue. Just copy-and-paste these:
[source::....log...] TRANSFORMS-debug_log = debug_log_transform
[debug_log_transform] REGEX=DEBUG\s\[ DEST_KEY = queue FORMAT = nullQueue
If there's a props/transforms that is overriding yours, it's likely going to be in one of the /local folder, not the /default folders. I would make this change on the forwarders, so the irrelevant data is never sent to the indexer.
I found many props.conf under $SPLUNK_HOME/etc/apps/ and I am not suer which one is really used by splunk. just to be on the same page, I am looking only on the indexer and not teh forwarders, let me know if you meant to look on the forwarders. teh list of props.conf is:
list of transorms.conf:
I tried this in the lab and it works for me, using the log entry you posted. This will work for any file whose filename contains .log anywhere in the path/filename, unless you have another props.conf/transforms.conf that is overriding these settings.
It's difficult to read with the formatting of your question, but off-hand, it looks like there are a couple issues that might cause your transform to fail:
P.S. You might want to edit your question and highlight your "code" sections that aren't formatting properly and click the "101010" button on the editor bar.