topic Having trouble with log filtering in Getting Data In

Having trouble with log filtering

leune — Thu, 18 Oct 2012 01:07:51 GMT

I am running into trouble with filtering logs before they are handed off to the indexer. Right now, my setup is that a I am co-hosting an rsyslog daemon on the same box that holds Splunk. The reason for that is that I want to retain all original logs coming in via syslog, but only a subset should go into Splunk.

My concept was that I have the syslogs deposited in a directory and have Splunk monitor that. Using these instructions, I set up props.conf and transforms.conf as follows:

# cat props.conf
[source::/var/log/remote/log1.log]
TRANSFORMS-wireless = setnull,setparsing

# cat transforms.conf
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = Assoc success|User Authentication
DEST_KEY = queue
FORMAT = indexQueue

However, when I activate the log file, data starts pouring into Splunk, without filtering, and quickly running through my license. I did run an "|extract reload=true" search to make sure that the new configurations were loaded before I tested.

The regular expression appears to be correct, as confirmed by a quick grep -E on the file.

Re: Having trouble with log filtering

Ayn — Thu, 18 Oct 2012 07:04:27 GMT

| extract reload=true is not valid for reloading index-time settings - and in fact, as far as I know it's not needed for reloading search-time settings nowadays either, so I doubt there is ever a situation now when running | extract reload=true changes anything.

To activate changes that you've made to index-time settings, you need to restart Splunk.

Also you're saying that you want to perform filtering before logs are handed off to the indexer. Does this mean you're attempting to perform this filtering on a forwarder of some kind? Light forwarders (such as the Universal Forwarder, or a Light Forwarder) do not perform this kind of filtering - it needs to go on the indexer.

Re: Having trouble with log filtering

leune — Thu, 18 Oct 2012 20:03:44 GMT

Thank you for the feedback; the online documentation could do with some improvement on this one 🙂 The help pages for props.conf mention in several places that an |extract reload=T should work. Restarting Splunk did indeed do the trick.