topic Re: Props and Transforms - include base folder, but not some sub folders in Getting Data In

Props and Transforms - include base folder, but not some sub folders

mrgibbon — Thu, 13 Oct 2016 01:04:16 GMT

Hi all, Im trying to do file nullQueue filtering on my HWF.
I want to keep the log entries for /sausages but drop the ones for /sausages/data

So far I have this: (test setup on desktop)
PROPS.CONF
[source::/home/splunk/Desktop/xxx/fs-audit.log*]
TRANSFORMS-set= setnull,whitelist,blacklist

TRANSFORMS.CONF
[setnull]
REGEX= .
DEST_KEY = queue
FORMAT = nullQueue

[whitelist]
REGEX = /sausages
DEST_KEY = queue
FORMAT = indexQueue

[blacklist]
REGEX = /sausages/data
DEST_KEY = queue
FORMAT = nullQueue

I did play with escaping the slashes like \/sausages\/ but that didnt work either.

Thanks in advance.

Re: Props and Transforms - include base folder, but not some sub folders

lquinn — Thu, 13 Oct 2016 02:10:24 GMT

These configurations are filtering your data on an event by event basis. Am I right in thinking you are wanting to filter out whole log files depending on their location? If so, you probably want to configure this in inputs.conf rather than props.
Also, for your whitelist and blacklist stanzas, Splunk is looking in _raw for /sausages and /sausages/data. Can these phrases be found in the events that you are filtering out? If so, can you provide an example event?

Re: Props and Transforms - include base folder, but not some sub folders

mrgibbon — Thu, 13 Oct 2016 02:19:26 GMT

Yes, all the data is in one audit.log file, I want to remove any entries with /sausages/data but keep everything else with /sausages.
I thought the . regex at the start would kill off any other entries in the file too.

Re: Props and Transforms - include base folder, but not some sub folders

lquinn — Thu, 13 Oct 2016 02:22:39 GMT

So are you receiving any events at the moment?

Re: Props and Transforms - include base folder, but not some sub folders

hunters_splunk — Thu, 13 Oct 2016 02:22:53 GMT

Hi mrgibbon,

I wonder if you could just directly define your filter criteria in the regex rather than use whitelist and blacklist:

TRANSFORMS.CONF
[setnull]
REGEX= /sausages/data
DEST_KEY = queue
FORMAT = nullQueue

It's worth a try. Thanks!
Hunter

Re: Props and Transforms - include base folder, but not some sub folders

mrgibbon — Thu, 13 Oct 2016 02:23:04 GMT

So keep the first log entry and nullQueue the 2nd one:

2016-09-26T10:17:38+10:00 fort audit: [ID 702911 audit.notice] open(2) - read,write ok session 2315219746 by user as user:user from 23.23.23.23 obj /sausages/KEEPME.DAT

2016-09-26T10:17:38+10:00 fort audit: [ID 702911 audit.notice] open(2) - read,write ok session 2315219746 by user as user:user from 23.23.23.23 obj /sausages/data/somecorp/test.DAT

Re: Props and Transforms - include base folder, but not some sub folders

mrgibbon — Thu, 13 Oct 2016 02:24:40 GMT

Yes, everything, it actually looks like the filtering might not be being applied at all.

Re: Props and Transforms - include base folder, but not some sub folders

mrgibbon — Thu, 13 Oct 2016 02:26:23 GMT

yeah, the problem is, that this is just one example in this file, there are many others to add too.
I just want to start solving a small issue and work on it from there, its driving me nuts.

My original transforms.conf looked like this:

TRANSFORMS.CONF
[setnull]
REGEX= .
DEST_KEY = queue
FORMAT = nullQueue

[whitelist]
REGEX = /etc|/usr|/bin|/sbin|/opt|/uniworks|/u|/lib
DEST_KEY = queue
FORMAT = indexQueue

Re: Props and Transforms - include base folder, but not some sub folders

hunters_splunk — Thu, 13 Oct 2016 02:42:32 GMT

Is it possible that you define a sourcetype for all the events you want to exclude from indexing and then you can send data of that sourcetype to nullQueue?
And just like lguinn suggested, it's advisable to use SOURCE=MetaData.Source to just filter the source.
Thanks!

Re: Props and Transforms - include base folder, but not some sub folders

mrgibbon — Thu, 13 Oct 2016 02:46:20 GMT

Sorted it.
There was a second file added to the inputs.conf and it didn't have the transforms applied to it!
So I was filtering on just one file, everything from the 2nd file was getting through.
Thanks so much for the help!!

Re: Props and Transforms - include base folder, but not some sub folders

hunters_splunk — Thu, 13 Oct 2016 02:50:24 GMT

Glad to know you have figured it out. Cheers! 🙂