Getting Data In

Props and Transforms - include base folder, but not some sub folders

Contributor

Hi all, Im trying to do file nullQueue filtering on my HWF.
I want to keep the log entries for /sausages but drop the ones for /sausages/data

So far I have this: (test setup on desktop)
PROPS.CONF
[source::/home/splunk/Desktop/xxx/fs-audit.log*]
TRANSFORMS-set= setnull,whitelist,blacklist

TRANSFORMS.CONF
[setnull]
REGEX= .
DEST_KEY = queue
FORMAT = nullQueue

[whitelist]
REGEX = /sausages
DEST_KEY = queue
FORMAT = indexQueue

[blacklist]
REGEX = /sausages/data
DEST_KEY = queue
FORMAT = nullQueue

I did play with escaping the slashes like \/sausages\/ but that didnt work either.

Thanks in advance.

0 Karma

Contributor

Sorted it.
There was a second file added to the inputs.conf and it didn't have the transforms applied to it!
So I was filtering on just one file, everything from the 2nd file was getting through.
Thanks so much for the help!!

0 Karma

Splunk Employee
Splunk Employee

Glad to know you have figured it out. Cheers! 🙂

0 Karma

Splunk Employee
Splunk Employee

Hi mrgibbon,

I wonder if you could just directly define your filter criteria in the regex rather than use whitelist and blacklist:

TRANSFORMS.CONF
[setnull]
REGEX= /sausages/data
DEST_KEY = queue
FORMAT = nullQueue

It's worth a try. Thanks!
Hunter

0 Karma

Contributor

yeah, the problem is, that this is just one example in this file, there are many others to add too.
I just want to start solving a small issue and work on it from there, its driving me nuts.

My original transforms.conf looked like this:

TRANSFORMS.CONF
[setnull]
REGEX= .
DEST_KEY = queue
FORMAT = nullQueue

[whitelist]
REGEX = /etc|/usr|/bin|/sbin|/opt|/uniworks|/u|/lib
DEST_KEY = queue
FORMAT = indexQueue

[blacklist]
REGEX = /var|/tmp|/vol|/system|/rpool|/proc|/net|/mnt|/backup|/archive|/devices|/export|/kernel|/platform|/uniworks/data
DEST_KEY = queue
FORMAT = nullQueue

0 Karma

Splunk Employee
Splunk Employee

Is it possible that you define a sourcetype for all the events you want to exclude from indexing and then you can send data of that sourcetype to nullQueue?
And just like lguinn suggested, it's advisable to use SOURCE=MetaData.Source to just filter the source.
Thanks!

0 Karma

Contributor

These configurations are filtering your data on an event by event basis. Am I right in thinking you are wanting to filter out whole log files depending on their location? If so, you probably want to configure this in inputs.conf rather than props.
Also, for your whitelist and blacklist stanzas, Splunk is looking in _raw for /sausages and /sausages/data. Can these phrases be found in the events that you are filtering out? If so, can you provide an example event?

0 Karma

Contributor

Yes, all the data is in one audit.log file, I want to remove any entries with /sausages/data but keep everything else with /sausages.
I thought the . regex at the start would kill off any other entries in the file too.

0 Karma

Contributor

So keep the first log entry and nullQueue the 2nd one:

2016-09-26T10:17:38+10:00 fort audit: [ID 702911 audit.notice] open(2) - read,write ok session 2315219746 by user as user:user from 23.23.23.23 obj /sausages/KEEPME.DAT

2016-09-26T10:17:38+10:00 fort audit: [ID 702911 audit.notice] open(2) - read,write ok session 2315219746 by user as user:user from 23.23.23.23 obj /sausages/data/somecorp/test.DAT

0 Karma

Contributor

So are you receiving any events at the moment?

0 Karma

Contributor

Yes, everything, it actually looks like the filtering might not be being applied at all.

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!