Getting Data In

Props and Transforms - include base folder, but not some sub folders

mrgibbon
Contributor

Hi all, Im trying to do file nullQueue filtering on my HWF.
I want to keep the log entries for /sausages but drop the ones for /sausages/data

So far I have this: (test setup on desktop)
PROPS.CONF
[source::/home/splunk/Desktop/xxx/fs-audit.log*]
TRANSFORMS-set= setnull,whitelist,blacklist

TRANSFORMS.CONF
[setnull]
REGEX= .
DEST_KEY = queue
FORMAT = nullQueue

[whitelist]
REGEX = /sausages
DEST_KEY = queue
FORMAT = indexQueue

[blacklist]
REGEX = /sausages/data
DEST_KEY = queue
FORMAT = nullQueue

I did play with escaping the slashes like \/sausages\/ but that didnt work either.

Thanks in advance.

0 Karma

mrgibbon
Contributor

Sorted it.
There was a second file added to the inputs.conf and it didn't have the transforms applied to it!
So I was filtering on just one file, everything from the 2nd file was getting through.
Thanks so much for the help!!

0 Karma

hunters_splunk
Splunk Employee
Splunk Employee

Glad to know you have figured it out. Cheers! 🙂

0 Karma

hunters_splunk
Splunk Employee
Splunk Employee

Hi mrgibbon,

I wonder if you could just directly define your filter criteria in the regex rather than use whitelist and blacklist:

TRANSFORMS.CONF
[setnull]
REGEX= /sausages/data
DEST_KEY = queue
FORMAT = nullQueue

It's worth a try. Thanks!
Hunter

0 Karma

mrgibbon
Contributor

yeah, the problem is, that this is just one example in this file, there are many others to add too.
I just want to start solving a small issue and work on it from there, its driving me nuts.

My original transforms.conf looked like this:

TRANSFORMS.CONF
[setnull]
REGEX= .
DEST_KEY = queue
FORMAT = nullQueue

[whitelist]
REGEX = /etc|/usr|/bin|/sbin|/opt|/uniworks|/u|/lib
DEST_KEY = queue
FORMAT = indexQueue

[blacklist]
REGEX = /var|/tmp|/vol|/system|/rpool|/proc|/net|/mnt|/backup|/archive|/devices|/export|/kernel|/platform|/uniworks/data
DEST_KEY = queue
FORMAT = nullQueue

0 Karma

hunters_splunk
Splunk Employee
Splunk Employee

Is it possible that you define a sourcetype for all the events you want to exclude from indexing and then you can send data of that sourcetype to nullQueue?
And just like lguinn suggested, it's advisable to use SOURCE=MetaData.Source to just filter the source.
Thanks!

0 Karma

lquinn
Contributor

These configurations are filtering your data on an event by event basis. Am I right in thinking you are wanting to filter out whole log files depending on their location? If so, you probably want to configure this in inputs.conf rather than props.
Also, for your whitelist and blacklist stanzas, Splunk is looking in _raw for /sausages and /sausages/data. Can these phrases be found in the events that you are filtering out? If so, can you provide an example event?

0 Karma

mrgibbon
Contributor

Yes, all the data is in one audit.log file, I want to remove any entries with /sausages/data but keep everything else with /sausages.
I thought the . regex at the start would kill off any other entries in the file too.

0 Karma

mrgibbon
Contributor

So keep the first log entry and nullQueue the 2nd one:

2016-09-26T10:17:38+10:00 fort audit: [ID 702911 audit.notice] open(2) - read,write ok session 2315219746 by user as user:user from 23.23.23.23 obj /sausages/KEEPME.DAT

2016-09-26T10:17:38+10:00 fort audit: [ID 702911 audit.notice] open(2) - read,write ok session 2315219746 by user as user:user from 23.23.23.23 obj /sausages/data/somecorp/test.DAT

0 Karma

lquinn
Contributor

So are you receiving any events at the moment?

0 Karma

mrgibbon
Contributor

Yes, everything, it actually looks like the filtering might not be being applied at all.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...