Getting Data In

Getting Data in from S3 : Blacklist option

saty586
Explorer

Trying to get a part of file in S3 into Splunk.

Can i use blacklist option to filter out part of file and ingest rest of data in Splunk?

Labels (3)
0 Karma
1 Solution

96nick
Communicator

Instead of a blacklist, you could also use the whitelist value:

whitelist = <Override regex for blacklist when using a folder key.>. For example, whitelist = ^.\/cloudtrail2\/.$. Regex should match the full path.

 

Instead of making a complicated regex with a blacklist, I would make a simpler one with that will allow you to pull in anything with the 'Y' value.  I'd use Regex101 to make sure it's correct, and remember to use the full path!

View solution in original post

0 Karma

96nick
Communicator

Yes, you can use the blacklist option in the inputs.conf stanza to ignore section of the directory you're ingesting from S3. You just have to make sure the regex matches the full path which means your regex has to use asterisks. 

More info here:  

https://community.splunk.com/t5/All-Apps-and-Add-ons/Splunk-Add-on-for-Amazon-Web-Services-3-0-Why-d...

saty586
Explorer

Thanks @96nick 

But is it possible to filter out 2 events out of 10 events present in same file in a bucket.

For example -

I have only 1 file in the bucket - "xyz.txt"

Content of xyz.txt

event_id    flag     user_id

U123           Y              123

I123             N              101

U125           Y                234

is it possible to ingest only rows with   flag with value "Y" in splunk . using regex in blacklist option?

0 Karma

96nick
Communicator

Instead of a blacklist, you could also use the whitelist value:

whitelist = <Override regex for blacklist when using a folder key.>. For example, whitelist = ^.\/cloudtrail2\/.$. Regex should match the full path.

 

Instead of making a complicated regex with a blacklist, I would make a simpler one with that will allow you to pull in anything with the 'Y' value.  I'd use Regex101 to make sure it's correct, and remember to use the full path!

0 Karma
Get Updates on the Splunk Community!

Splunk Forwarders and Forced Time Based Load Balancing

Splunk customers use universal forwarders to collect and send data to Splunk. A universal forwarder can send ...

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...