Getting Data In

Why is my blacklist being so greedy when going through a Universal Forwarder?

bimord
Path Finder

I have an inputs.conf file that has multiple monitor stanzas and it appears that the blacklist used on one of the stanzas is being applied to all ...

My aim is to have 4 sourcetypes for the same index with the last sourcetype (search) not showing the logs from the first 3 logs.

Testing locally worked wonders -- grabbed all logs and put them all in their respective sourcetypes and filtered the blacklisted elements from the 4th
BUT
when being processed through to the Universal Forwarder, the blacklist seems to be overriding the entire file thus never getting the applications, server, or audit logs

inputs.conf
`[monitor:///data/web/defaultroot/newlogs/test/applications.log]
index=test
sourcetype=applications
disabled=0

[monitor:///data/web/defaultroot/newlogs/test/server.log]
index=test
sourcetype=server
disabled=0

[monitor:///data/web/defaultroot/newlogs/test/audit.log]
index=test
sourcetype=audit
disabled=0

[monitor:///data/web/defaultroot/newlogs/test/.log]
index=test
sourcetype=search
disabled=0
blacklist1=*gz
blacklist2=applications

blacklist3=server*
blacklist4=audit*`

0 Karma
1 Solution

DEAD_BEEF
Builder

From the docs, the blacklist option is a regex pattern. Your regex of *gz doesn't make sense because * means zero or more of preceding character, which there is no preceding character (I presume you are trying to match on .gz files). Anyway, I believe you can simply combine them into one blacklist and if you know the file names (server.log), I'm not sure why you simply don't specify that in your blacklist rather than server*. Try the below and let us know if it works. If it does, please accept as answer.

[monitor:///data/web/defaultroot/newlogs/test/applications.log]
index=test
sourcetype=applications
disabled=0 

[monitor:///data/web/defaultroot/newlogs/test/server.log]
index=test
sourcetype=server
disabled=0

[monitor:///data/web/defaultroot/newlogs/test/audit.log]
index=test
sourcetype=audit
disabled=0

[monitor:///data/web/defaultroot/newlogs/test/*.log]
index=test
sourcetype=search
disabled=0
blacklist = (\.gz$|applications\.log|server\.log|audit\.log)

View solution in original post

DEAD_BEEF
Builder

@bimord did you find a solution?

0 Karma

DEAD_BEEF
Builder

From the docs, the blacklist option is a regex pattern. Your regex of *gz doesn't make sense because * means zero or more of preceding character, which there is no preceding character (I presume you are trying to match on .gz files). Anyway, I believe you can simply combine them into one blacklist and if you know the file names (server.log), I'm not sure why you simply don't specify that in your blacklist rather than server*. Try the below and let us know if it works. If it does, please accept as answer.

[monitor:///data/web/defaultroot/newlogs/test/applications.log]
index=test
sourcetype=applications
disabled=0 

[monitor:///data/web/defaultroot/newlogs/test/server.log]
index=test
sourcetype=server
disabled=0

[monitor:///data/web/defaultroot/newlogs/test/audit.log]
index=test
sourcetype=audit
disabled=0

[monitor:///data/web/defaultroot/newlogs/test/*.log]
index=test
sourcetype=search
disabled=0
blacklist = (\.gz$|applications\.log|server\.log|audit\.log)

bimord
Path Finder

Hi dead_beef
Sorry for the delay -- was trying to get the permissions to access btool.log to futher debug
Good pickup on the regex mistake on *gz - I've updated that everywhere now haha (I inherited the scripting from my predecessor and am trying to work out the kinks)
Unfortunately, even with the updated blacklist you provided, it appears that the the universal forwarder is continuing to apply it to the whole script rather than just that stanza 😞

My solution was to have a separate inputs.conf app to alleviate this with success 🙂

0 Karma

DEAD_BEEF
Builder

Interesting solution, glad you figured it out!

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...