Getting Data In

Why is my blacklist being so greedy when going through a Universal Forwarder?

bimord
Path Finder

I have an inputs.conf file that has multiple monitor stanzas and it appears that the blacklist used on one of the stanzas is being applied to all ...

My aim is to have 4 sourcetypes for the same index with the last sourcetype (search) not showing the logs from the first 3 logs.

Testing locally worked wonders -- grabbed all logs and put them all in their respective sourcetypes and filtered the blacklisted elements from the 4th
BUT
when being processed through to the Universal Forwarder, the blacklist seems to be overriding the entire file thus never getting the applications, server, or audit logs

inputs.conf
`[monitor:///data/web/defaultroot/newlogs/test/applications.log]
index=test
sourcetype=applications
disabled=0

[monitor:///data/web/defaultroot/newlogs/test/server.log]
index=test
sourcetype=server
disabled=0

[monitor:///data/web/defaultroot/newlogs/test/audit.log]
index=test
sourcetype=audit
disabled=0

[monitor:///data/web/defaultroot/newlogs/test/.log]
index=test
sourcetype=search
disabled=0
blacklist1=*gz
blacklist2=applications

blacklist3=server*
blacklist4=audit*`

0 Karma
1 Solution

DEAD_BEEF
Builder

From the docs, the blacklist option is a regex pattern. Your regex of *gz doesn't make sense because * means zero or more of preceding character, which there is no preceding character (I presume you are trying to match on .gz files). Anyway, I believe you can simply combine them into one blacklist and if you know the file names (server.log), I'm not sure why you simply don't specify that in your blacklist rather than server*. Try the below and let us know if it works. If it does, please accept as answer.

[monitor:///data/web/defaultroot/newlogs/test/applications.log]
index=test
sourcetype=applications
disabled=0 

[monitor:///data/web/defaultroot/newlogs/test/server.log]
index=test
sourcetype=server
disabled=0

[monitor:///data/web/defaultroot/newlogs/test/audit.log]
index=test
sourcetype=audit
disabled=0

[monitor:///data/web/defaultroot/newlogs/test/*.log]
index=test
sourcetype=search
disabled=0
blacklist = (\.gz$|applications\.log|server\.log|audit\.log)

View solution in original post

DEAD_BEEF
Builder

@bimord did you find a solution?

0 Karma

DEAD_BEEF
Builder

From the docs, the blacklist option is a regex pattern. Your regex of *gz doesn't make sense because * means zero or more of preceding character, which there is no preceding character (I presume you are trying to match on .gz files). Anyway, I believe you can simply combine them into one blacklist and if you know the file names (server.log), I'm not sure why you simply don't specify that in your blacklist rather than server*. Try the below and let us know if it works. If it does, please accept as answer.

[monitor:///data/web/defaultroot/newlogs/test/applications.log]
index=test
sourcetype=applications
disabled=0 

[monitor:///data/web/defaultroot/newlogs/test/server.log]
index=test
sourcetype=server
disabled=0

[monitor:///data/web/defaultroot/newlogs/test/audit.log]
index=test
sourcetype=audit
disabled=0

[monitor:///data/web/defaultroot/newlogs/test/*.log]
index=test
sourcetype=search
disabled=0
blacklist = (\.gz$|applications\.log|server\.log|audit\.log)

bimord
Path Finder

Hi dead_beef
Sorry for the delay -- was trying to get the permissions to access btool.log to futher debug
Good pickup on the regex mistake on *gz - I've updated that everywhere now haha (I inherited the scripting from my predecessor and am trying to work out the kinks)
Unfortunately, even with the updated blacklist you provided, it appears that the the universal forwarder is continuing to apply it to the whole script rather than just that stanza 😞

My solution was to have a separate inputs.conf app to alleviate this with success 🙂

0 Karma

DEAD_BEEF
Builder

Interesting solution, glad you figured it out!

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...