topic Why is my blacklist being so greedy when going through a Universal Forwarder? in Getting Data In

Why is my blacklist being so greedy when going through a Universal Forwarder?

bimord — Wed, 30 Sep 2020 00:19:37 GMT

I have an inputs.conf file that has multiple monitor stanzas and it appears that the blacklist used on one of the stanzas is being applied to all ...

My aim is to have 4 sourcetypes for the same index with the last sourcetype (search) not showing the logs from the first 3 logs.

Testing locally worked wonders -- grabbed all logs and put them all in their respective sourcetypes and filtered the blacklisted elements from the 4th
BUT
when being processed through to the Universal Forwarder, the blacklist seems to be overriding the entire file thus never getting the applications, server, or audit logs

inputs.conf
`[monitor:///data/web/defaultroot/newlogs/test/applications.log]
index=test
sourcetype=applications
disabled=0

[monitor:///data/web/defaultroot/newlogs/test/server.log]
index=test
sourcetype=server
disabled=0

[monitor:///data/web/defaultroot/newlogs/test/audit.log]
index=test
sourcetype=audit
disabled=0

[monitor:///data/web/defaultroot/newlogs/test/.log]
index=test
sourcetype=search
disabled=0
blacklist1=*gz
blacklist2=applications
blacklist3=server*
blacklist4=audit*`

Re: Why is my blacklist being so greedy when going through a Universal Forwarder?

DEAD_BEEF — Sun, 05 May 2019 14:22:27 GMT

From the docs, the blacklist option is a regex pattern. Your regex of *gz doesn't make sense because * means zero or more of preceding character, which there is no preceding character (I presume you are trying to match on .gz files). Anyway, I believe you can simply combine them into one blacklist and if you know the file names (server.log), I'm not sure why you simply don't specify that in your blacklist rather than server*. Try the below and let us know if it works. If it does, please accept as answer.

[monitor:///data/web/defaultroot/newlogs/test/applications.log]
index=test
sourcetype=applications
disabled=0 

[monitor:///data/web/defaultroot/newlogs/test/server.log]
index=test
sourcetype=server
disabled=0

[monitor:///data/web/defaultroot/newlogs/test/audit.log]
index=test
sourcetype=audit
disabled=0

[monitor:///data/web/defaultroot/newlogs/test/*.log]
index=test
sourcetype=search
disabled=0
blacklist = (\.gz$|applications\.log|server\.log|audit\.log)

Re: Why is my blacklist being so greedy when going through a Universal Forwarder?

DEAD_BEEF — Wed, 08 May 2019 12:23:19 GMT

@bimord did you find a solution?

Re: Why is my blacklist being so greedy when going through a Universal Forwarder?

bimord — Wed, 08 May 2019 22:26:21 GMT

Hi dead_beef
Sorry for the delay -- was trying to get the permissions to access btool.log to futher debug
Good pickup on the regex mistake on *gz - I've updated that everywhere now haha (I inherited the scripting from my predecessor and am trying to work out the kinks)
Unfortunately, even with the updated blacklist you provided, it appears that the the universal forwarder is continuing to apply it to the whole script rather than just that stanza 😞

My solution was to have a separate inputs.conf app to alleviate this with success 🙂

Re: Why is my blacklist being so greedy when going through a Universal Forwarder?

DEAD_BEEF — Wed, 08 May 2019 23:41:59 GMT

Interesting solution, glad you figured it out!