Manager » Data inputs » Files & directories » /var/log >> Blacklist

[olafbecker]

Hi guys,

I try to setup a prevention filter so that Spunk isn't collection data by accident on the Server side. I found there is a blacklist option under :

Manager » Data inputs » Files & directories » /var/log >> Blacklist

(lastlog|munin.|driveclient.|.(gz|zip|htm|html|z|bz2|[0-9])$)

and I saw that lastlog was already blacklist, but I need to blacklist a bunch more because I don't like to collect data twice because of the log rotation from linux but also munin (/var/log/munin/) dumps a lot of data which I try to exclude as well.

So I created this filter but I do have a feeling that I overlook or miss out on something. What do I miss here with this filter :

(lastlog|munin.|driveclient.|.(gz|zip|htm|html|z|bz2|[0-9])$)

if I do an ls in my /var/log folder I do have this :

10.177.127.70.log daemon.log.3.gz lpr.log nginx
10.20.4.1.log daemon.log.4.gz mail.err nova-agent.log
alternatives.log dbconfig-common mail.info ntpstats
alternatives.log.1 debug mail.log oxy-temp-path
apt debug.1 mail.log.1 passenger-analytics
aptitude debug.2.gz mail.log.2.gz php5-fpm.log
auth.log debug.3.gz mail.log.3.gz pycentral.log
auth.log.1 debug.4.gz mail.log.4.gz redis
auth.log.2.gz dist-upgrade mail.warn samba
auth.log.3.gz dmesg mckick syslog
auth.log.4.gz dpkg.log messages syslog.1
boot dpkg.log.1 messages.1 syslog.2.gz
boot.log dpkg.log.2.gz messages.2.gz syslog.3.gz
bootstrap.log driveclient.log messages.3.gz syslog.4.gz
btmp error messages.4.gz syslog.5.gz
btmp.1.gz fail2ban.log munin syslog.6.gz
chkrootkit fail2ban.log.1 mysql syslog.7.gz
clamav fail2ban.log.2.gz mysql.err sysstat
ConsoleKit fail2ban.log.3.gz mysql.log tallylog
cron.log fail2ban.log.4.gz mysql.log.1.gz tiger
cron.log.1 faillog mysql.log.2.gz udev
cron.log.2.gz fontconfig.log mysql.log.3.gz ufw.log
cron.log.3.gz fsck mysql.log.4.gz unattended-upgrades
cron.log.4.gz jcard mysql.log.5.gz upstart
daemon.log kern.log mysql.log.6.gz user.log
daemon.log.1 kern.log.1 mysql.log.7.gz wtmp
daemon.log.2.gz lastlog news wtmp.1.gz

thx

Olaf

[jlanders]

I would recommend checking out a REGEX tester like: http://regexpal.com/

On this web site, if we put this in the first box (the regex box):

(lastlog|munin.|driveclient.|.(gz|zip|htm|html|z|bz2|[0-9])$)

And this in the second box (the text to test on):


10.177.127.70.log
daemon.log.3.gz
lpr.log
nginx
10.20.4.1.log
daemon.log.4.gz
mail.err
nova-agent.log
alternatives.log
dbconfig-common
mail.info
ntpstats
alternatives.log.1
debug
mail.log
oxy-temp-path
apt
debug.1
mail.log.1


You will see the following files have highlighting:


daemon.log.3.gz
daemon.log.4.g
alternatives.log.1
debug.1
mail.log.1


You can reasonably assume that these files with highlighting will be caught by your blacklist and ignored by Splunk.

A few notes though:

1. You will want your file list to be one file per line
2. The full file name won't be highlighted, so don't worry about that.