Manager » Data inputs » Files & directories » /var/log >> Blacklist

olafbecker — Thu, 17 Jan 2013 21:13:57 GMT

Hi guys,

I try to setup a prevention filter so that Spunk isn't collection data by accident on the Server side. I found there is a blacklist option under :

(lastlog|munin.|driveclient.|.(gz|zip|htm|html|z|bz2|[0-9])$)

and I saw that lastlog was already blacklist, but I need to blacklist a bunch more because I don't like to collect data twice because of the log rotation from linux but also munin (/var/log/munin/) dumps a lot of data which I try to exclude as well.

So I created this filter but I do have a feeling that I overlook or miss out on something. What do I miss here with this filter :

(lastlog|munin.|driveclient.|.(gz|zip|htm|html|z|bz2|[0-9])$)

if I do an ls in my /var/log folder I do have this :

10.177.127.70.log daemon.log.3.gz lpr.log nginx
10.20.4.1.log daemon.log.4.gz mail.err nova-agent.log
alternatives.log dbconfig-common mail.info ntpstats
alternatives.log.1 debug mail.log oxy-temp-path
apt debug.1 mail.log.1 passenger-analytics
aptitude debug.2.gz mail.log.2.gz php5-fpm.log
auth.log debug.3.gz mail.log.3.gz pycentral.log
auth.log.1 debug.4.gz mail.log.4.gz redis
auth.log.2.gz dist-upgrade mail.warn samba
auth.log.3.gz dmesg mckick syslog
auth.log.4.gz dpkg.log messages syslog.1
boot dpkg.log.1 messages.1 syslog.2.gz
boot.log dpkg.log.2.gz messages.2.gz syslog.3.gz
bootstrap.log driveclient.log messages.3.gz syslog.4.gz
btmp error messages.4.gz syslog.5.gz
btmp.1.gz fail2ban.log munin syslog.6.gz
chkrootkit fail2ban.log.1 mysql syslog.7.gz
clamav fail2ban.log.2.gz mysql.err sysstat
ConsoleKit fail2ban.log.3.gz mysql.log tallylog
cron.log fail2ban.log.4.gz mysql.log.1.gz tiger
cron.log.1 faillog mysql.log.2.gz udev
cron.log.2.gz fontconfig.log mysql.log.3.gz ufw.log
cron.log.3.gz fsck mysql.log.4.gz unattended-upgrades
cron.log.4.gz jcard mysql.log.5.gz upstart
daemon.log kern.log mysql.log.6.gz user.log
daemon.log.1 kern.log.1 mysql.log.7.gz wtmp
daemon.log.2.gz lastlog news wtmp.1.gz

thx

Olaf

Re: Manager » Data inputs » Files & directories » /var/log >> Blacklist

jlanders — Thu, 17 Jan 2013 22:09:52 GMT

I would recommend checking out a REGEX tester like: http://regexpal.com/

On this web site, if we put this in the first box (the regex box):

(lastlog|munin.|driveclient.|.(gz|zip|htm|html|z|bz2|[0-9])$)

And this in the second box (the text to test on):

10.177.127.70.log daemon.log.3.gz lpr.log nginx 10.20.4.1.log daemon.log.4.gz mail.err nova-agent.log alternatives.log dbconfig-common mail.info ntpstats alternatives.log.1 debug mail.log oxy-temp-path apt debug.1 mail.log.1

You will see the following files have highlighting:

daemon.log.3.gz daemon.log.4.g alternatives.log.1 debug.1 mail.log.1

You can reasonably assume that these files with highlighting will be caught by your blacklist and ignored by Splunk.

A few notes though:

You will want your file list to be one file per line
The full file name won't be highlighted, so don't worry about that.

topic Manager » Data inputs » Files & directories » /var/log >> Blacklist in Getting Data In

Manager » Data inputs » Files & directories » /var/log >> Blacklist

Re: Manager » Data inputs » Files & directories » /var/log >> Blacklist