Getting Data In

Manager » Data inputs » Files & directories » /var/log >> Blacklist

olafbecker
New Member

Hi guys,

I try to setup a prevention filter so that Spunk isn't collection data by accident on the Server side. I found there is a blacklist option under :

Manager » Data inputs » Files & directories » /var/log >> Blacklist

(lastlog|munin.|driveclient.|.(gz|zip|htm|html|z|bz2|[0-9])$)

and I saw that lastlog was already blacklist, but I need to blacklist a bunch more because I don't like to collect data twice because of the log rotation from linux but also munin (/var/log/munin/) dumps a lot of data which I try to exclude as well.

So I created this filter but I do have a feeling that I overlook or miss out on something. What do I miss here with this filter :

(lastlog|munin.|driveclient.|.(gz|zip|htm|html|z|bz2|[0-9])$)

if I do an ls in my /var/log folder I do have this :

10.177.127.70.log daemon.log.3.gz lpr.log nginx
10.20.4.1.log daemon.log.4.gz mail.err nova-agent.log
alternatives.log dbconfig-common mail.info ntpstats
alternatives.log.1 debug mail.log oxy-temp-path
apt debug.1 mail.log.1 passenger-analytics
aptitude debug.2.gz mail.log.2.gz php5-fpm.log
auth.log debug.3.gz mail.log.3.gz pycentral.log
auth.log.1 debug.4.gz mail.log.4.gz redis
auth.log.2.gz dist-upgrade mail.warn samba
auth.log.3.gz dmesg mckick syslog
auth.log.4.gz dpkg.log messages syslog.1
boot dpkg.log.1 messages.1 syslog.2.gz
boot.log dpkg.log.2.gz messages.2.gz syslog.3.gz
bootstrap.log driveclient.log messages.3.gz syslog.4.gz
btmp error messages.4.gz syslog.5.gz
btmp.1.gz fail2ban.log munin syslog.6.gz
chkrootkit fail2ban.log.1 mysql syslog.7.gz
clamav fail2ban.log.2.gz mysql.err sysstat
ConsoleKit fail2ban.log.3.gz mysql.log tallylog
cron.log fail2ban.log.4.gz mysql.log.1.gz tiger
cron.log.1 faillog mysql.log.2.gz udev
cron.log.2.gz fontconfig.log mysql.log.3.gz ufw.log
cron.log.3.gz fsck mysql.log.4.gz unattended-upgrades
cron.log.4.gz jcard mysql.log.5.gz upstart
daemon.log kern.log mysql.log.6.gz user.log
daemon.log.1 kern.log.1 mysql.log.7.gz wtmp
daemon.log.2.gz lastlog news wtmp.1.gz

thx

Olaf

Tags (3)
0 Karma

jlanders
Path Finder

I would recommend checking out a REGEX tester like: http://regexpal.com/

On this web site, if we put this in the first box (the regex box):

(lastlog|munin.|driveclient.|.(gz|zip|htm|html|z|bz2|[0-9])$)

And this in the second box (the text to test on):


10.177.127.70.log
daemon.log.3.gz
lpr.log
nginx
10.20.4.1.log
daemon.log.4.gz
mail.err
nova-agent.log
alternatives.log
dbconfig-common
mail.info
ntpstats
alternatives.log.1
debug
mail.log
oxy-temp-path
apt
debug.1
mail.log.1

You will see the following files have highlighting:


daemon.log.3.gz
daemon.log.4.g
alternatives.log.1
debug.1
mail.log.1

You can reasonably assume that these files with highlighting will be caught by your blacklist and ignored by Splunk.

A few notes though:

  1. You will want your file list to be one file per line
  2. The full file name won't be highlighted, so don't worry about that.
0 Karma
Get Updates on the Splunk Community!

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...

Splunkbase | Splunk Dashboard Examples App for SimpleXML End of Life

The Splunk Dashboard Examples App for SimpleXML will reach end of support on Dec 19, 2024, after which no new ...

Understanding Generative AI Techniques and Their Application in Cybersecurity

Watch On-Demand Artificial intelligence is the talk of the town nowadays, with industries of all kinds ...