topic Re: Duplicate files being indexed in Getting Data In

Duplicate files being indexed

eelisio2 — Tue, 09 Nov 2010 01:29:14 GMT

Using the Unix App, monitoring Radius log files. /var/log/radius/radius.log Current log file gets renamed and gzipped. Splunk is indexing radius.log but also indexing radius.log-20101105 and radius.log-20101105.gz.

Suggestions? Thanks.

Re: Duplicate files being indexed

ziegfried — Tue, 09 Nov 2010 01:49:32 GMT

Try blacklisting the rolled/gzipped logs:

$SPLUNK_HOME/etc/apps/unix/local/inputs.conf:

[monitor:///var/log]
disabled = 0
_blacklist = (lastlog|radius\.log.+)

Re: Duplicate files being indexed

Jason_1 — Wed, 10 Nov 2010 02:32:46 GMT

That worked but only for the radius.log file. Modified the blacklist value to include a more generic form that also covered other log files in /var/log/ with the same naming convention (filename.log-<somedate>[.gz|.bz2])

_blacklist=(lastlog|(.log-\d{8}.*)$)

Used the command 'splunk list monitor | grep filename' to confirm the dated files were no longer being monitored, which seems to be the best way to test that the syntax is correct. Thanks for the help!