Will Splunk re-index a log file if I compress it after it's been indexed?

[Branden]

I have an environment set-up such that syslog-ng redirects syslog to my central log host in this format: /server/host/year/month/day/syslog.log

Splunk monitors that directory tree and indexes it. No big deal.

I'd now like to compress older syslog files and leave the compressed file in that directory. So /server/foo/2010/05/04/syslog.log would become /server/foo/2010/05/04/syslog.log.gz.

I know Splunk can index compressed files. If I compress the syslog file, will the monitor detect the 'new' compressed file and attempt to index it? Or do I have to move the compressed logs somewhere out of the way? I don't want Splunk index data it's already indexed.

Thanks!

[David · Accepted solution]

You can add a blacklist, telling splunk to ignore the compressed log files, e.g.:

blacklist = \.(gz|bz2|z|zip)$ 

(From http://www.splunk.com/base/Documentation/4.2/Data/HowLogFileRotationIsHandled )

Edit: Based on Steven Sorkin's reply, this is only true for 4.1.7 and below. 4.2 will not require a blacklist.

[Stephen_Sorkin · Accepted solution]

In Splunk 4.2, we will not re-index a log file it's been compressed as we compare the decompressed payload of the compressed file to our internal file tracking database. In Splunk 4.1 and before, the best practice is to blacklist the compressed files.

[Branden]

Thank you for the info!

[David]

That's good to know. Note that the documentation is out of date for that. ( http://www.splunk.com/base/Documentation/4.2/Data/HowLogFileRotationIsHandled )

[David · Accepted solution]

You can add a blacklist, telling splunk to ignore the compressed log files, e.g.:

blacklist = \.(gz|bz2|z|zip)$ 

(From http://www.splunk.com/base/Documentation/4.2/Data/HowLogFileRotationIsHandled )

Edit: Based on Steven Sorkin's reply, this is only true for 4.1.7 and below. 4.2 will not require a blacklist.

[Branden]

I'm running 4.1.7 (for now). Looks like I'll need a blacklist.

Thank you for your help.