Getting Data In

Will Splunk re-index a log file if I compress it after it's been indexed?

Builder

I have an environment set-up such that syslog-ng redirects syslog to my central log host in this format: /server/host/year/month/day/syslog.log

Splunk monitors that directory tree and indexes it. No big deal.

I'd now like to compress older syslog files and leave the compressed file in that directory. So /server/foo/2010/05/04/syslog.log would become /server/foo/2010/05/04/syslog.log.gz.

I know Splunk can index compressed files. If I compress the syslog file, will the monitor detect the 'new' compressed file and attempt to index it? Or do I have to move the compressed logs somewhere out of the way? I don't want Splunk index data it's already indexed.

Thanks!

Tags (1)
2 Solutions

Splunk Employee
Splunk Employee

You can add a blacklist, telling splunk to ignore the compressed log files, e.g.:

blacklist = \.(gz|bz2|z|zip)$ 

(From http://www.splunk.com/base/Documentation/4.2/Data/HowLogFileRotationIsHandled )

Edit: Based on Steven Sorkin's reply, this is only true for 4.1.7 and below. 4.2 will not require a blacklist.

View solution in original post

Splunk Employee
Splunk Employee

In Splunk 4.2, we will not re-index a log file it's been compressed as we compare the decompressed payload of the compressed file to our internal file tracking database. In Splunk 4.1 and before, the best practice is to blacklist the compressed files.

View solution in original post

Splunk Employee
Splunk Employee

In Splunk 4.2, we will not re-index a log file it's been compressed as we compare the decompressed payload of the compressed file to our internal file tracking database. In Splunk 4.1 and before, the best practice is to blacklist the compressed files.

View solution in original post

Builder

Thank you for the info!

0 Karma

Splunk Employee
Splunk Employee

That's good to know. Note that the documentation is out of date for that. ( http://www.splunk.com/base/Documentation/4.2/Data/HowLogFileRotationIsHandled )

0 Karma

Splunk Employee
Splunk Employee

You can add a blacklist, telling splunk to ignore the compressed log files, e.g.:

blacklist = \.(gz|bz2|z|zip)$ 

(From http://www.splunk.com/base/Documentation/4.2/Data/HowLogFileRotationIsHandled )

Edit: Based on Steven Sorkin's reply, this is only true for 4.1.7 and below. 4.2 will not require a blacklist.

View solution in original post

Builder

I'm running 4.1.7 (for now). Looks like I'll need a blacklist.

Thank you for your help.

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!