Getting Data In

Will Splunk re-index a log file if I compress it after it's been indexed?

Branden
Builder

I have an environment set-up such that syslog-ng redirects syslog to my central log host in this format: /server/host/year/month/day/syslog.log

Splunk monitors that directory tree and indexes it. No big deal.

I'd now like to compress older syslog files and leave the compressed file in that directory. So /server/foo/2010/05/04/syslog.log would become /server/foo/2010/05/04/syslog.log.gz.

I know Splunk can index compressed files. If I compress the syslog file, will the monitor detect the 'new' compressed file and attempt to index it? Or do I have to move the compressed logs somewhere out of the way? I don't want Splunk index data it's already indexed.

Thanks!

Tags (1)
2 Solutions

David
Splunk Employee
Splunk Employee

You can add a blacklist, telling splunk to ignore the compressed log files, e.g.:

blacklist = \.(gz|bz2|z|zip)$ 

(From http://www.splunk.com/base/Documentation/4.2/Data/HowLogFileRotationIsHandled )

Edit: Based on Steven Sorkin's reply, this is only true for 4.1.7 and below. 4.2 will not require a blacklist.

View solution in original post

Stephen_Sorkin
Splunk Employee
Splunk Employee

In Splunk 4.2, we will not re-index a log file it's been compressed as we compare the decompressed payload of the compressed file to our internal file tracking database. In Splunk 4.1 and before, the best practice is to blacklist the compressed files.

View solution in original post

Stephen_Sorkin
Splunk Employee
Splunk Employee

In Splunk 4.2, we will not re-index a log file it's been compressed as we compare the decompressed payload of the compressed file to our internal file tracking database. In Splunk 4.1 and before, the best practice is to blacklist the compressed files.

Branden
Builder

Thank you for the info!

0 Karma

David
Splunk Employee
Splunk Employee

That's good to know. Note that the documentation is out of date for that. ( http://www.splunk.com/base/Documentation/4.2/Data/HowLogFileRotationIsHandled )

0 Karma

David
Splunk Employee
Splunk Employee

You can add a blacklist, telling splunk to ignore the compressed log files, e.g.:

blacklist = \.(gz|bz2|z|zip)$ 

(From http://www.splunk.com/base/Documentation/4.2/Data/HowLogFileRotationIsHandled )

Edit: Based on Steven Sorkin's reply, this is only true for 4.1.7 and below. 4.2 will not require a blacklist.

Branden
Builder

I'm running 4.1.7 (for now). Looks like I'll need a blacklist.

Thank you for your help.

0 Karma
Get Updates on the Splunk Community!

Improve Your Security Posture

Watch NowImprove Your Security PostureCustomers are at the center of everything we do at Splunk and security ...

Maximize the Value from Microsoft Defender with Splunk

 Watch NowJoin Splunk and Sens Consulting for this Security Edition Tech TalkWho should attend:  Security ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...