Getting Data In
Highlighted

How to monitor and index tar.gz files in Splunk?

Communicator

I have a tar.gz file and I wan't to continuously monitor it. I tried to index it to Splunk Enterprise via Settings>Data Inputs>Files&Directories, but when I run a search, Splunk doesn't return a result.
What are the steps to continuously monitor tar.gz files to index in Splunk? Do I need to write a script that automatically decompress tar.gz file so Splunk can index it? Thanks.

0 Karma
Highlighted

Re: How to monitor and index tar.gz files in Splunk?

SplunkTrust
SplunkTrust

Splunk won't index compressed files because they look like binaries. A script is one idea. Or you could have Splunk monitor the files before they are tarred.

---
If this reply helps you, an upvote would be appreciated.

View solution in original post

Highlighted

Re: How to monitor and index tar.gz files in Splunk?

Motivator
0 Karma
Highlighted

Re: How to monitor and index tar.gz files in Splunk?

Contributor

I downvoted this post because this answer is incorrect. Splunk is capable of monitoring compressed files. There must be some other issue here.

http://docs.splunk.com/Documentation/Splunk/latest/Data/Monitorfilesanddirectories

0 Karma
Highlighted

Re: How to monitor and index tar.gz files in Splunk?

Path Finder

I downvoted this post because splunk can index compressed files

0 Karma
Highlighted

Re: How to monitor and index tar.gz files in Splunk?

Communicator

In my case splunk enterprise did not index compressed file so we created a bash script to uncompressed the data and proceed with the indexing.

0 Karma
Highlighted

Re: How to monitor and index tar.gz files in Splunk?

Engager

I downvoted this post because splunk does index the compress files, it just doesn't perform parallel monitoring but sequential one. cpu is the key if you are going to decompress a lot of files like > 50k

0 Karma
Highlighted

Re: How to monitor and index tar.gz files in Splunk?

Engager

I downvoted this post because the answer provided is incorrect.

0 Karma
Highlighted

Re: How to monitor and index tar.gz files in Splunk?

Motivator

According to the most recent docs Splunk does index compressed files

http://docs.splunk.com/Documentation/Splunk/6.5.0/Data/Monitorfilesanddirectories

How Splunk Enterprise monitors archive files
Archive files (such as a .tar or .zip file, are decompressed before being indexed. The following types of archive files are supported:

 .tar
 .gz
 .bz2
 .tar.gz and .tgz
 .tbz and .tbz2
 .zip
 .z
If you add new data to an existing archive file, the entire file is reindexed, not just the new data. This can result in event duplication.
Highlighted

Re: How to monitor and index tar.gz files in Splunk?

Motivator

I use the Universal forwarder to monitor compressed files, haven't tried it with the gui though.....

0 Karma