- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
How to uncompress and index the compressed log data coming from the forwarder
I have few application which sends application logs in both compressed and uncompressed format. There is a log attribute which identifies what the format of the log data is, so I can create 2 different source types. At the indexing time, i need to uncompress the data(compress) and feed to splunk for further analysis.
Is it possible to uncompress the data at index time ? What is the recommended approach to do this ?
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

If you set up an input on the forwarder like this:
[monitor:///path/to/files]
etc.
It will just work for the compressed files that it finds.
Splunk decompresses archive files before it indexes them. It can handle these common archive file types: tar, gz, bz2, tar.gz, tgz, tbz, tbz2, zip, and z. You do not need to do anything to these files except pass them to Splunk.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Followup Question
One of a saas provider provides access log files in zlib compressed format.
how is the zlib compressed logs files handled by spunk system ?
Or the files needs to uncompressed first and then send to the Splunk for indexing
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Does the Splunk also index zlib compressed files or need script to uncompress and then send to Indexer ?
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
main INFO TxnSummaryLogWriter - LOGTYPE=COMPRESSED, JSON_TXN_SUMMARIES H4sIAAAAAAAAANWbW4vaQBiG/8tcZyEzE 3O63Cil..../veCqx1ATmg4KJlgB5AT3rsDT2P+LFI2qwuOHP4C9fIIkZIyAAA=
main INFO TxnSummaryLogWriter - LOGTYPE=UNCOMPRESSED, JSON_TXN_SUMMARIES {"txnsummaries":[{"el":82,"cpu":15, "startTime":"2014-03-12T16:31:02","metadata":{"dummyParam":"foo0","DCN":"143"},"integrations":[{"cat":"JDBC","id":"fooDataSource","cnt":2,"el":17}
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Thank You. The problem is the logs are not coming as zip file type. It will be in application.log which has both compressed and uncompressed data mixed together. Here is an example from the application.log. Based on the LOGTYPE=COMPRESSED, I can identify what type of data this event is. At the Index time, I want to be able to uncompress the data and index it. (I have added the rest of the details below.
