topic Re: How to uncompress and index the compressed log data coming from the forwarder in Getting Data In

How to uncompress and index the compressed log data coming from the forwarder

Sriram — Tue, 18 Mar 2014 21:32:31 GMT

I have few application which sends application logs in both compressed and uncompressed format. There is a log attribute which identifies what the format of the log data is, so I can create 2 different source types. At the indexing time, i need to uncompress the data(compress) and feed to splunk for further analysis.

Is it possible to uncompress the data at index time ? What is the recommended approach to do this ?

Re: How to uncompress and index the compressed log data coming from the forwarder

lguinn2 — Wed, 19 Mar 2014 00:12:11 GMT

If you set up an input on the forwarder like this:

[monitor:///path/to/files]
etc.

It will just work for the compressed files that it finds.

Splunk decompresses archive files before it indexes them. It can handle these common archive file types: tar, gz, bz2, tar.gz, tgz, tbz, tbz2, zip, and z. You do not need to do anything to these files except pass them to Splunk.

Re: How to uncompress and index the compressed log data coming from the forwarder

Sriram — Wed, 19 Mar 2014 17:53:14 GMT

Thank You. The problem is the logs are not coming as zip file type. It will be in application.log which has both compressed and uncompressed data mixed together. Here is an example from the application.log. Based on the LOGTYPE=COMPRESSED, I can identify what type of data this event is. At the Index time, I want to be able to uncompress the data and index it. (I have added the rest of the details below.

Re: How to uncompress and index the compressed log data coming from the forwarder

Sriram — Wed, 19 Mar 2014 18:17:44 GMT

main INFO TxnSummaryLogWriter - LOGTYPE=COMPRESSED, JSON_TXN_SUMMARIES H4sIAAAAAAAAANWbW4vaQBiG/8tcZyEzE 3O63Cil..../veCqx1ATmg4KJlgB5AT3rsDT2P+LFI2qwuOHP4C9fIIkZIyAAA= main INFO TxnSummaryLogWriter - LOGTYPE=UNCOMPRESSED, JSON_TXN_SUMMARIES {"txnsummaries":[{"el":82,"cpu":15, "startTime":"2014-03-12T16:31:02","metadata":{"dummyParam":"foo0","DCN":"143"},"integrations":[{"cat":"JDBC","id":"fooDataSource","cnt":2,"el":17}

Re: How to uncompress and index the compressed log data coming from the forwarder

abhisheksplunka — Thu, 29 Jun 2017 16:04:03 GMT

Does the Splunk also index zlib compressed files or need script to uncompress and then send to Indexer ?

Re: How to uncompress and index the compressed log data coming from the forwarder

abhisheksplunka — Thu, 29 Jun 2017 16:06:33 GMT

Followup Question

One of a saas provider provides access log files in zlib compressed format.
how is the zlib compressed logs files handled by spunk system ?
Or the files needs to uncompressed first and then send to the Splunk for indexing