Getting Data In
Highlighted

How to uncompress and index the compressed log data coming from the forwarder

Communicator

I have few application which sends application logs in both compressed and uncompressed format. There is a log attribute which identifies what the format of the log data is, so I can create 2 different source types. At the indexing time, i need to uncompress the data(compress) and feed to splunk for further analysis.

Is it possible to uncompress the data at index time ? What is the recommended approach to do this ?

0 Karma
Highlighted

Re: How to uncompress and index the compressed log data coming from the forwarder

Legend

If you set up an input on the forwarder like this:

[monitor:///path/to/files]
etc.

It will just work for the compressed files that it finds.

Splunk decompresses archive files before it indexes them. It can handle these common archive file types: tar, gz, bz2, tar.gz, tgz, tbz, tbz2, zip, and z. You do not need to do anything to these files except pass them to Splunk.

Highlighted

Re: How to uncompress and index the compressed log data coming from the forwarder

Communicator

Thank You. The problem is the logs are not coming as zip file type. It will be in application.log which has both compressed and uncompressed data mixed together. Here is an example from the application.log. Based on the LOGTYPE=COMPRESSED, I can identify what type of data this event is. At the Index time, I want to be able to uncompress the data and index it. (I have added the rest of the details below.

0 Karma
Highlighted

Re: How to uncompress and index the compressed log data coming from the forwarder

Communicator

main INFO TxnSummaryLogWriter - LOGTYPE=COMPRESSED, JSON_TXN_SUMMARIES H4sIAAAAAAAAANWbW4vaQBiG/8tcZyEzE 3O63Cil..../veCqx1ATmg4KJlgB5AT3rsDT2P+LFI2qwuOHP4C9fIIkZIyAAA=
main INFO TxnSummaryLogWriter - LOGTYPE=UNCOMPRESSED, JSON_TXN_SUMMARIES {"txnsummaries":[{"el":82,"cpu":15, "startTime":"2014-03-12T16:31:02","metadata":{"dummyParam":"foo0","DCN":"143"},"integrations":[{"cat":"JDBC","id":"fooDataSource","cnt":2,"el":17}

0 Karma
Highlighted

Re: How to uncompress and index the compressed log data coming from the forwarder

New Member

Does the Splunk also index zlib compressed files or need script to uncompress and then send to Indexer ?

0 Karma
Highlighted

Re: How to uncompress and index the compressed log data coming from the forwarder

New Member

Followup Question

One of a saas provider provides access log files in zlib compressed format.
how is the zlib compressed logs files handled by spunk system ?
Or the files needs to uncompressed first and then send to the Splunk for indexing

0 Karma