Getting Data In

How to uncompress and index the compressed log data coming from the forwarder

Sriram
Communicator

I have few application which sends application logs in both compressed and uncompressed format. There is a log attribute which identifies what the format of the log data is, so I can create 2 different source types. At the indexing time, i need to uncompress the data(compress) and feed to splunk for further analysis.

Is it possible to uncompress the data at index time ? What is the recommended approach to do this ?

0 Karma

lguinn2
Legend

If you set up an input on the forwarder like this:

[monitor:///path/to/files]
etc.

It will just work for the compressed files that it finds.

Splunk decompresses archive files before it indexes them. It can handle these common archive file types: tar, gz, bz2, tar.gz, tgz, tbz, tbz2, zip, and z. You do not need to do anything to these files except pass them to Splunk.

abhisheksplunka
New Member

Followup Question

One of a saas provider provides access log files in zlib compressed format.
how is the zlib compressed logs files handled by spunk system ?
Or the files needs to uncompressed first and then send to the Splunk for indexing

0 Karma

abhisheksplunka
New Member

Does the Splunk also index zlib compressed files or need script to uncompress and then send to Indexer ?

0 Karma

Sriram
Communicator

main INFO TxnSummaryLogWriter - LOGTYPE=COMPRESSED, JSON_TXN_SUMMARIES H4sIAAAAAAAAANWbW4vaQBiG/8tcZyEzE 3O63Cil..../veCqx1ATmg4KJlgB5AT3rsDT2P+LFI2qwuOHP4C9fIIkZIyAAA=
main INFO TxnSummaryLogWriter - LOGTYPE=UNCOMPRESSED, JSON_TXN_SUMMARIES {"txnsummaries":[{"el":82,"cpu":15, "startTime":"2014-03-12T16:31:02","metadata":{"dummyParam":"foo0","DCN":"143"},"integrations":[{"cat":"JDBC","id":"fooDataSource","cnt":2,"el":17}

0 Karma

Sriram
Communicator

Thank You. The problem is the logs are not coming as zip file type. It will be in application.log which has both compressed and uncompressed data mixed together. Here is an example from the application.log. Based on the LOGTYPE=COMPRESSED, I can identify what type of data this event is. At the Index time, I want to be able to uncompress the data and index it. (I have added the rest of the details below.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Using the Splunk Threat Research Team’s Latest Security Content

REGISTER HERE Tech Talk | Security Edition Did you know the Splunk Threat Research Team regularly releases ...

SplunkTrust | 2024 SplunkTrust Application Period is Open!

It's that time again, folks! That's right, the application/nomination period for the 2024 SplunkTrust is ...