Getting Data In

How to uncompress and index the compressed log data coming from the forwarder

Sriram
Communicator

I have few application which sends application logs in both compressed and uncompressed format. There is a log attribute which identifies what the format of the log data is, so I can create 2 different source types. At the indexing time, i need to uncompress the data(compress) and feed to splunk for further analysis.

Is it possible to uncompress the data at index time ? What is the recommended approach to do this ?

0 Karma

lguinn2
Legend

If you set up an input on the forwarder like this:

[monitor:///path/to/files]
etc.

It will just work for the compressed files that it finds.

Splunk decompresses archive files before it indexes them. It can handle these common archive file types: tar, gz, bz2, tar.gz, tgz, tbz, tbz2, zip, and z. You do not need to do anything to these files except pass them to Splunk.

abhisheksplunka
New Member

Followup Question

One of a saas provider provides access log files in zlib compressed format.
how is the zlib compressed logs files handled by spunk system ?
Or the files needs to uncompressed first and then send to the Splunk for indexing

0 Karma

abhisheksplunka
New Member

Does the Splunk also index zlib compressed files or need script to uncompress and then send to Indexer ?

0 Karma

Sriram
Communicator

main INFO TxnSummaryLogWriter - LOGTYPE=COMPRESSED, JSON_TXN_SUMMARIES H4sIAAAAAAAAANWbW4vaQBiG/8tcZyEzE 3O63Cil..../veCqx1ATmg4KJlgB5AT3rsDT2P+LFI2qwuOHP4C9fIIkZIyAAA=
main INFO TxnSummaryLogWriter - LOGTYPE=UNCOMPRESSED, JSON_TXN_SUMMARIES {"txnsummaries":[{"el":82,"cpu":15, "startTime":"2014-03-12T16:31:02","metadata":{"dummyParam":"foo0","DCN":"143"},"integrations":[{"cat":"JDBC","id":"fooDataSource","cnt":2,"el":17}

0 Karma

Sriram
Communicator

Thank You. The problem is the logs are not coming as zip file type. It will be in application.log which has both compressed and uncompressed data mixed together. Here is an example from the application.log. Based on the LOGTYPE=COMPRESSED, I can identify what type of data this event is. At the Index time, I want to be able to uncompress the data and index it. (I have added the rest of the details below.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...