Getting Data In

In Splunk, where and when are data.gz files unzipped?

Log_wrangler
Builder

If I have an app on a heavy forwarder that is pulling in .gz files and sending them to my indexers (i.e. distributed environment), when and where are the files unzipped?

Are the files unzipped at the heavy forwarder or sent zipped to the indexers?

I am trying to troubleshoot why I am seeing unreadable garbage in the search head with a source of file.gz...

It appears that the file was not unzipped.

Please advise.

Thank you

0 Karma

MuS
Legend

Hi Log_wrangler,

Compressed files (like .gz and .zip) are handled by the Archive processor, and are processed in serial. The Archive processor reads the compressed file during the input phase, on your HWF, and the HWF will send the uncompressed events to the indexer.

Unreadable garbage could be either you are having actual unreadable garbage in the file or the indexer is listening on TCP <YourPortNumberHere> instead of SplunkTCP. Sometimes it could also be related to forwarder sending using SSL and receiver not using SSL or vice versa.

Hope this helps ...

cheers, MuS

0 Karma

MuS
Legend

Just another hint regarding the garbage events, check the charset of the file and set it in the props.conf. See the docs for more http://docs.splunk.com/Documentation/Splunk/latest/Troubleshooting/Garbledevents#Symptom

cheers, MuS

0 Karma

Log_wrangler
Builder

So this issue is specifically related to an app residing on a HF.
The Splunk_TA_microsoft-cloudservices unzips files in tables inputs but not blobs inputs.
We can unzip the file with the indexer directly (as a local file input) so it appears the problem is in the app.
Any other ideas how to find the root cause?

Thanks

0 Karma

Log_wrangler
Builder

thank you for your response.

0 Karma
Get Updates on the Splunk Community!

Introducing Splunk Enterprise 9.2

WATCH HERE! Watch this Tech Talk to learn about the latest features and enhancements shipped in the new Splunk ...

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...