Getting Data In

In Splunk, where and when are data.gz files unzipped?

Log_wrangler
Builder

If I have an app on a heavy forwarder that is pulling in .gz files and sending them to my indexers (i.e. distributed environment), when and where are the files unzipped?

Are the files unzipped at the heavy forwarder or sent zipped to the indexers?

I am trying to troubleshoot why I am seeing unreadable garbage in the search head with a source of file.gz...

It appears that the file was not unzipped.

Please advise.

Thank you

0 Karma

MuS
SplunkTrust
SplunkTrust

Hi Log_wrangler,

Compressed files (like .gz and .zip) are handled by the Archive processor, and are processed in serial. The Archive processor reads the compressed file during the input phase, on your HWF, and the HWF will send the uncompressed events to the indexer.

Unreadable garbage could be either you are having actual unreadable garbage in the file or the indexer is listening on TCP <YourPortNumberHere> instead of SplunkTCP. Sometimes it could also be related to forwarder sending using SSL and receiver not using SSL or vice versa.

Hope this helps ...

cheers, MuS

0 Karma

MuS
SplunkTrust
SplunkTrust

Just another hint regarding the garbage events, check the charset of the file and set it in the props.conf. See the docs for more http://docs.splunk.com/Documentation/Splunk/latest/Troubleshooting/Garbledevents#Symptom

cheers, MuS

0 Karma

Log_wrangler
Builder

So this issue is specifically related to an app residing on a HF.
The Splunk_TA_microsoft-cloudservices unzips files in tables inputs but not blobs inputs.
We can unzip the file with the indexer directly (as a local file input) so it appears the problem is in the app.
Any other ideas how to find the root cause?

Thanks

0 Karma

Log_wrangler
Builder

thank you for your response.

0 Karma
Get Updates on the Splunk Community!

Synthetic Monitoring: Not your Grandma’s Polyester! Tech Talk: DevOps Edition

Register today and join TekStream on Tuesday, February 28 at 11am PT/2pm ET for a demonstration of Splunk ...

Instrumenting Java Websocket Messaging

Instrumenting Java Websocket MessagingThis article is a code-based discussion of passing OpenTelemetry trace ...

Announcing General Availability of Splunk Incident Intelligence!

Digital transformation is real! Across industries, companies big and small are going through rapid digital ...