topic Gziped file to be directly processed by Splunk in Getting Data In

Gziped file to be directly processed by Splunk

Said75015 — Mon, 11 Oct 2021 09:18:16 GMT

Hi,

I have several files on a AWS s3 bucket and I have configured an input to get data from these files. Is there a way to make Splunk process them without unzip them manually ?

I use Splunk Enteprise 8.2.2.1 and AWS plugin.

Thanks for your help

Saïd

Re: Gziped file to be directly processed by Splunk

venkatasri — Thu, 14 Oct 2021 00:17:57 GMT

Hi @Said75015

AWS Plugin you mean the AWS Add-on? Add-on does the extract internally and ingest the logs wihtout manual intervention. How is your gzip archive one file having all wrapped in them or they individually gzipped?

Have you tested it in test env?

Re: Gziped file to be directly processed by Splunk

Said75015 — Fri, 15 Oct 2021 08:33:54 GMT

Hi @venkatasri

Yes I mean AWS Add-on, files are individually gzipped. I tested it in a local test env.

I'll retry and keep you informed

Saïd

Re: Gziped file to be directly processed by Splunk

Said75015 — Fri, 15 Oct 2021 16:41:38 GMT

I confirm that auto gzip files doesn' t work for me with the AWS add-on. Do I need to explicit it ?

Here is a sample input test entry for which it doesn't work

Re: Gziped file to be directly processed by Splunk

venkatasri — Mon, 18 Oct 2021 05:56:57 GMT

@Said75015

You can try looking at internal to find out the gzip files being downloaded from S3 before ingestion. Add-on will be able to do the auto extraction.

Try -

index=_internal sourcetype=aws:s3:log

https://docs.splunk.com/Documentation/AddOns/released/AWS/Troubleshooting