Getting Data In

Gziped file to be directly processed by Splunk

Said75015
Explorer

Hi,

I have several files on a AWS s3 bucket and I have configured an input to get data from these files. Is there a way to make Splunk process them without unzip them manually ? 

I use Splunk Enteprise 8.2.2.1 and AWS plugin.

Thanks for your help

Saïd

 

Labels (2)
0 Karma

Said75015
Explorer

I confirm that auto gzip files doesn' t work for me with the AWS add-on. Do I need to explicit it ?

Here is a sample input test entry for which it doesn't work

Said75015_0-1634316061448.png

 

0 Karma

venkatasri
Influencer

@Said75015 

You can try looking at internal to find out the gzip files being downloaded from S3 before ingestion. Add-on will be able to do the auto extraction. 

Try - 

index=_internal sourcetype=aws:s3:log

https://docs.splunk.com/Documentation/AddOns/released/AWS/Troubleshooting

 

0 Karma

venkatasri
Influencer

Hi @Said75015 

AWS Plugin you mean the AWS Add-on?  Add-on does the extract internally and ingest the logs wihtout manual intervention. How is your gzip archive one file having all wrapped in them or they individually gzipped?

Have you tested it  in test env?

 

Said75015
Explorer

Hi @venkatasri 

Yes I mean AWS Add-on, files are individually gzipped. I tested it in a local test env.

I'll retry and keep you informed

Saïd

 

0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!