Getting Data In

Gziped file to be directly processed by Splunk

Said75015
Explorer

Hi,

I have several files on a AWS s3 bucket and I have configured an input to get data from these files. Is there a way to make Splunk process them without unzip them manually ? 

I use Splunk Enteprise 8.2.2.1 and AWS plugin.

Thanks for your help

Saïd

 

Labels (2)
0 Karma

Said75015
Explorer

I confirm that auto gzip files doesn' t work for me with the AWS add-on. Do I need to explicit it ?

Here is a sample input test entry for which it doesn't work

Said75015_0-1634316061448.png

 

0 Karma

venkatasri
SplunkTrust
SplunkTrust

@Said75015 

You can try looking at internal to find out the gzip files being downloaded from S3 before ingestion. Add-on will be able to do the auto extraction. 

Try - 

index=_internal sourcetype=aws:s3:log

https://docs.splunk.com/Documentation/AddOns/released/AWS/Troubleshooting

 

0 Karma

venkatasri
SplunkTrust
SplunkTrust

Hi @Said75015 

AWS Plugin you mean the AWS Add-on?  Add-on does the extract internally and ingest the logs wihtout manual intervention. How is your gzip archive one file having all wrapped in them or they individually gzipped?

Have you tested it  in test env?

 

Said75015
Explorer

Hi @venkatasri 

Yes I mean AWS Add-on, files are individually gzipped. I tested it in a local test env.

I'll retry and keep you informed

Saïd

 

0 Karma
Get Updates on the Splunk Community!

New Dates, New City: Save the Date for .conf25!

Wake up, babe! New .conf25 dates AND location just dropped!! That's right, this year, .conf25 is taking place ...

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud  In today’s fast-paced digital ...

Observability protocols to know about

Observability protocols define the specifications or formats for collecting, encoding, transporting, and ...