Getting Data In

Reading AWS s3 gz files without extension in Splunk?

prash
Loves-to-Learn Everything

I am having difficulties to get Splunk to ingest gzipped logs files from an S3 bucket, the files itself do not have extensions and Splunk is reading them as binaries.

I tried archive_cmd to auto, gunzip -c, gzip -d in props.conf with no luck

[source::/xxx/*]

unarchive_cmd = gunzip -c

NO_BINARY_CHECK = true

gunzip -c works in shell, gzip -d doesn't without gz suffix

*using AWS addon

due to the nature of the environment, the files can't be renamed. Anyone experienced this before?

Labels (1)
0 Karma

prash
Loves-to-Learn Everything

I did try that, made no difference. As per docs, the unarchive_cmd is only invoked when invalid_cause is specified.

 

0 Karma

richgalloway
SplunkTrust
SplunkTrust

I noticed the contradiction in the docs.  I suggest contacting Splunk Support.

---
If this reply helps you, Karma would be appreciated.
0 Karma

richgalloway
SplunkTrust
SplunkTrust

Have you tried adding invalid_cause=archive to the stanza?  The docs have conflicting information about it, but it's worth a try.

---
If this reply helps you, Karma would be appreciated.
0 Karma
Get Updates on the Splunk Community!

[Puzzles] Solve, Learn, Repeat: Dynamic formatting from XML events

This challenge was first posted on Slack #puzzles channelFor a previous puzzle, I needed a set of fixed-length ...

Enter the Agentic Era with Splunk AI Assistant for SPL 1.4

  🚀 Your data just got a serious AI upgrade — are you ready? Say hello to the Agentic Era with the ...

Stronger Security with Federated Search for S3, GCP SQL & Australian Threat ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...