topic Reading AWS s3 gz files without extension in Splunk? in Getting Data In

Reading AWS s3 gz files without extension in Splunk?

prash — Wed, 24 May 2023 14:24:55 GMT

I am having difficulties to get Splunk to ingest gzipped logs files from an S3 bucket, the files itself do not have extensions and Splunk is reading them as binaries.

I tried archive_cmd to auto, gunzip -c, gzip -d in props.conf with no luck

[source::/xxx/*]

unarchive_cmd = gunzip -c

NO_BINARY_CHECK = true

gunzip -c works in shell, gzip -d doesn't without gz suffix

*using AWS addon

due to the nature of the environment, the files can't be renamed. Anyone experienced this before?

Re: Reading AWS s3 gz files without extension in Splunk?

richgalloway — Wed, 24 May 2023 15:21:55 GMT

Have you tried adding invalid_cause=archive to the stanza? The docs have conflicting information about it, but it's worth a try.

Re: Reading AWS s3 gz files without extension in Splunk?

prash — Wed, 24 May 2023 23:26:02 GMT

I did try that, made no difference. As per docs, the unarchive_cmd is only invoked when invalid_cause is specified.

Re: Reading AWS s3 gz files without extension in Splunk?

richgalloway — Thu, 25 May 2023 00:05:12 GMT

I noticed the contradiction in the docs. I suggest contacting Splunk Support.