All Apps and Add-ons
Highlighted

Splunk App and Add-on for Amazon Web Services: How to index Cloudfront .gz log files?

New Member

I'm trying to ingest Cloudfront logs from an AWS. Currently they are dumped to an S3 bucket in the form of a .gz file.

The only data I'm getting returned is:

 4/4/16
9:05:23.000 AM  
#Version: 1.0
#Fields: date time x-edge-location sc-bytes c-ip cs-method cs(Host) cs-uri-stem sc-status cs(Referer) cs(User-Agent) cs-uri-query cs(Cookie) x-edge-result-type x-edge-request-id x-host-header cs-protocol cs-bytes time-taken x-forwarded-for ssl-protocol ssl-cipher x-edge-response-result-type

    host = **Redacted**
    source = s3://mybucketname/EL90IIKCKI7FS.2016-04-04-12.9f3c0869.gz
    sourcetype = aws:cloudfront:accesslog

This is the result of the input being pointed at the root of the bucket.

However, if I point the input directly at the .gz file within the bucket, it will ingest it and i can see my access logs. This won't work long term because the logfile rolls regularly and spawns a new one.

Is there something I'm missing? Splunk seems to be aware that the .gz file exists when pointed at the root of the bucket, but it doesn't seem to be ingesting the file.

Thanks!

0 Karma
Highlighted

Re: Splunk App and Add-on for Amazon Web Services: How to index Cloudfront .gz log files?

Splunk Employee
Splunk Employee

Sharing the stanza in inputs.conf will help.
1. Please make sure "Prefix" is configured correctly
2. Please make sure the datetime is configured correctly.

Highlighted

Re: Splunk App and Add-on for Amazon Web Services: How to index Cloudfront .gz log files?

New Member

Thanks for your reply.

What should the prefix be?

Here is the stanza from input.conf:

[awss3://svc-aws-splunk-app20160406092105]
awsaccount = myawsaccount
bucket
name = mybucketname
characterset = auto
ct
blacklist = ^$
hostname = s3.amazonaws.com
index = aws
initial
scandatetime = 1971-01-01T00:00:00Z
max
items = 100000
maxretries = 3
polling
interval = 60
recursion_depth = -1
sourcetype = aws:cloudfront:accesslogs

0 Karma
Highlighted

Re: Splunk App and Add-on for Amazon Web Services: How to index Cloudfront .gz log files?

Splunk Employee
Splunk Employee

I think kchen is referring to the "S3 key prefix" which is the key_name parameter in the S3 input. Looking at your input.conf file, it does not appear you have this configured:

http://docs.splunk.com/Documentation/AddOns/latest/AWS/S3
"The key prefix or full key name to scan for available files. The add-on searches all the objects in all directories under this key name. Leave blank to scan all files in the bucket."

0 Karma