I'm trying to ingest Cloudfront logs from an AWS. Currently they are dumped to an S3 bucket in the form of a .gz file.
The only data I'm getting returned is:
4/4/16 9:05:23.000 AM #Version: 1.0 #Fields: date time x-edge-location sc-bytes c-ip cs-method cs(Host) cs-uri-stem sc-status cs(Referer) cs(User-Agent) cs-uri-query cs(Cookie) x-edge-result-type x-edge-request-id x-host-header cs-protocol cs-bytes time-taken x-forwarded-for ssl-protocol ssl-cipher x-edge-response-result-type host = **Redacted** source = s3://mybucketname/EL90IIKCKI7FS.2016-04-04-12.9f3c0869.gz sourcetype = aws:cloudfront:accesslog
This is the result of the input being pointed at the root of the bucket.
However, if I point the input directly at the .gz file within the bucket, it will ingest it and i can see my access logs. This won't work long term because the logfile rolls regularly and spawns a new one.
Is there something I'm missing? Splunk seems to be aware that the .gz file exists when pointed at the root of the bucket, but it doesn't seem to be ingesting the file.
Sharing the stanza in inputs.conf will help.
1. Please make sure "Prefix" is configured correctly
2. Please make sure the datetime is configured correctly.
Thanks for your reply.
What should the prefix be?
Here is the stanza from input.conf:
awsaccount = myawsaccount
bucketname = mybucketname
characterset = auto
ctblacklist = ^$
hostname = s3.amazonaws.com
index = aws
initialscandatetime = 1971-01-01T00:00:00Z
maxitems = 100000
maxretries = 3
pollinginterval = 60
recursion_depth = -1
sourcetype = aws:cloudfront:accesslogs
I think kchen is referring to the "S3 key prefix" which is the key_name parameter in the S3 input. Looking at your input.conf file, it does not appear you have this configured:
"The key prefix or full key name to scan for available files. The add-on searches all the objects in all directories under this key name. Leave blank to scan all files in the bucket."