I've recently setup Splunk Add-on for Amazon Web Services and noticed that there was no props file for importing AWS Cloudfront logs from the S3 buckets. I'm fairly new to Splunk but took a shot at trying to build one which so far hasn't work out. So I figure I reach out here and see if anyone has built a props file for Cloudfront?
Since AWS CloudFront access logs use W3C extended log file format, the following works fine when ingesting them as raw files:
props.conf
[cloudfront-access]
INDEXED_EXTRACTIONS = W3C
This enables automatic header-based field extractions by Splunk at index-time.
Keep in mind this means Splunk will store all these fields in the index, thereby increasing the size of the searchable index.
However:
As of current Splunk 6.3 release, the above header-based indexed extractions is not supported with modular inputs (vs one shot or monitored files). Unfortunately, importing logs from S3 is commonly done via modular input. So I found the following configs for search-time extractions to be a reasonable alternative (with one caveat below):
props.conf:
[cloudfront-access-modinput]
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE = false
TRANSFORMS-1trash = filter_preamble
REPORT-1 = cloudfront_fields
transforms.conf:
# Field Extractions for CloudFront Access Logs (for modular input only; otherwise use w3c)
[cloudfront_fields]
DELIMS = "\t"
FIELDS = date,time,x-edge-location,sc-bytes,c-ip,cs-method,cs-host,cs-uri-stem,sc-status,cs-referer,cs-user-agent,cs-uri-query,cs-cookie,x-edge-result-type,x-edge-request-id,x-host-header,cs-protocol,cs-bytes,time-taken,x-forwarded-for,ssl-protocol,ssl-cipher,x-edge,response-result-type
[filter_preamble]
DEST_KEY = queue
REGEX = (?m)^\#(Version|Fields)
FORMAT = nullQueue
Caveat: the above hardcoded list of fields is based on the current CloudFront file format version 1.0, and may change the future. Something you would not have to worry about if you're using INDEXED_EXTRACTIONS = W3C
Does that limitation exist for any indexed extraction, including json?
That's correct. This modular input limitation exists at least for now (Splunk 6.3).