All Apps and Add-ons

AWS Cloudfront

Explorer

I've recently setup Splunk Add-on for Amazon Web Services and noticed that there was no props file for importing AWS Cloudfront logs from the S3 buckets. I'm fairly new to Splunk but took a shot at trying to build one which so far hasn't work out. So I figure I reach out here and see if anyone has built a props file for Cloudfront?

Splunk Employee
Splunk Employee

Since AWS CloudFront access logs use W3C extended log file format, the following works fine when ingesting them as raw files:
props.conf

[cloudfront-access]
INDEXED_EXTRACTIONS = W3C

This enables automatic header-based field extractions by Splunk at index-time.
Keep in mind this means Splunk will store all these fields in the index, thereby increasing the size of the searchable index.

However:
As of current Splunk 6.3 release, the above header-based indexed extractions is not supported with modular inputs (vs one shot or monitored files). Unfortunately, importing logs from S3 is commonly done via modular input. So I found the following configs for search-time extractions to be a reasonable alternative (with one caveat below):

props.conf:

[cloudfront-access-modinput]
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE = false
TRANSFORMS-1trash = filter_preamble
REPORT-1 = cloudfront_fields

transforms.conf:

# Field Extractions for CloudFront Access Logs (for modular input only; otherwise use w3c)
[cloudfront_fields]
DELIMS = "\t"
FIELDS = date,time,x-edge-location,sc-bytes,c-ip,cs-method,cs-host,cs-uri-stem,sc-status,cs-referer,cs-user-agent,cs-uri-query,cs-cookie,x-edge-result-type,x-edge-request-id,x-host-header,cs-protocol,cs-bytes,time-taken,x-forwarded-for,ssl-protocol,ssl-cipher,x-edge,response-result-type

[filter_preamble]
DEST_KEY = queue
REGEX = (?m)^\#(Version|Fields)
FORMAT = nullQueue

Caveat: the above hardcoded list of fields is based on the current CloudFront file format version 1.0, and may change the future. Something you would not have to worry about if you're using INDEXED_EXTRACTIONS = W3C

Motivator

Does that limitation exist for any indexed extraction, including json?

0 Karma

Splunk Employee
Splunk Employee

That's correct. This modular input limitation exists at least for now (Splunk 6.3).

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!