All Apps and Add-ons

AWS Cloudfront

davidhofmann
Explorer

I've recently setup Splunk Add-on for Amazon Web Services and noticed that there was no props file for importing AWS Cloudfront logs from the S3 buckets. I'm fairly new to Splunk but took a shot at trying to build one which so far hasn't work out. So I figure I reach out here and see if anyone has built a props file for Cloudfront?

rarsan_splunk
Splunk Employee
Splunk Employee

Since AWS CloudFront access logs use W3C extended log file format, the following works fine when ingesting them as raw files:
props.conf

[cloudfront-access]
INDEXED_EXTRACTIONS = W3C

This enables automatic header-based field extractions by Splunk at index-time.
Keep in mind this means Splunk will store all these fields in the index, thereby increasing the size of the searchable index.

However:
As of current Splunk 6.3 release, the above header-based indexed extractions is not supported with modular inputs (vs one shot or monitored files). Unfortunately, importing logs from S3 is commonly done via modular input. So I found the following configs for search-time extractions to be a reasonable alternative (with one caveat below):

props.conf:

[cloudfront-access-modinput]
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE = false
TRANSFORMS-1trash = filter_preamble
REPORT-1 = cloudfront_fields

transforms.conf:

# Field Extractions for CloudFront Access Logs (for modular input only; otherwise use w3c)
[cloudfront_fields]
DELIMS = "\t"
FIELDS = date,time,x-edge-location,sc-bytes,c-ip,cs-method,cs-host,cs-uri-stem,sc-status,cs-referer,cs-user-agent,cs-uri-query,cs-cookie,x-edge-result-type,x-edge-request-id,x-host-header,cs-protocol,cs-bytes,time-taken,x-forwarded-for,ssl-protocol,ssl-cipher,x-edge,response-result-type

[filter_preamble]
DEST_KEY = queue
REGEX = (?m)^\#(Version|Fields)
FORMAT = nullQueue

Caveat: the above hardcoded list of fields is based on the current CloudFront file format version 1.0, and may change the future. Something you would not have to worry about if you're using INDEXED_EXTRACTIONS = W3C

Jeremiah
Motivator

Does that limitation exist for any indexed extraction, including json?

0 Karma

rarsan_splunk
Splunk Employee
Splunk Employee

That's correct. This modular input limitation exists at least for now (Splunk 6.3).

0 Karma
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In September, the Splunk Threat Research Team had two releases of new security content via the Enterprise ...

New in Observability - Improvements to Custom Metrics SLOs, Log Observer Connect & ...

The latest enhancements to the Splunk observability portfolio deliver improved SLO management accuracy, better ...

Improve Data Pipelines Using Splunk Data Management

  Register Now   This Tech Talk will explore the pipeline management offerings Edge Processor and Ingest ...