All Apps and Add-ons

AWS Cloudfront

davidhofmann
Explorer

I've recently setup Splunk Add-on for Amazon Web Services and noticed that there was no props file for importing AWS Cloudfront logs from the S3 buckets. I'm fairly new to Splunk but took a shot at trying to build one which so far hasn't work out. So I figure I reach out here and see if anyone has built a props file for Cloudfront?

rarsan_splunk
Splunk Employee
Splunk Employee

Since AWS CloudFront access logs use W3C extended log file format, the following works fine when ingesting them as raw files:
props.conf

[cloudfront-access]
INDEXED_EXTRACTIONS = W3C

This enables automatic header-based field extractions by Splunk at index-time.
Keep in mind this means Splunk will store all these fields in the index, thereby increasing the size of the searchable index.

However:
As of current Splunk 6.3 release, the above header-based indexed extractions is not supported with modular inputs (vs one shot or monitored files). Unfortunately, importing logs from S3 is commonly done via modular input. So I found the following configs for search-time extractions to be a reasonable alternative (with one caveat below):

props.conf:

[cloudfront-access-modinput]
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE = false
TRANSFORMS-1trash = filter_preamble
REPORT-1 = cloudfront_fields

transforms.conf:

# Field Extractions for CloudFront Access Logs (for modular input only; otherwise use w3c)
[cloudfront_fields]
DELIMS = "\t"
FIELDS = date,time,x-edge-location,sc-bytes,c-ip,cs-method,cs-host,cs-uri-stem,sc-status,cs-referer,cs-user-agent,cs-uri-query,cs-cookie,x-edge-result-type,x-edge-request-id,x-host-header,cs-protocol,cs-bytes,time-taken,x-forwarded-for,ssl-protocol,ssl-cipher,x-edge,response-result-type

[filter_preamble]
DEST_KEY = queue
REGEX = (?m)^\#(Version|Fields)
FORMAT = nullQueue

Caveat: the above hardcoded list of fields is based on the current CloudFront file format version 1.0, and may change the future. Something you would not have to worry about if you're using INDEXED_EXTRACTIONS = W3C

Jeremiah
Motivator

Does that limitation exist for any indexed extraction, including json?

0 Karma

rarsan_splunk
Splunk Employee
Splunk Employee

That's correct. This modular input limitation exists at least for now (Splunk 6.3).

0 Karma
Get Updates on the Splunk Community!

How to Monitor Google Kubernetes Engine (GKE)

We’ve looked at how to integrate Kubernetes environments with Splunk Observability Cloud, but what about ...

Index This | How can you make 45 using only 4?

October 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this ...

Splunk Education Goes to Washington | Splunk GovSummit 2024

If you’re in the Washington, D.C. area, this is your opportunity to take your career and Splunk skills to the ...