All Apps and Add-ons

Splunk AWS TA S3 Access Logs Bugs



There are two bugs in the Splunk Add-on for Amazon Web Services regarding S3 Access Logs (using Generic S3 Input)

1) Splunk_TA_aws/bin/splunk_ta_aws/modinputs/generic_s3/ line 27
asc.aws_s3_accesslogs: r".+\d{4}-\d{2}-\d{2}-\d{2}-\d{2}-\d{2}-.+$"

Per Amazon S3 Server Access Logging, the log file key format is:
TargetPrefixYYYY-mm-DD-HH-MM-SS-UniqueString, where TargetPrefix is an "(Optional) A prefix for Amazon S3 to assign to all log object keys.".
The regex on line 27 makes TargetPrefix required and fails to fetch logs without a prefix. It should instead be:
asc.aws_s3_accesslogs: r".*\d{4}-\d{2}-\d{2}-\d{2}-\d{2}-\d{2}-.+$"

2) Splunk_TA_aws/default/props.conf line 441

EXTRACT-s3 = ^\s*(?P<bucket_owner>[^\s]+)\s+(?P<bucket_name>[^\s]+)\s+\[(?P<request_time>.+)\]\s+(?P<remote_ip>[^\s]+)\s+(?P<requester>[^\s]+)\s+(?P<request_id>[^\s]+)\s+(?P<operation>[^\s]+)\s+(?P<key>[^\s]+)\s+"(?P<request_uri>.+)"\s+(?P<http_status>[^\s]+)\s+(?P<error_code>[^\s]+)\s+(?P<bytes_sent>[^\s]+)\s+(?P<object_size>[^\s]+)\s+(?P<total_time>[^\s]+)\s+(?P<turn_around_time>[^\s]+)\s+"(?P<referrer>.+)"\s+"(?P<user_agent>.+)"\s+(?P<version_id>[-\w]+)$

Per Amazon S3 Server Access Log Format, AWS "might extend the access log record format by adding new fields to the end of each line. Code that parses server access logs must be written to handle trailing fields that it does not understand."
The regex on line 441 does not allow trailing fields. it should instead be:

EXTRACT-s3 = ^\s*(?P<bucket_owner>[^\s]+)\s+(?P<bucket_name>[^\s]+)\s+\[(?P<request_time>.+)\]\s+(?P<remote_ip>[^\s]+)\s+(?P<requester>[^\s]+)\s+(?P<request_id>[^\s]+)\s+(?P<operation>[^\s]+)\s+(?P<key>[^\s]+)\s+"(?P<request_uri>.+)"\s+(?P<http_status>[^\s]+)\s+(?P<error_code>[^\s]+)\s+(?P<bytes_sent>[^\s]+)\s+(?P<object_size>[^\s]+)\s+(?P<total_time>[^\s]+)\s+(?P<turn_around_time>[^\s]+)\s+"(?P<referrer>.+)"\s+"(?P<user_agent>.+)"\s+(?P<version_id>[-\w]+)

``Or, to preserve the trailing fields, it could be extended to parse them into a "other_fields" field or something similar.

I can't post links, so search for the bolded titles to find the relevant AWS docs.

Path Finder

Guys, is anybody else seeing this issue?

0 Karma

Path Finder

I had this issue as well with the newest version. Thanks. It fixed my issue.

0 Karma
Get Updates on the Splunk Community!

Observability Highlights | January 2023 Newsletter

 January 2023New Product Releases Splunk Network Explorer for Infrastructure MonitoringSplunk unveils Network ...

Security Highlights | January 2023 Newsletter

January 2023 Splunk Security Essentials (SSE) 3.7.0 ReleaseThe free Splunk Security Essentials (SSE) 3.7.0 app ...

Platform Highlights | January 2023 Newsletter

 January 2023Peace on Earth and Peace of Mind With Business ResilienceAll organizations can start the new year ...