Splunk AWS TA S3 Access Logs Bugs

[danint]

Hello,

There are two bugs in the Splunk Add-on for Amazon Web Services regarding S3 Access Logs (using Generic S3 Input)

1) Splunk_TA_aws/bin/splunk_ta_aws/modinputs/generic_s3/aws_s3_common.py line 27
asc.aws_s3_accesslogs: r".+\d{4}-\d{2}-\d{2}-\d{2}-\d{2}-\d{2}-.+$"

Per Amazon S3 Server Access Logging, the log file key format is:
TargetPrefixYYYY-mm-DD-HH-MM-SS-UniqueString, where TargetPrefix is an "(Optional) A prefix for Amazon S3 to assign to all log object keys.".
The regex on line 27 makes TargetPrefix required and fails to fetch logs without a prefix. It should instead be:
asc.aws_s3_accesslogs: r".*\d{4}-\d{2}-\d{2}-\d{2}-\d{2}-\d{2}-.+$"

2) Splunk_TA_aws/default/props.conf line 441

EXTRACT-s3 = ^\s*(?P<bucket_owner>[^\s]+)\s+(?P<bucket_name>[^\s]+)\s+\[(?P<request_time>.+)\]\s+(?P<remote_ip>[^\s]+)\s+(?P<requester>[^\s]+)\s+(?P<request_id>[^\s]+)\s+(?P<operation>[^\s]+)\s+(?P<key>[^\s]+)\s+"(?P<request_uri>.+)"\s+(?P<http_status>[^\s]+)\s+(?P<error_code>[^\s]+)\s+(?P<bytes_sent>[^\s]+)\s+(?P<object_size>[^\s]+)\s+(?P<total_time>[^\s]+)\s+(?P<turn_around_time>[^\s]+)\s+"(?P<referrer>.+)"\s+"(?P<user_agent>.+)"\s+(?P<version_id>[-\w]+)$

Per Amazon S3 Server Access Log Format, AWS "might extend the access log record format by adding new fields to the end of each line. Code that parses server access logs must be written to handle trailing fields that it does not understand."
The regex on line 441 does not allow trailing fields. it should instead be:

EXTRACT-s3 = ^\s*(?P<bucket_owner>[^\s]+)\s+(?P<bucket_name>[^\s]+)\s+\[(?P<request_time>.+)\]\s+(?P<remote_ip>[^\s]+)\s+(?P<requester>[^\s]+)\s+(?P<request_id>[^\s]+)\s+(?P<operation>[^\s]+)\s+(?P<key>[^\s]+)\s+"(?P<request_uri>.+)"\s+(?P<http_status>[^\s]+)\s+(?P<error_code>[^\s]+)\s+(?P<bytes_sent>[^\s]+)\s+(?P<object_size>[^\s]+)\s+(?P<total_time>[^\s]+)\s+(?P<turn_around_time>[^\s]+)\s+"(?P<referrer>.+)"\s+"(?P<user_agent>.+)"\s+(?P<version_id>[-\w]+)

``Or, to preserve the trailing fields, it could be extended to parse them into a "other_fields" field or something similar.

I can't post links, so search for the bolded titles to find the relevant AWS docs.

[yarick]

Guys, is anybody else seeing this issue?

[gbeatty]

I had this issue as well with the newest version. Thanks. It fixed my issue.