All Apps and Add-ons

Splunk App for AWS: Why is Cloudwatch data not displaying with error "A possible timestamp match is outside of the acceptable time window"?

New Member

Hi

We are having an issue with the Splunk App for AWS not displaying Cloudtrail info e.g. (VPC Flow Logs - Security)

The Splunk Add-on for Amazon Web Services is receiving data from AWS i.e. if I search index=aws-cloudwatchlogs, I get results returned of the form:

2 968645151068 eni-5e026f04 10.68.23.116 10.68.3.220 389 53532 6 7 486 1456224314 1456224370 ACCEPT OK
host = ourhost.com source = eu-west-1:FlowLogs/vpc-xxxxxxx:eni-5e026f04-all sourcetype = aws:cloudwatchlogs:vpcflow

The splunkd.log indicates repeated WARN's entries of the form:

02-23-2016 10:36:56.156 +0000 WARN  DateParserVerbose - A possible timestamp match (Mon Sep 11 04:05:51 2000) is outside of the acceptable time window. If this timestamp is correct, consider adjusting MAX_DAYS_AGO and MAX_DAYS_HENCE. Context: source::eu-west-1:FlowLogs/vpc-xxxxxxx:eni-0ba23051-all|host::ourhost.com|aws:cloudwatchlogs:vpcflow|

Other AWS input is being received correctly e.g. Billing, Description, Config

The datetime in the error message (Mon Sep 11 04:05:51 2000), correlates to our account number (the account id is embedded in 1 of the raw fields (using this http://www.onlineconversion.com/unix_time.htm)

Any ideas as to what is going wrong / where to look? Would be appreciated.

Thanks

0 Karma

Esteemed Legend

This is a clear indication that the events that you are sending into Splunk are mis-timestamped. Splunk will only allow timestamps to deviate from "now" by a few days forwards (default is 2) or backwards (default is 2000). If the timestamp that splunk identifies inside of your event it outside of this window, the event will be given

http://docs.splunk.com/Documentation/Splunk/6.3.3/Data/Configuretimestamprecognition#Edit_timestamp_...

You need to take a look at your timestamp configuration definitions in props.conf and compare them with your events. If this is correct, then you need to make sure that you do not have a timezone issue.

0 Karma