The raw data looks like:
... blah, blah, blah ... "detail-type": "GuardDuty Finding", "time": "2019-03-14T14:40:39Z"}
On our Heavy Forwarder I've setup in SplunkTAaws/local
TIMEPREFIX = \s+\"time\":\s+\"
TIMEFORMAT = %Y-%m-%dT%H:%M:%SZ
MAXTIMESTAMPLOOKAHEAD = 40960
The regex works on the standalone Splunk instance I have on my laptop, it works on regex101.com
But when the data is indexed, the time Splunk indexes on my cluster is: 3/14/19 2:46:29.090 PM
Any clues as to what might be happening here?
Support came up with the fix for this issue. Setting use_hec = false under [global_settings] in aws_kinesis.conf resolved my problem, and the time being stamped now matches the time in the json.
Thank you. I'm not looking to have the time reformatted. Only to match what's in the event, yet no matter what I do, Splunk seems to set _time to the time that the event was indexed.
detail-type time _time index_time GuardDuty Finding 2019-03-15T16:30:02Z 2019-03-15 16:31:03.682 03/15/2019 16:31:04 GuardDuty Finding 2019-03-15T16:30:06Z 2019-03-15 16:31:01.686 03/15/2019 16:31:02 GuardDuty Finding 2019-03-15T16:25:11Z 2019-03-15 16:26:13.278 03/15/2019 16:26:13
I understand now.
Your max time stamp lookahead is wrong.
Set it to
The lookahead begins after the prefix match, not the start of the event, so smaller is better and faster, and could be your issue.
Yes, I've played around with multiple values for lookahead and still end up with about a 1 minute difference between the value of 'time" and what Splunk stamps ... I've opened up a ticket with support. It appears that the collection is doing its' own thing. ( Not the first issue we've had with the AWS Add-On. )