All Apps and Add-ons

Splunk_TA_aws - guardduty Time parsing.

Contributor

The raw data looks like:

... blah, blah, blah ... "detail-type": "GuardDuty Finding", "time": "2019-03-14T14:40:39Z"}

On our Heavy Forwarder I've setup in SplunkTAaws/local

[aws:kinesis]
TIMEPREFIX = \s+\"time\":\s+\"
TIME
FORMAT = %Y-%m-%dT%H:%M:%SZ
MAXTIMESTAMPLOOKAHEAD = 40960

The regex works on the standalone Splunk instance I have on my laptop, it works on regex101.com

But when the data is indexed, the time Splunk indexes on my cluster is: 3/14/19 2:46:29.090 PM

Any clues as to what might be happening here?

0 Karma

Contributor

Support came up with the fix for this issue. Setting use_hec = false under [global_settings] in aws_kinesis.conf resolved my problem, and the time being stamped now matches the time in the json.

0 Karma

SplunkTrust
SplunkTrust

en-US may be? (locale)

0 Karma

Contributor

Thank you ...
LANG is en_US.UTF-8, CHARSET (in btool) is also UTF-8

Same goes for my laptop.

Cheers - PK

0 Karma

Ultra Champion

If you switch your locale to en_GB or another locale which uses the DAY/MONTH/YEAR format, it will reformat the timestamps for you.

0 Karma

Contributor

Thank you. I'm not looking to have the time reformatted. Only to match what's in the event, yet no matter what I do, Splunk seems to set _time to the time that the event was indexed.

detail-type                  time                       _time                     index_time
GuardDuty Finding  2019-03-15T16:30:02Z        2019-03-15 16:31:03.682    03/15/2019 16:31:04
GuardDuty Finding  2019-03-15T16:30:06Z        2019-03-15 16:31:01.686    03/15/2019 16:31:02
GuardDuty Finding  2019-03-15T16:25:11Z        2019-03-15 16:26:13.278    03/15/2019 16:26:13
0 Karma

Ultra Champion

I understand now.

Your max time stamp lookahead is wrong.

Set it to 20

The lookahead begins after the prefix match, not the start of the event, so smaller is better and faster, and could be your issue.

0 Karma

Contributor

Thanks again,

Yes, I've played around with multiple values for lookahead and still end up with about a 1 minute difference between the value of 'time" and what Splunk stamps ... I've opened up a ticket with support. It appears that the collection is doing its' own thing. ( Not the first issue we've had with the AWS Add-On. )

Cheers.

0 Karma