All Apps and Add-ons

Splunk_TA_aws - guardduty Time parsing.

pkeller
Contributor

The raw data looks like:

... blah, blah, blah ... "detail-type": "GuardDuty Finding", "time": "2019-03-14T14:40:39Z"}

On our Heavy Forwarder I've setup in Splunk_TA_aws/local

[aws:kinesis]
TIME_PREFIX = \s+\"time\":\s+\"
TIME_FORMAT = %Y-%m-%dT%H:%M:%SZ
MAX_TIMESTAMP_LOOKAHEAD = 40960

The regex works on the standalone Splunk instance I have on my laptop, it works on regex101.com

But when the data is indexed, the time Splunk indexes on my cluster is: 3/14/19 2:46:29.090 PM

Any clues as to what might be happening here?

0 Karma

pkeller
Contributor

Support came up with the fix for this issue. Setting use_hec = false under [global_settings] in aws_kinesis.conf resolved my problem, and the time being stamped now matches the time in the json.

0 Karma

lakshman239
SplunkTrust
SplunkTrust

en-US may be? (locale)

0 Karma

pkeller
Contributor

Thank you ...
LANG is en_US.UTF-8, CHARSET (in btool) is also UTF-8

Same goes for my laptop.

Cheers - PK

0 Karma

nickhills
Ultra Champion

If you switch your locale to en_GB or another locale which uses the DAY/MONTH/YEAR format, it will reformat the timestamps for you.

If my comment helps, please give it a thumbs up!
0 Karma

pkeller
Contributor

Thank you. I'm not looking to have the time reformatted. Only to match what's in the event, yet no matter what I do, Splunk seems to set _time to the time that the event was indexed.

detail-type                  time                       _time                     index_time
GuardDuty Finding  2019-03-15T16:30:02Z        2019-03-15 16:31:03.682    03/15/2019 16:31:04
GuardDuty Finding  2019-03-15T16:30:06Z        2019-03-15 16:31:01.686    03/15/2019 16:31:02
GuardDuty Finding  2019-03-15T16:25:11Z        2019-03-15 16:26:13.278    03/15/2019 16:26:13
0 Karma

nickhills
Ultra Champion

I understand now.

Your max time stamp lookahead is wrong.

Set it to 20

The lookahead begins after the prefix match, not the start of the event, so smaller is better and faster, and could be your issue.

If my comment helps, please give it a thumbs up!
0 Karma

pkeller
Contributor

Thanks again,

Yes, I've played around with multiple values for lookahead and still end up with about a 1 minute difference between the value of 'time" and what Splunk stamps ... I've opened up a ticket with support. It appears that the collection is doing its' own thing. ( Not the first issue we've had with the AWS Add-On. )

Cheers.

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...