topic Re: Source type Timestamp settings. in Getting Data In

Source type Timestamp settings.

spilepich — Wed, 18 Jul 2018 06:07:07 GMT

Hi,

I'm trying to set up a source type that parses the date from an inner field (message.date in the below example) however the _time value is always set as the processed time. Any guesses as to what i might be doing wrong?

The configuration is as follows:

This is a sample event.
{"message":{"_id":"some_value","date":"2018-07-18T04:40:58.071Z","type":"fsa","description":"Login required","client_id":"some_value","client_name":"some_value","ip":"some_value","user_agent":"Chrome 67.0.3396 / Windows 10 0.0.0","details":{"body":{"tenant":"some_value"},"qs":{"client_id":"some_value","response_type":"id_token","response_mode":"web_message","redirect_uri":"some_value","scope":"openid email profile","audience":"some_value","leeway":"60","state":"some_value","nonce":"some_value","prompt":"none","auth0Client":"some_value","tenant":"hirer"},"connection":null,"error":{"message":"Login required","oauthError":"login_required","type":"oauth-authorization"}},"hostname":"a","session_connection":null,"session_connection_id":null,"audience":"o","scope":["openid","email","profile"],"isMobile":false},"severity":"info"}

Re: Source type Timestamp settings.

poete — Wed, 18 Jul 2018 14:28:00 GMT

Hello @spilepich,

why bother with configuring the extraction? I did import the sample of data successfuly using the Auto Extraction.

But maybe you are trying to do something I did not get. Did, you try with Auto?

Re: Source type Timestamp settings.

CarsonZa — Wed, 18 Jul 2018 15:21:06 GMT

i would try what @poete is suggesting instead of changing it to _json leave it as default and then add your timestamp format in. I tested it with your sample event and it works great.

Re: Source type Timestamp settings.

spilepich — Thu, 19 Jul 2018 06:26:02 GMT

yes, that's the first thing i tried. The data imports with the load time as _time instead of what is in message.date

Re: Source type Timestamp settings.

poete — Thu, 19 Jul 2018 06:53:53 GMT

Hello @spilepich,
Well, I think you get both. In _time, you get the value extracted from message.date, which enable to display it according to the user timezone, as configured in Splunk. In message.date, you get the value contained in the original json event.

This is the standard behaviour in Splunk

Re: Source type Timestamp settings.

eavent_splunk — Tue, 04 Sep 2018 03:07:42 GMT

This was a HEC input going to the standard endpoint, which expects data to be formatted a certain way:
http://docs.splunk.com/Documentation/Splunk/7.1.2/Data/FormateventsforHTTPEventCollector#Event_metadata

It will not extract timestamps from JSON fields other than "time" in epoch format.
To resolve, use the raw endpoint, which will do default field extraction as per sourcetype configuration:
http://docs.splunk.com/Documentation/Splunk/7.1.2/Data/FormateventsforHTTPEventCollector#Raw_event_parsing

Hope that helps!