topic Timestamp Parsing in JSON in Getting Data In

Timestamp Parsing in JSON

RichaSingh — Mon, 02 Nov 2015 06:15:01 GMT

Hi,

Can anyone help me with best configurations for timestamp parsing (where "DateTime" is the actual time) for following JSON format :

{ [-] 
    AccessKey:  ----------------------
    Account:  some-value
    AccountId:  somevalue
    Action:  GetInstances 
    Class:  ----------------------------------------------------------
    **DateTime:  2007-10-27T00:51:57.91Z** 
    ElapsedTime:  00:00:00 
    File:  null 
    Level:  Trace 
    Line:  null 
    LogId:  ID-347t58734534-6565ghmhmm
    Message:  Vapor URL - https://www.randomURL.com,ImageId)
Header - {
  "Signature": "$^%$#@)(#4t4r65gvjhloojk",
  "AccessKey": "123854y3957349534-0fdjvbkjds90234u02394-23940=dsffsvbkdjadsvfnk0923480",
  "RequestType": 0,
  "SkipOwnerLookup": false,
  "UserAgent": "456934",
  "UserAgentVersion": null,
  "Timestamp": "2007-10-27T00:51:57Z",
  "User": "3254832564873`12",
  "TestMode": false,
  "ScheduledAction": false
}

What I have already tried in props.conf is:

[API]
TIMESTAMP_FIELDS=DateTime 
INDEXED_EXTRACTIONS=json
NO_BINARY_CHECK=true
KV_MODE=none

And also:

[API]
TIME_PREFIX = "\"DateTime\":\"
INDEXED_EXTRACTIONS=json

Also, I have been through the link:

https://answers.splunk.com/answers/104500/transforming-timestamps.html

None of the two seem to be working. Any comments/ suggestions will be highly appreciated.

Re: Timestamp Parsing in JSON

acharlieh — Tue, 29 Sep 2020 07:44:48 GMT

If that's an accurate copy/paste of the data you're attempting to send in, that is not actually JSON formatted text. It's close, but not quite. This could definitely throw a wrench into using INDEXED_EXTRACTIONS. If it was actually JSON text there would be a lot more double quotes at least.

If you're using INDEXED_EXTRACTIONS=json with your sourcetype, the props.conf stanza specifying INDEXED_EXTRACTIONS and all parsing options should live on the originating Splunk instance instead of the usual parsing Splunk instance. (In most environments, this means this configuration is on your universal forwarder instead of your indexer). I would try TIMESTAMP_FIELDS on the UF (assuming you can actually use INDEXED_EXTRACTIONS.)

If you cannot use INDEXED_EXTRACTIONS, don't surround your setting of TIME_PREFIX in quotes. You should also consider also setting TIME_FORMAT and MAX_TIMESTAMP_LOOKAHEAD if the timestamp is consistent.

Re: Timestamp Parsing in JSON

SierraX — Fri, 30 Sep 2016 05:55:01 GMT

Hi,
got a similar problem at an customer and worked yesterday on a solution based on your Question and the first answer, on a testsetup.

The Splunk Server has a different system time than the sender (see in the lower half)

The JSON is send by following python line:

response = urllib2.urlopen(req, json.dumps({"sourcetype":"indoor_tph", "source":"", "host":"", "event": {"temperature": str(sense.get_temperature()-7.0), "timestamp": str(datetime.utcnow().strftime('%Y-%m-%dT%H:%M:%S.%f')), "pressure": str(sense.get_pressure()), "humidity": str(sense.get_humidity())}}))

The props.conf entry for the upper half example looks:

[indoor_tph]
INDEXED_EXTRACTIONS = JSON
TIME_PREFIX = "timestamp": "
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%6N
TZ = UTC

Hope this will help other splunkers with the same or similar problems.

Kind Regards
SierraX