Getting Data In

Timestamp Parsing in JSON

RichaSingh
Path Finder

Hi,

Can anyone help me with best configurations for timestamp parsing (where "DateTime" is the actual time) for following JSON format :

{ [-] 
    AccessKey:  ----------------------
    Account:  some-value
    AccountId:  somevalue
    Action:  GetInstances 
    Class:  ----------------------------------------------------------
    **DateTime:  2007-10-27T00:51:57.91Z** 
    ElapsedTime:  00:00:00 
    File:  null 
    Level:  Trace 
    Line:  null 
    LogId:  ID-347t58734534-6565ghmhmm
    Message:  Vapor URL - https://www.randomURL.com,ImageId)
Header - {
  "Signature": "$^%$#@)(#4t4r65gvjhloojk",
  "AccessKey": "123854y3957349534-0fdjvbkjds90234u02394-23940=dsffsvbkdjadsvfnk0923480",
  "RequestType": 0,
  "SkipOwnerLookup": false,
  "UserAgent": "456934",
  "UserAgentVersion": null,
  "Timestamp": "2007-10-27T00:51:57Z",
  "User": "3254832564873`12",
  "TestMode": false,
  "ScheduledAction": false
} 

What I have already tried in props.conf is:

[API]
TIMESTAMP_FIELDS=DateTime 
INDEXED_EXTRACTIONS=json
NO_BINARY_CHECK=true
KV_MODE=none 

And also:

[API]
TIME_PREFIX = "\"DateTime\":\"
INDEXED_EXTRACTIONS=json

Also, I have been through the link:

https://answers.splunk.com/answers/104500/transforming-timestamps.html

None of the two seem to be working. Any comments/ suggestions will be highly appreciated.

1 Solution

SierraX
Communicator

Hi,
got a similar problem at an customer and worked yesterday on a solution based on your Question and the first answer, on a testsetup.

Rebuild Timestamp

As raw

The Splunk Server has a different system time than the sender (see in the lower half)

The JSON is send by following python line:

response = urllib2.urlopen(req, json.dumps({"sourcetype":"indoor_tph", "source":"", "host":"", "event": {"temperature": str(sense.get_temperature()-7.0), "timestamp": str(datetime.utcnow().strftime('%Y-%m-%dT%H:%M:%S.%f')), "pressure": str(sense.get_pressure()), "humidity": str(sense.get_humidity())}}))

The props.conf entry for the upper half example looks:

[indoor_tph]
INDEXED_EXTRACTIONS = JSON
TIME_PREFIX = "timestamp": "
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%6N
TZ = UTC

Hope this will help other splunkers with the same or similar problems.

Kind Regards
SierraX

View solution in original post

0 Karma

SierraX
Communicator

Hi,
got a similar problem at an customer and worked yesterday on a solution based on your Question and the first answer, on a testsetup.

Rebuild Timestamp

As raw

The Splunk Server has a different system time than the sender (see in the lower half)

The JSON is send by following python line:

response = urllib2.urlopen(req, json.dumps({"sourcetype":"indoor_tph", "source":"", "host":"", "event": {"temperature": str(sense.get_temperature()-7.0), "timestamp": str(datetime.utcnow().strftime('%Y-%m-%dT%H:%M:%S.%f')), "pressure": str(sense.get_pressure()), "humidity": str(sense.get_humidity())}}))

The props.conf entry for the upper half example looks:

[indoor_tph]
INDEXED_EXTRACTIONS = JSON
TIME_PREFIX = "timestamp": "
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%6N
TZ = UTC

Hope this will help other splunkers with the same or similar problems.

Kind Regards
SierraX

0 Karma

acharlieh
Influencer

If that's an accurate copy/paste of the data you're attempting to send in, that is not actually JSON formatted text. It's close, but not quite. This could definitely throw a wrench into using INDEXED_EXTRACTIONS. If it was actually JSON text there would be a lot more double quotes at least.

If you're using INDEXED_EXTRACTIONS=json with your sourcetype, the props.conf stanza specifying INDEXED_EXTRACTIONS and all parsing options should live on the originating Splunk instance instead of the usual parsing Splunk instance. (In most environments, this means this configuration is on your universal forwarder instead of your indexer). I would try TIMESTAMP_FIELDS on the UF (assuming you can actually use INDEXED_EXTRACTIONS.)

If you cannot use INDEXED_EXTRACTIONS, don't surround your setting of TIME_PREFIX in quotes. You should also consider also setting TIME_FORMAT and MAX_TIMESTAMP_LOOKAHEAD if the timestamp is consistent.

0 Karma
Get Updates on the Splunk Community!

Database Performance Sidebar Panel Now on APM Database Query Performance & Service ...

We’ve streamlined the troubleshooting experience for database-related service issues by adding a database ...

IM Landing Page Filter - Now Available

We’ve added the capability for you to filter across the summary details on the main Infrastructure Monitoring ...

Dynamic Links from Alerts to IM Navigators - New in Observability Cloud

Splunk continues to improve the troubleshooting experience in Observability Cloud with this latest enhancement ...