How to define timestamp in props.conf for the JSON event

[princemanto2580]

Hi,

I have a sample json file where I have to index the time. I created the stanza at props.conf.

{"readIOps":0.0,"writeIOps":0.0,"otherIOps":0.0,"combinedIOps":0.0,"readThroughput":0.0,"writeThroughput":0.0,"combinedThroughput":0.0,"readResponseTime":0.0,"writeResponseTime":0.0,"combinedResponseTime":0.0,"volumeId":"0200000060080E500029094000009209525E8C08","volumeName":"VC-g0003v0011-H2","readPhysicalIOps":0.0,"writePhysicalIOps":0.0,"observedTime":"2015-07-25T22:07:34.000+0000"}
{"readIOps":0.0,"writeIOps":0.0,"otherIOps":0.0,"combinedIOps":0.0,"readThroughput":0.0,"writeThroughput":0.0,"combinedThroughput":0.0,"readResponseTime":0.0,"writeResponseTime":0.0,"combinedResponseTime":0.0,"volumeId":"0200000060080E500029094000009494525EB56B","volumeName":"dp0012v0044","readPhysicalIOps":0.0,"writePhysicalIOps":0.0,"observedTime":"2015-07-25T22:07:34.000+0000"}

[santa-banta]
INDEXED_EXTRACTIONS = json
TIME_PREFIX = "observedTime\":\"
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N+%4N
MAX_TIMESTAMP_LOOKAHEAD =
LINE_BREAKER =
SHOULD_LINEMERGE = false
TRUNCATE = 0

Questions:

1. Is it compulsory to use LINE_BREAKER and MAX_TIMESTAMP_LOOKAHEAD at here, if INDEXED_EXTRACTION is defined as json ?
2. If I do not wish to put INDEXED_EXTRACTIONS = json, what I have to put at LINE_BREAKER and MAX_TIMESTAMP_LOOKAHEAD ?

Highly appreciate if anyone guide on this.

[niketn · Accepted solution]

Following should work for you,

1) LINE_BREAKER is not required since you have SHOULD_LINEMERGE turned off, which means each line will be a new event.
2) MAX_TIMESTAMP_LOOKAHEAD will look for number of characters defined to identify timestamp(in your case it is 28 characters after the TIME_PREFIX.
3) The same props.conf for your sourcetype should work for you even without INDEXED_EXTRACTIONS=json, only difference would be that automatic field extractions will not occur for JSON fields.

PS: There are several other props.conf properties which can be set for Timestamp and event extractions. like BREAK_ONLY_BEFORE, BREAK_ONLY_AFTER etc. Refer to the following documentation: http://docs.splunk.com/Documentation/Splunk/latest/admin/Propsconf

INDEXED_EXTRACTIONS =json
SHOULD_LINEMERGE=false
NO_BINARY_CHECK=true
TIME_FORMAT=%Y-%m-%dT%H:%M:%S.%3N+%4N
TIME_PREFIX="observedTime":"
MAX_TIMESTAMP_LOOKAHEAD=28

[abhinav_bel]

Hi ,

The TIME_FORMAT you provided doesn't work for me.

My timestamp in data looks like: 2020-07-02T18:00:18+02:00 with name log_modified_date.

i have written below props.conf:

[_json]
INDEXED_EXTRACTIONS = json
KV_MODE = none
AUTO_KV_JSON = false
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
TIMESTAMP_FIELDS = last_modified_date
TIME_FORMAT = %Y-%m-%dT%H:%M:%S+%2N:%2N
MAX_TIMESTAMP_LOOKAHEAD = 25

and getting time extracted as : 

7/2/20
6:00:18.020 PM
 

I want the time field extracted in same way as in data with + value as well like: 

 7/2/20
6:00:18+02:00 PM    something like this

Please let me know what i am doing wrong as i am not getting expected output.