Getting Data In

How to define timestamp in props.conf for the JSON event

Hi,

I have a sample JSON file where I have to index the time. I created the stanza at props.conf.

 

{"readIOps":0.0,"writeIOps":0.0,"otherIOps":0.0,"combinedIOps":0.0,"readThroughput":0.0,"writeThroughput":0.0,"combinedThroughput":0.0,"readResponseTime":0.0,"writeResponseTime":0.0,"combinedResponseTime":0.0,"volumeId":"0200000060080E500029094000009209525E8C08","volumeName":"VC-g0003v0011-H2","readPhysicalIOps":0.0,"writePhysicalIOps":0.0,"observedTime":"2015-07-25T22:07:34.000+0000"}
{"readIOps":0.0,"writeIOps":0.0,"otherIOps":0.0,"combinedIOps":0.0,"readThroughput":0.0,"writeThroughput":0.0,"combinedThroughput":0.0,"readResponseTime":0.0,"writeResponseTime":0.0,"combinedResponseTime":0.0,"volumeId":"0200000060080E500029094000009494525EB56B","volumeName":"dp0012v0044","readPhysicalIOps":0.0,"writePhysicalIOps":0.0,"observedTime":"2015-07-25T22:07:34.000+0000"}
[santa-banta]
INDEXED_EXTRACTIONS = json
TIMEPREFIX = "observedTime\":\"
TIMEFORMAT = %Y-%m-%dT%H:%M:%S.%3N+%4N
MAXTIMESTAMPLOOKAHEAD =
LINE_BREAKER =
SHOULD_LINEMERGE = false
TRUNCATE = 0

 

Questions:

  1. Is it compulsory to use LINEBREAKER and MAXTIMESTAMPLOOKAHEAD at here, if INDEXEDEXTRACTION is defined as json ?
  2. If I do not wish to put INDEXEDEXTRACTIONS = json, what I have to put at LINEBREAKER and MAXTIMESTAMPLOOKAHEAD ?

Highly appreciate if anyone guide on this.

Labels (2)
0 Karma
1 Solution

SplunkTrust
SplunkTrust

Following should work for you,

1) LINEBREAKER is not required since you have SHOULDLINEMERGE turned off, which means each line will be a new event.
2) MAXTIMESTAMPLOOKAHEAD will look for number of characters defined to identify timestamp(in your case it is 28 characters after the TIMEPREFIX.
3) The same props.conf for your sourcetype should work for you even without INDEXED
EXTRACTIONS=json, only difference would be that automatic field extractions will not occur for JSON fields.

PS: There are several other props.conf properties which can be set for Timestamp and event extractions. like BREAKONLYBEFORE, BREAKONLYAFTER etc. Refer to the following documentation: http://docs.splunk.com/Documentation/Splunk/latest/admin/Propsconf

INDEXED_EXTRACTIONS =json
SHOULD_LINEMERGE=false
NO_BINARY_CHECK=true
TIME_FORMAT=%Y-%m-%dT%H:%M:%S.%3N+%4N
TIME_PREFIX="observedTime":"
MAX_TIMESTAMP_LOOKAHEAD=28
____________________________________________
| makeresults | eval message= "Happy Splunking!!!"

View solution in original post

SplunkTrust
SplunkTrust

Following should work for you,

1) LINEBREAKER is not required since you have SHOULDLINEMERGE turned off, which means each line will be a new event.
2) MAXTIMESTAMPLOOKAHEAD will look for number of characters defined to identify timestamp(in your case it is 28 characters after the TIMEPREFIX.
3) The same props.conf for your sourcetype should work for you even without INDEXED
EXTRACTIONS=json, only difference would be that automatic field extractions will not occur for JSON fields.

PS: There are several other props.conf properties which can be set for Timestamp and event extractions. like BREAKONLYBEFORE, BREAKONLYAFTER etc. Refer to the following documentation: http://docs.splunk.com/Documentation/Splunk/latest/admin/Propsconf

INDEXED_EXTRACTIONS =json
SHOULD_LINEMERGE=false
NO_BINARY_CHECK=true
TIME_FORMAT=%Y-%m-%dT%H:%M:%S.%3N+%4N
TIME_PREFIX="observedTime":"
MAX_TIMESTAMP_LOOKAHEAD=28
____________________________________________
| makeresults | eval message= "Happy Splunking!!!"

View solution in original post

Loves-to-Learn Lots

Hi ,

 

The TIME_FORMAT you provided doesn't work for me.

My timestamp in data looks like: 2020-07-02T18:00:18+02:00 with name log_modified_date.

i have written below props.conf:

[_json]
INDEXED_EXTRACTIONS = json
KV_MODE = none
AUTO_KV_JSON = false
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
TIMESTAMP_FIELDS = last_modified_date
TIME_FORMAT = %Y-%m-%dT%H:%M:%S+%2N:%2N
MAX_TIMESTAMP_LOOKAHEAD = 25

 

 

and getting time extracted as : 

7/2/20
6:00:18.020 PM

 

 

I want the time field extracted in same way as in data with + value as well like: 

 7/2/20
6:00:18+02:00 PM    something like this

Please let me know what i am doing wrong as i am not getting expected output.

0 Karma