Getting Data In

How to define timestamp in props.conf for the JSON event

princemanto2580
Path Finder

Hi,

I have a sample json file where I have to index the time. I created the stanza at props.conf.

{"readIOps":0.0,"writeIOps":0.0,"otherIOps":0.0,"combinedIOps":0.0,"readThroughput":0.0,"writeThroughput":0.0,"combinedThroughput":0.0,"readResponseTime":0.0,"writeResponseTime":0.0,"combinedResponseTime":0.0,"volumeId":"0200000060080E500029094000009209525E8C08","volumeName":"VC-g0003v0011-H2","readPhysicalIOps":0.0,"writePhysicalIOps":0.0,"observedTime":"2015-07-25T22:07:34.000+0000"}
{"readIOps":0.0,"writeIOps":0.0,"otherIOps":0.0,"combinedIOps":0.0,"readThroughput":0.0,"writeThroughput":0.0,"combinedThroughput":0.0,"readResponseTime":0.0,"writeResponseTime":0.0,"combinedResponseTime":0.0,"volumeId":"0200000060080E500029094000009494525EB56B","volumeName":"dp0012v0044","readPhysicalIOps":0.0,"writePhysicalIOps":0.0,"observedTime":"2015-07-25T22:07:34.000+0000"}

[santa-banta]
INDEXED_EXTRACTIONS = json
TIME_PREFIX = "observedTime\":\"
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N+%4N
MAX_TIMESTAMP_LOOKAHEAD =
LINE_BREAKER =

SHOULD_LINEMERGE = false
TRUNCATE = 0

Questions:

  1. Is it compulsory to use LINE_BREAKER and MAX_TIMESTAMP_LOOKAHEAD at here, if INDEXED_EXTRACTION is defined as json ?
  2. If I do not wish to put INDEXED_EXTRACTIONS = json, what I have to put at LINE_BREAKER and MAX_TIMESTAMP_LOOKAHEAD ?

Highly appreciate if anyone guide on this.

Labels (2)
0 Karma
1 Solution

niketn
Legend

Following should work for you,

1) LINE_BREAKER is not required since you have SHOULD_LINEMERGE turned off, which means each line will be a new event.
2) MAX_TIMESTAMP_LOOKAHEAD will look for number of characters defined to identify timestamp(in your case it is 28 characters after the TIME_PREFIX.
3) The same props.conf for your sourcetype should work for you even without INDEXED_EXTRACTIONS=json, only difference would be that automatic field extractions will not occur for JSON fields.

PS: There are several other props.conf properties which can be set for Timestamp and event extractions. like BREAK_ONLY_BEFORE, BREAK_ONLY_AFTER etc. Refer to the following documentation: http://docs.splunk.com/Documentation/Splunk/latest/admin/Propsconf

INDEXED_EXTRACTIONS =json
SHOULD_LINEMERGE=false
NO_BINARY_CHECK=true
TIME_FORMAT=%Y-%m-%dT%H:%M:%S.%3N+%4N
TIME_PREFIX="observedTime":"
MAX_TIMESTAMP_LOOKAHEAD=28
____________________________________________
| makeresults | eval message= "Happy Splunking!!!"

View solution in original post

niketn
Legend

Following should work for you,

1) LINE_BREAKER is not required since you have SHOULD_LINEMERGE turned off, which means each line will be a new event.
2) MAX_TIMESTAMP_LOOKAHEAD will look for number of characters defined to identify timestamp(in your case it is 28 characters after the TIME_PREFIX.
3) The same props.conf for your sourcetype should work for you even without INDEXED_EXTRACTIONS=json, only difference would be that automatic field extractions will not occur for JSON fields.

PS: There are several other props.conf properties which can be set for Timestamp and event extractions. like BREAK_ONLY_BEFORE, BREAK_ONLY_AFTER etc. Refer to the following documentation: http://docs.splunk.com/Documentation/Splunk/latest/admin/Propsconf

INDEXED_EXTRACTIONS =json
SHOULD_LINEMERGE=false
NO_BINARY_CHECK=true
TIME_FORMAT=%Y-%m-%dT%H:%M:%S.%3N+%4N
TIME_PREFIX="observedTime":"
MAX_TIMESTAMP_LOOKAHEAD=28
____________________________________________
| makeresults | eval message= "Happy Splunking!!!"

abhinav_bel
Loves-to-Learn Lots

Hi ,

 

The TIME_FORMAT you provided doesn't work for me.

My timestamp in data looks like: 2020-07-02T18:00:18+02:00 with name log_modified_date.

i have written below props.conf:

[_json]
INDEXED_EXTRACTIONS = json
KV_MODE = none
AUTO_KV_JSON = false
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
TIMESTAMP_FIELDS = last_modified_date
TIME_FORMAT = %Y-%m-%dT%H:%M:%S+%2N:%2N
MAX_TIMESTAMP_LOOKAHEAD = 25

 

 

and getting time extracted as : 

7/2/20
6:00:18.020 PM

 

 

I want the time field extracted in same way as in data with + value as well like: 

 7/2/20
6:00:18+02:00 PM    something like this

Please let me know what i am doing wrong as i am not getting expected output.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...