Getting Data In
Highlighted

How to edit props.conf in order to have JSON log events listed in chronological order?

Path Finder

We have the following logs coming into Splunk:

{"log":"\u0009at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:113)\n","stream":"stdout","time":"2016-10-07T10:10:38.971217557Z"}
{"log":"\u0009at org.elasticsearch.index.store.Store$StoreDirectory.openInput(Store.java:683)\n","stream":"stdout","time":"2016-10-07T10:10:38.971213862Z"}
{"log":"\u0009at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:80)\n","stream":"stdout","time":"2016-10-07T10:10:38.971206809Z"}

Assume there are lots more log lines in that stack, how do we end up with the following in Splunk as one final log line?

…
at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:80)
at org.elasticsearch.index.store.Store$StoreDirectory.openInput(Store.java:683)
at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:113)
..

Note how the final log line has been put into chronological order.

Here is my Props.conf that I have set up:

[my_sourcetype]
#TIME_PREFIX = ^
#MAX_TIMESTAMP_LOOKAHEAD = 25
#TZ = GMT
#TIME_FORMAT = %Y-%m-%d %H:%M:%S,%3N
LINE_BREAKER = ([\r\n]+)
SHOULD_LINEMERGE = True
# 10000 is default, should be set on a case by case basis
TRUNCATE = 0
KV_MODE = JSON
# comment this line if they need to use PUNCT
#ANNOTATE_PUNCT = false
0 Karma
Highlighted

Re: How to edit props.conf in order to have JSON log events listed in chronological order?

Legend

I am not clear about what you want. In the first block, you show several lines of JSON logs. What is the criteria for defining an event? Do all the lines in the block belong in the same event? Do all the lines in the file belong together as one event? Do you ever want to select certain lines or perform statistical analysis on the data?

If all you want is for the data to appear in chronological order, remember that Splunk search always returns the newest events first. The underlying search actually works from newer to older data as it retrieves the data. The default is for Splunk to display the search results in this reverse chronological order. But the ordering is a function of the search, not the parsing. You can specify that you want the search results displayed in chronological order, by appending | reverse to the end of any search. Be aware that this may make the search take much longer, and consume much more memory and CPU.

0 Karma