topic Re: How to get two lines of JSON to break as two events? in Getting Data In

How to get two lines of JSON to break as two events?

fitterman — Mon, 29 Sep 2014 20:30:17 GMT

I am using a simple receiver to upload some lines of JSON. The input file has one JSON object (hash) per line, terminated with a newline. When I upload 12 JSON objects, they report as 10 events. In the cases where I get the 2-for-1 behavior, there is an Object (hash) that embeds another Object (hash) in the second of the two lines. For example, these two lines come back as one event:

{"sstime":1411843443,"value":"151236","event_id":"_view_","d1":"eng","d2":"primary","device_time":"2014-09-27 18:44:03","obj_type":"v","format":"2","rev":"1","version":"1.2.15","device_id":"86ec200468586be","appl_id":15204}
{"sstime":1411843443,"value":{"url":"local_data_53786/eng_welcome_primary.mp3","name":"welcome_primary"},"event_id":"_audiostart_","d1":"eng","d2":"primary","device_time":"2014-09-27 18:44:04","obj_type":"","audio_url":"local_data_53786/eng_welcome_primary.mp3","audio_name":"welcome_primary","format":"2","rev":"1","version":"1.2.15","device_id":"86ec200468586be","appl_id":15204}

As you can see, the second Object has a key with the name "value" that defines a subordinate Object. All the problem lines, are like this pair. Does anyone know how to get Splunk to recognize these are two events?

Re: How to get two lines of JSON to break as two events?

aaronkorn — Mon, 29 Sep 2014 21:16:34 GMT

You will want to modify your prop.conf stanza, specifically the LINE_BREAKER option.

Give this a try:

[your_sourcetype]
SHOULD_LINEMERGE = false
LINE_BREAKER = \{\"sstime
TIME_PREFIX = sstime\"\:
MAX_TIMESTAMP_LOOKAHEAD = 10
TIME_FORMAT = %s

Re: How to get two lines of JSON to break as two events?

aaronkorn — Mon, 29 Sep 2014 21:19:44 GMT

You will want to restart your indexer after the change also this wont apply to events that have already been indexed.

Re: How to get two lines of JSON to break as two events?

fitterman — Tue, 30 Sep 2014 14:54:24 GMT

I'm truly a newb, so bear with me. We don't have such a file, but can add one. I guess this means we're getting a default file.

So my real problem with your suggestion is that because there's no assurance that the "sstime" element will be the first in the object definition, this can't be used. I want it to break every time it gets a complete hash. Is there a way to tell it to break after each complete object?

Re: How to get two lines of JSON to break as two events?

aaronkorn — Tue, 30 Sep 2014 15:12:18 GMT

If you are unsure if sstime wont be in the message the you could just set it to:

LINE_BREAKER = ^\{

Which will tell Splunk to break a new event if the beginning of the line begins with a "{".

Go ahead and add this into a new props.conf and let me know how it works for you.

Re: How to get two lines of JSON to break as two events?

fitterman — Tue, 30 Sep 2014 22:22:26 GMT

In the end, I added a new props.conf, but just turned off the default line merging, as for this source, I know for certain that the newline only occurs between each event.

[host::stats-ziploader.production]
SHOULD_LINEMERGE = false

Thanks for your help.

Re: How to get two lines of JSON to break as two events?

rchotalia — Wed, 15 Nov 2017 02:26:36 GMT

If we have the LINE_BREAKER = ^\{ then what will happen to nested Json?