How to get two lines of JSON to break as two events?

[fitterman]

I am using a simple receiver to upload some lines of JSON. The input file has one JSON object (hash) per line, terminated with a newline. When I upload 12 JSON objects, they report as 10 events. In the cases where I get the 2-for-1 behavior, there is an Object (hash) that embeds another Object (hash) in the second of the two lines. For example, these two lines come back as one event:

{"sstime":1411843443,"value":"151236","event_id":"_view_","d1":"eng","d2":"primary","device_time":"2014-09-27 18:44:03","obj_type":"v","format":"2","rev":"1","version":"1.2.15","device_id":"86ec200468586be","appl_id":15204}
{"sstime":1411843443,"value":{"url":"local_data_53786/eng_welcome_primary.mp3","name":"welcome_primary"},"event_id":"_audiostart_","d1":"eng","d2":"primary","device_time":"2014-09-27 18:44:04","obj_type":"","audio_url":"local_data_53786/eng_welcome_primary.mp3","audio_name":"welcome_primary","format":"2","rev":"1","version":"1.2.15","device_id":"86ec200468586be","appl_id":15204}

As you can see, the second Object has a key with the name "value" that defines a subordinate Object. All the problem lines, are like this pair. Does anyone know how to get Splunk to recognize these are two events?

[aaronkorn · Accepted solution]

You will want to modify your prop.conf stanza, specifically the LINE_BREAKER option.

Give this a try:

[your_sourcetype]
SHOULD_LINEMERGE = false
LINE_BREAKER = \{\"sstime
TIME_PREFIX = sstime\"\:
MAX_TIMESTAMP_LOOKAHEAD = 10
TIME_FORMAT = %s

[aaronkorn]

If you are unsure if sstime wont be in the message the you could just set it to:

LINE_BREAKER = ^\{ 

Which will tell Splunk to break a new event if the beginning of the line begins with a "{".

Go ahead and add this into a new props.conf and let me know how it works for you.

[rchotalia]

If we have the LINE_BREAKER = ^\{ then what will happen to nested Json?

[fitterman]

In the end, I added a new props.conf, but just turned off the default line merging, as for this source, I know for certain that the newline only occurs between each event.

[host::stats-ziploader.production]
SHOULD_LINEMERGE = false

Thanks for your help.

[fitterman]

I'm truly a newb, so bear with me. We don't have such a file, but can add one. I guess this means we're getting a default file.

So my real problem with your suggestion is that because there's no assurance that the "sstime" element will be the first in the object definition, this can't be used. I want it to break every time it gets a complete hash. Is there a way to tell it to break after each complete object?

[aaronkorn]

You will want to restart your indexer after the change also this wont apply to events that have already been indexed.