Getting Data In

How to get two lines of JSON to break as two events?

fitterman
Explorer

I am using a simple receiver to upload some lines of JSON. The input file has one JSON object (hash) per line, terminated with a newline. When I upload 12 JSON objects, they report as 10 events. In the cases where I get the 2-for-1 behavior, there is an Object (hash) that embeds another Object (hash) in the second of the two lines. For example, these two lines come back as one event:

{"sstime":1411843443,"value":"151236","event_id":"_view_","d1":"eng","d2":"primary","device_time":"2014-09-27 18:44:03","obj_type":"v","format":"2","rev":"1","version":"1.2.15","device_id":"86ec200468586be","appl_id":15204}
{"sstime":1411843443,"value":{"url":"local_data_53786/eng_welcome_primary.mp3","name":"welcome_primary"},"event_id":"_audiostart_","d1":"eng","d2":"primary","device_time":"2014-09-27 18:44:04","obj_type":"","audio_url":"local_data_53786/eng_welcome_primary.mp3","audio_name":"welcome_primary","format":"2","rev":"1","version":"1.2.15","device_id":"86ec200468586be","appl_id":15204}

As you can see, the second Object has a key with the name "value" that defines a subordinate Object. All the problem lines, are like this pair. Does anyone know how to get Splunk to recognize these are two events?

Tags (2)
1 Solution

aaronkorn
Splunk Employee
Splunk Employee

You will want to modify your prop.conf stanza, specifically the LINE_BREAKER option.

Give this a try:

[your_sourcetype]
SHOULD_LINEMERGE = false
LINE_BREAKER = \{\"sstime
TIME_PREFIX = sstime\"\:
MAX_TIMESTAMP_LOOKAHEAD = 10
TIME_FORMAT = %s

View solution in original post

aaronkorn
Splunk Employee
Splunk Employee

You will want to modify your prop.conf stanza, specifically the LINE_BREAKER option.

Give this a try:

[your_sourcetype]
SHOULD_LINEMERGE = false
LINE_BREAKER = \{\"sstime
TIME_PREFIX = sstime\"\:
MAX_TIMESTAMP_LOOKAHEAD = 10
TIME_FORMAT = %s

aaronkorn
Splunk Employee
Splunk Employee

If you are unsure if sstime wont be in the message the you could just set it to:

LINE_BREAKER = ^\{ 

Which will tell Splunk to break a new event if the beginning of the line begins with a "{".

Go ahead and add this into a new props.conf and let me know how it works for you.

0 Karma

rchotalia
New Member

If we have the LINE_BREAKER = ^\{ then what will happen to nested Json?

0 Karma

fitterman
Explorer

In the end, I added a new props.conf, but just turned off the default line merging, as for this source, I know for certain that the newline only occurs between each event.

[host::stats-ziploader.production]
SHOULD_LINEMERGE = false

Thanks for your help.

fitterman
Explorer

I'm truly a newb, so bear with me. We don't have such a file, but can add one. I guess this means we're getting a default file.

So my real problem with your suggestion is that because there's no assurance that the "sstime" element will be the first in the object definition, this can't be used. I want it to break every time it gets a complete hash. Is there a way to tell it to break after each complete object?

0 Karma

aaronkorn
Splunk Employee
Splunk Employee

You will want to restart your indexer after the change also this wont apply to events that have already been indexed.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...