How to prevent splunk from merging few JSON strings into single event?

[spellanser]

Example raw data:

{"field1": "value1", "field2": "value2", ..., "string": "1" } 
{"field1": "value1", "field2": "value2", ... ,"string":"2"} 
{"field1": "value1", "field2": "value2", ..., "string":"3" }
{"field1": "value1", "field2": "value2", ..., "string":"4" }

Splunk merge few of raw data string into single event, as result you got 2 events.
Event 1:

{"field1": "value1", "field2": "value2", ..., "string": "1" } 
{"field1": "value1", "field2": "value2", ... ,"string":"2"} 
{"field1": "value1", "field2": "value2", ..., "string":"3" }

Event 2:

{ [-]
    field1: value1
    field2: value2
     ...
    string: 4
}

So, 80% of events looks like event 1 in example. But some events caught by single row and parsed as JSON type.

Using cluster of Splunk Enterprise and splunkforwarder for data delivery, version 6.5.5.

I have tried to setup props.conf on splunkforwarder (app which work with JSON log files), tried to use different LINE_BREAKER:

1. (\})
2. \}
3. "(^)\{" 

Current props.conf:

[json-logs]
SHOULD_LINEMERGE = false
KV_MODE = json
LINE_BREAKER = (\})
TIME_PREFIX = \"time\": \"

Have same problem not only with JSON format logs, looks like props.conf line breaking options not work at all.
What I am doing wrong?

[syunwei]

Hi spellanser,

I had the same problem with that merged JSON logs.
I've tried various props.conf setting that were not working until I change the TIME_PREFIX, TIME_FORMAT and MAX_TIMESTAMP_LOOKAHEAD settings.
My guess is when Splunk try to parsing logs and couldn't find event timestamp so then the logs were not splitted.

I can see from Splunk document:
TIME_PREFIX:
* If the TIME_PREFIX cannot be found in the event text, timestamp extraction
will not occur.
* Defaults to empty.

Here is my JSON logs look like:

{"asctime": "2018-06-22T09:13Z+0000", "exception": "xxxx", "function_name": "xxxx"}
{"asctime": "2018-06-22T09:15Z+0000", "exc_duration": 100, "exc_memory": "70 MB"}

Props.conf:

[my_sourcetype]
INDEXED_EXTRACTIONS = json
KV_MODE=none
SHOULD_LINEMERGE=true
NO_BINARY_CHECK=true
BREAK_ONLY_BEFORE=([\r\n]+)
TIME_PREFIX=asctime:\s
MAX_TIMESTAMP_LOOKAHEAD=25
TIME_FORMAT=%Y-%m-%dT%H:%M:%SZ%z

Hope this may help in your situation.
Cheers

[spellanser]

Thank you, will try it!

[burwell]

Hello Spellanser: Did you have any luck?

[s2_splunk · Accepted solution]

props.conf settings have to be present where the event parsing occurs. The forwarder (if it is a universal forwarder) does NOT parse events. If you have a UF sending your data to your indexer(s), put your props.conf for the sourcetype on the indexer(s).

While a bit dated, this Wiki article still contains accurate information on what settings apply to which phase of the event processing pipeline.

You can also try to use BREAK_ONLY_BEFORE=^\{ instead of LINE_BREAKER if you are certain that all your events start with a '{' at the beginning of a line.

[spellanser]

Thank you for answer. I was confused with this in wiki article, which you mentioned:

Since splunk 6, some source can be parsed for structured data (like headers, or json) and be populated at the forwarder level. see http://docs.splunk.com/Documentation/Splunk/6.1.2/Data/Extractfieldsfromfileheadersatindextime#Forwa... Those setting have to be on the forwarders (and indexers if they monitor files)

Looks like it's not work at all.

Also i have question, what is the right way to distribute props.conf to indexers on cluster of Splunk Enterprise? Using master and master-apps or creating new custom application?