Getting Data In
Highlighted

How to prevent splunk from merging few JSON strings into single event?

Explorer

Example raw data:

{"field1": "value1", "field2": "value2", ..., "string": "1" } 
{"field1": "value1", "field2": "value2", ... ,"string":"2"} 
{"field1": "value1", "field2": "value2", ..., "string":"3" }
{"field1": "value1", "field2": "value2", ..., "string":"4" }

Splunk merge few of raw data string into single event, as result you got 2 events.
Event 1:

{"field1": "value1", "field2": "value2", ..., "string": "1" } 
{"field1": "value1", "field2": "value2", ... ,"string":"2"} 
{"field1": "value1", "field2": "value2", ..., "string":"3" }

Event 2:

{ [-]
    field1: value1
    field2: value2
     ...
    string: 4
}

So, 80% of events looks like event 1 in example. But some events caught by single row and parsed as JSON type.

Using cluster of Splunk Enterprise and splunkforwarder for data delivery, version 6.5.5.

I have tried to setup props.conf on splunkforwarder (app which work with JSON log files), tried to use different LINE_BREAKER:

1. (\})
2. \}
3. "(^)\{" 

Current props.conf:

[json-logs]
SHOULD_LINEMERGE = false
KV_MODE = json
LINE_BREAKER = (\})
TIME_PREFIX = \"time\": \"

Have same problem not only with JSON format logs, looks like props.conf line breaking options not work at all.
What I am doing wrong?

0 Karma
Highlighted

Re: How to prevent splunk from merging few JSON strings into single event?

Splunk Employee
Splunk Employee

props.conf settings have to be present where the event parsing occurs. The forwarder (if it is a universal forwarder) does NOT parse events. If you have a UF sending your data to your indexer(s), put your props.conf for the sourcetype on the indexer(s).

While a bit dated, this Wiki article still contains accurate information on what settings apply to which phase of the event processing pipeline.

You can also try to use BREAK_ONLY_BEFORE=^\{ instead of LINE_BREAKER if you are certain that all your events start with a '{' at the beginning of a line.

View solution in original post

Highlighted

Re: How to prevent splunk from merging few JSON strings into single event?

Explorer

Thank you for answer. I was confused with this in wiki article, which you mentioned:

Since splunk 6, some source can be parsed for structured data (like headers, or json) and be populated at the forwarder level. see http://docs.splunk.com/Documentation/Splunk/6.1.2/Data/Extractfieldsfromfileheadersatindextime#Forwa... Those setting have to be on the forwarders (and indexers if they monitor files)

Looks like it's not work at all.

Also i have question, what is the right way to distribute props.conf to indexers on cluster of Splunk Enterprise? Using master and master-apps or creating new custom application?

0 Karma
Highlighted

Re: How to prevent splunk from merging few JSON strings into single event?

Engager

Hi spellanser,

I had the same problem with that merged JSON logs.
I've tried various props.conf setting that were not working until I change the TIMEPREFIX, TIMEFORMAT and MAXTIMESTAMPLOOKAHEAD settings.
My guess is when Splunk try to parsing logs and couldn't find event timestamp so then the logs were not splitted.

I can see from Splunk document:
TIMEPREFIX:
* If the TIME
PREFIX cannot be found in the event text, timestamp extraction
will not occur.
* Defaults to empty.

Here is my JSON logs look like:

{"asctime": "2018-06-22T09:13Z+0000", "exception": "xxxx", "function_name": "xxxx"}
{"asctime": "2018-06-22T09:15Z+0000", "exc_duration": 100, "exc_memory": "70 MB"}

Props.conf:

[my_sourcetype]
INDEXED_EXTRACTIONS = json
KV_MODE=none
SHOULD_LINEMERGE=true
NO_BINARY_CHECK=true
BREAK_ONLY_BEFORE=([\r\n]+)
TIME_PREFIX=asctime:\s
MAX_TIMESTAMP_LOOKAHEAD=25
TIME_FORMAT=%Y-%m-%dT%H:%M:%SZ%z

Hope this may help in your situation.
Cheers

0 Karma
Highlighted

Re: How to prevent splunk from merging few JSON strings into single event?

Explorer

Thank you, will try it!

0 Karma
Highlighted

Re: How to prevent splunk from merging few JSON strings into single event?

SplunkTrust
SplunkTrust

Hello Spellanser: Did you have any luck?

0 Karma