topic Re: Lines break when indexing JSON data using props.conf attributes in Getting Data In

Lines break when indexing JSON data using props.conf attributes

anantdeshpande — Tue, 22 Aug 2017 06:24:45 GMT

Hi team,

I am not able to index below JSON data in Splunk 6.2 with below props.conf attributes. Its breaking at every line and treating as separate event with no field extraction. When I add the same file from Search head using add data option and selects _json as source type, the fields are correctly extracted. But do not work when mention same attributes and customized sourcetype name in props. Please suggest.

Data:
{
  "messageId" : "VIPJAPAN40001JCOMPLETE2017220818015450",
  "messageType" : "EVENT",
  "sendingAppId" : "P1",
  "sendTimeStamp" : "2017-22-08T17:09:27.526-05:00",
}

Props:
[APP1_INOUT_App2]
INDEXED_EXTRACTIONS = json
KV_MODE = none
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
category = Structured
disabled = false
DATETIME_CONFIG = CURRENT

Re: Lines break when indexing JSON data using props.conf attributes

niketn — Tue, 29 Sep 2020 15:24:45 GMT

@anantdeshpande, Can you try with the following?

SHOULD_LINEMERGE = true
Also if you know event break pattern like start or end you should LINE_BREAKER, BREAK_ONLY_BEFORE, MUST_BREAK_AFTER etc. to identify events correctly. Refer to documentation: http://docs.splunk.com/Documentation/Splunk/latest/Data/Configureeventlinebreaking

Where is the field for timestamp in your data? Can you please one complete event as sample (or few)?

Re: Lines break when indexing JSON data using props.conf attributes

anantdeshpande — Tue, 29 Sep 2020 15:28:46 GMT

I tried with both SHOULD_LINEMERGE = true and false, but same results.

I can write BREAK_ONLY_BEFORE = ^{ or (^){ - but JSON is recognized format by Splunk and I should not be writing other stuff like normal logs.

In my data there are multiple fields with time stamp.

Re: Lines break when indexing JSON data using props.conf attributes

vasanthmss — Tue, 22 Aug 2017 21:37:04 GMT

Try the below config,

[APP1_INOUT_App2]
DATETIME_CONFIG=CURRENT
SHOULD_LINEMERGE=true
NO_BINARY_CHECK=true
CHARSET=UTF-8
INDEXED_EXTRACTIONS=json
KV_MODE=none
pulldown_type=true

Re: Lines break when indexing JSON data using props.conf attributes

rafiqrehman — Wed, 30 Sep 2020 01:53:47 GMT

This works perfect for json data, at least for me:

[sourcetype]
INDEXED_EXTRACTIONS=json
JSON_TRIM_BRACES_IN_ARRAY_NAMES=true
CHARSET = AUTO
MAX_DIFF_SECS_AGO = 604800
MAX_EVENTS = 10000
NO_BINARY_CHECK = 1
TRUNCATE = 0