Why is JSON parsing events inconsistently?

[LealP]

Hello,

I have an issue with the json data that is being ingested into Splunk using Universal Forwarder. Some times the json entries are ingested as individual entries in Splunk and other times the entire content is loaded as one single event. I tried to search for some special characters that might be causing this issue, but I wasn't able to find any.  Attached is a print screen with 2 examples, one that is being loaded as expected and the another where json is not correctly parsed.

Did someone already faced something similar? What should I do to fix it?

[LealP]

@richgalloway  here is the props.conf content:

CHARSET = UTF-8
LINE_BREAKER_LOOKBEHIND = 100
TRUNCATE = 10000
LB_CHUNK_BREAKER_TRUNCATE = 2000000
DATETIME_CONFIG = /etc/datetime.xml
ADD_EXTRA_TIME_FIELDS = True
ANNOTATE_PUNCT = True
HEADER_MODE =
MATCH_LIMIT = 100000
DEPTH_LIMIT = 1000
MAX_DAYS_HENCE=2
MAX_DAYS_AGO=2000
MAX_DIFF_SECS_AGO=3600
MAX_DIFF_SECS_HENCE=604800
MAX_TIMESTAMP_LOOKAHEAD = 128
DETERMINE_TIMESTAMP_DATE_WITH_SYSTEM_TIME = false
SHOULD_LINEMERGE = True
BREAK_ONLY_BEFORE =
BREAK_ONLY_BEFORE_DATE = True
MAX_EVENTS = 256
MUST_BREAK_AFTER =
MUST_NOT_BREAK_AFTER =
MUST_NOT_BREAK_BEFORE =
TRANSFORMS =
SEGMENTATION = indexing
SEGMENTATION-all = full
SEGMENTATION-inner = inner
SEGMENTATION-outer = outer
SEGMENTATION-raw = none
SEGMENTATION-standard = standard
LEARN_SOURCETYPE = true
LEARN_MODEL = true
termFrequencyWeightedDist = false
maxDist = 100
AUTO_KV_JSON = true
detect_trailing_nulls = false
sourcetype =
priority =

[richgalloway]

Assuming that is the correct set of props (I wonder because BREAK_ONLY_BEFORE_DATE is true and yet events are  not broken at dates) then these settings may help.  Put them in $SPLUNK_HOME/etc/system/local/props.conf and restart the indexers.

[<<mysourcetype>>]
TIME_PREFIX = timestamp" : "
TIME_FORMAT = %Y-%m-%d %H:%M:%S
LINE_BREAKER = ([\r\n]+)\{"severity
TRUNCATE = 10000
MAX_TIMESTAMP_LOOKAHEAD = 128
SHOULD_LINEMERGE = false
BREAK_ONLY_BEFORE_DATE = false
AUTO_KV_JSON = true

[LealP]

@richgalloway  do you know which one is used?

./splunkforwarder/etc/apps/search/default/props.conf
./splunkforwarder/etc/apps/SplunkUniversalForwarder/default/props.conf
./splunkforwarder/etc/apps/learned/local/props.conf
./splunkforwarder/etc/apps/splunk_internal_metrics/default/props.conf
./splunkforwarder/etc/system/default/props.conf

[richgalloway]

All of them.  Splunk merges the settings from all of the props.conf files to arrive at a final configuration.  You can use btool to get the same information.

splunk btool props list <<mysourcetype>>

Replace <<mysourcetype>> with the name of the sourcetype used by this input.

[richgalloway]

It looks like the LINE_BREAKER setting may need to be adjusted.  Please share the props.conf stanza for this input and we'll suggest some changes.