topic Re: Long json got truncated even though TRUNCATE is set to 0 in Getting Data In

Long json got truncated even though TRUNCATE is set to 0

nminale — Wed, 13 Nov 2013 17:28:13 GMT

Hello,

I would like to add a log file containing json documents - one json per line. The json documents are pretty long (longer than 10,000 characters) and I don't want them to get truncated so I set the props.conf as follows.

NO_BINARY_CHECK = 1
SHOULD_LINEMERGE = false
TRUNCATE = 0
KV_MODE=json
LINE_BREAKER=([\n\r]+)({)

During data preview, everything looks good. Nothing got truncated at all. However, once I completed the process and searched for it, the data is truncated to 10,000 characters, so Splunk doesn't interpret it as json.

I saw the warning in splunkd.log below.

WARN  LineBreakingProcessor - Truncating line because limit of 10000 has been exceeded with a line length >= 10506 - data_source="/opt/readonly/log-archive/mylogfile.log", data_host="test", data_sourcetype="test"

Any help would be appreciated.

Thank you.

Re: Long json got truncated even though TRUNCATE is set to 0

mwk1000 — Wed, 11 Dec 2013 18:54:51 GMT

I have the same problem, I have good results being returned but the truncate statement does not seem to work. One note the { is a special character and should be escaped (\{) if you are actually looking for it.

I expected to get the whole JSON structure but it is still being chopped yet sourcetyped as the stanza i have defined.

--- I see the web for stripped the \ so ignore if it happened to you as well - Cheers

Re: Long json got truncated even though TRUNCATE is set to 0

aelliott — Wed, 11 Dec 2013 19:05:51 GMT

The search may be truncating it.
Perhaps maxvaluesize (limits.conf)
See http://docs.splunk.com/Documentation/Splunk/6.0/Admin/Limitsconf

Also wanted to note that I found a similar post that helped others here:
http://answers.splunk.com/answers/60064/json-event-truncate0