topic Splunk truncating large input json data and not indexing. in Getting Data In

Splunk truncating large input json data and not indexing.

swithinb — Fri, 13 Dec 2019 06:16:55 GMT

I have used python script to call some api's and sending the response to splunk.

If the response is small, splunk taking the data and indexing it, but if the data is large, splunk is truncating the data at some point and displaying only the truncated data:

source_type= _Json

props.conf:

[sourcetype]
TRUNCATE = 1000000
KV_MODE = json
CHARSET = UTF-8
LINE_BREAKER = +[^}]+}(,[\r\n]+)
SHOULD_LINEMERGE = false

input.conf:

[default]
host = F1WQGT1
time_before_close = 5
multiline_event_extra_waittime = true

Please suggest me some better ideas.

Re: Splunk truncating large input json data and not indexing.

woodcock — Wed, 30 Sep 2020 03:22:27 GMT

If you are sure that your settings are correct (and it looks like TRUNCATE is), it must be something else. If you are doing a sourcetype override/overwrite, you must use the ORIGINAL value, NOT the new value. You must deploy your settings to the first full instance(s) of Splunk that handle the events (usually either the HF tier if you use one, or else your Indexer tier) UNLESS you are using HEC's JSON endpoint (it gets pre-cooked) or INDEXED_EXTRACTIONS (configs go on the UF in that case), then restart all Splunk instances there. When (re)evaluating, you must send in new events (old events will stay broken), then test using _index_earliest=-5m to be absolutely certain that you are only examining the newly indexed events.

Re: Splunk truncating large input json data and not indexing.

woodcock — Fri, 13 Dec 2019 10:59:48 GMT

It is sourcetype, not source_type. It is inputs.conf, not input.conf. It is _json, not _Json. You should NEVER use _json as your sourcetype; make up your own appropriate to your data and add KV_MODE = json to your props.conf on your Search Head to get all the "fee" JSON field extractions.