Splunk truncating large input json data and not indexing.

[swithinb]

Hi

I have used python script to call some api's and sending the response to splunk.

If the response is small, splunk taking the data and indexing it, but if the data is large, splunk is truncating the data at some point and displaying only the truncated data:

source_type= _Json

props.conf:

[sourcetype]
TRUNCATE = 1000000
KV_MODE = json
CHARSET = UTF-8
LINE_BREAKER = +[^}]+}(,[\r\n]+)
SHOULD_LINEMERGE = false

input.conf:

[default]
host = F1WQGT1
time_before_close = 5
multiline_event_extra_waittime = true

Please suggest me some better ideas.

[woodcock]

It is sourcetype, not source_type. It is inputs.conf, not input.conf. It is _json, not _Json. You should NEVER use _json as your sourcetype; make up your own appropriate to your data and add KV_MODE = json to your props.conf on your Search Head to get all the "fee" JSON field extractions.

[woodcock]

If you are sure that your settings are correct (and it looks like TRUNCATE is), it must be something else. If you are doing a sourcetype override/overwrite, you must use the ORIGINAL value, NOT the new value. You must deploy your settings to the first full instance(s) of Splunk that handle the events (usually either the HF tier if you use one, or else your Indexer tier) UNLESS you are using HEC's JSON endpoint (it gets pre-cooked) or INDEXED_EXTRACTIONS (configs go on the UF in that case), then restart all Splunk instances there. When (re)evaluating, you must send in new events (old events will stay broken), then test using _index_earliest=-5m to be absolutely certain that you are only examining the newly indexed events.