Getting Data In

Splunk truncating large input json data and not indexing.

swithinb
Explorer

Hi

I have used python script to call some api's and sending the response to splunk.

If the response is small, splunk taking the data and indexing it, but if the data is large, splunk is truncating the data at some point and displaying only the truncated data:

source_type= _Json

props.conf:

[sourcetype]
TRUNCATE = 1000000
KV_MODE = json
CHARSET = UTF-8
LINE_BREAKER = +[^}]+}(,[\r\n]+)
SHOULD_LINEMERGE = false

input.conf:

[default]
host = F1WQGT1
time_before_close = 5
multiline_event_extra_waittime = true

Please suggest me some better ideas.

Tags (2)
0 Karma

woodcock
Esteemed Legend

It is sourcetype, not source_type. It is inputs.conf, not input.conf. It is _json, not _Json. You should NEVER use _json as your sourcetype; make up your own appropriate to your data and add KV_MODE = json to your props.conf on your Search Head to get all the "fee" JSON field extractions.

0 Karma

woodcock
Esteemed Legend

If you are sure that your settings are correct (and it looks like TRUNCATE is), it must be something else. If you are doing a sourcetype override/overwrite, you must use the ORIGINAL value, NOT the new value. You must deploy your settings to the first full instance(s) of Splunk that handle the events (usually either the HF tier if you use one, or else your Indexer tier) UNLESS you are using HEC's JSON endpoint (it gets pre-cooked) or INDEXED_EXTRACTIONS (configs go on the UF in that case), then restart all Splunk instances there. When (re)evaluating, you must send in new events (old events will stay broken), then test using _index_earliest=-5m to be absolutely certain that you are only examining the newly indexed events.

0 Karma
Get Updates on the Splunk Community!

Splunk AppDynamics Agents Webinar Series

Mark your calendars! On June 24th at 12PM PST, we’re going live with the second session of our Splunk ...

SplunkTrust Application Period is Officially OPEN!

It's that time, folks! The application/nomination period for the 2025 SplunkTrust is officially open! If you ...

Splunk Answers Content Calendar, June Edition II

Get ready to dive into Splunk Dashboard panels this week! We'll be tackling common questions around ...