Getting Data In

Splunk truncating large input json data and not indexing.

swithinb
Explorer

Hi

I have used python script to call some api's and sending the response to splunk.

If the response is small, splunk taking the data and indexing it, but if the data is large, splunk is truncating the data at some point and displaying only the truncated data:

source_type= _Json

props.conf:

[sourcetype]
TRUNCATE = 1000000
KV_MODE = json
CHARSET = UTF-8
LINE_BREAKER = +[^}]+}(,[\r\n]+)
SHOULD_LINEMERGE = false

input.conf:

[default]
host = F1WQGT1
time_before_close = 5
multiline_event_extra_waittime = true

Please suggest me some better ideas.

Tags (2)
0 Karma

woodcock
Esteemed Legend

It is sourcetype, not source_type. It is inputs.conf, not input.conf. It is _json, not _Json. You should NEVER use _json as your sourcetype; make up your own appropriate to your data and add KV_MODE = json to your props.conf on your Search Head to get all the "fee" JSON field extractions.

0 Karma

woodcock
Esteemed Legend

If you are sure that your settings are correct (and it looks like TRUNCATE is), it must be something else. If you are doing a sourcetype override/overwrite, you must use the ORIGINAL value, NOT the new value. You must deploy your settings to the first full instance(s) of Splunk that handle the events (usually either the HF tier if you use one, or else your Indexer tier) UNLESS you are using HEC's JSON endpoint (it gets pre-cooked) or INDEXED_EXTRACTIONS (configs go on the UF in that case), then restart all Splunk instances there. When (re)evaluating, you must send in new events (old events will stay broken), then test using _index_earliest=-5m to be absolutely certain that you are only examining the newly indexed events.

0 Karma
Get Updates on the Splunk Community!

Updated Data Type Articles, Anniversary Celebrations, and More on Splunk Lantern

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

A Prelude to .conf25: Your Guide to Splunk University

Heading to Boston this September for .conf25? Get a jumpstart by arriving a few days early for Splunk ...

4 Ways the Splunk Community Helps You Prepare for .conf25

.conf25 is right around the corner, and whether you’re a first-time attendee or a seasoned Splunker, the ...