topic Re: Does Splunk have a good place to see Truncated Data from an input? in Getting Data In

Does Splunk have a good place to see Truncated Data from an input?

jaredlaney — Tue, 29 Sep 2020 07:26:30 GMT

I have an input that is being JSON. There are some files that get truncated and others that don't.

My config in props looks like this:

[test_json]
KV_MODE = json
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE = false
TRUNCATE = 999999

The file is pretty large and each json doc in the file is on one line. When I do a line count, sometimes the numbers match Splunk and sometimes they don't. Should Splunk put up a bulletin message when it truncates data??

Re: Does Splunk have a good place to see Truncated Data from an input?

maciep — Fri, 02 Oct 2015 19:28:06 GMT

I don't think Splunk tells you when data is truncated or whether any of the normal "health" apps like SoS, DMC, Fire Brigade, etc have panels with that info.

We use a simple search below in one of our own dashboards to help identify which sourcetypes are being truncated, so we can address appropriately in props.conf. The splunkd logs know how long the event was and the limit that it exceeded which caused it to be truncated.

Maybe you could find it helpful....

index=_internal sourcetype=splunkd truncating | rex "limit of (?<limit>\S+).+\>=\s+(?<length>\S+)" | stats max(length) as length max(limit) as limit count by data_source data_sourcetype | sort - count

Re: Does Splunk have a good place to see Truncated Data from an input?

GauravSplunxter — Thu, 29 Jun 2017 15:11:04 GMT

This search gave me exactly what i needed. Thanks.