topic Re: Truncate in props.conf in Getting Data In

Truncate in props.conf

simon21 — Sun, 07 Jul 2019 19:53:14 GMT

what is the expected impact of increasing the value for TRUNCATE, the log reception upper limit setting value that can be defined in the indexer props.conf.
Also, is there any problem cases with TRUNCATE increased or 0 (unlimited).
　　

Further, in the customer environment, with index server directly receiving proxy log by syslog transfer, we plan to change the value to TRUNCATE = 64,000.
　
Is there a way to verify that a log (truncated log) that exceeds the TRUNCATE value has occurred?

Can I assume that the log of the truncated part cannot be recovered?

Re: Truncate in props.conf

richgalloway — Sun, 07 Jul 2019 20:46:08 GMT

If you are seeing events being truncated (regularly seeing "truncating" in splunkd.log) then increasing the value of TRUNCATE for that sourcetype should help eliminate the truncations.
Increasing the value for TRUNCATE increases indexer memory use. Likewise, for setting the value to zero.

It's not Best Practice for an indexer to receive syslog events directly. It's preferred to have a dedicated syslog server and use a forwarder to send the events to Splunk.

The new value for TRUNCATE should be a little higher than the longest message you've seen truncated in splunkd.log.

Truncated data is lost and cannot be recovered unless it is re-indexed. Of course, re-indexing probably is not possible when receiving syslog directly (which is another reason to not do it).

Re: Truncate in props.conf

woodcock — Mon, 08 Jul 2019 16:47:01 GMT

I would push VERY HARD for migrating away from syslog-direct-to-Indexer by adding a syslog-ng node and either doing either this:
http://www.georgestarcher.com/splunk-success-with-syslog/
Or this:
https://www.splunk.com/blog/2017/03/30/syslog-ng-and-hec-scalable-aggregated-data-collection-in-splunk.html

If you get your RegEx wrong and you have huge volume, then you can crash your Indexers using TRUNCATE=0 which is why we just keep adding 9s to the order of magintude that we expect (e.g. TRUNCATE = 999999).

To look for truncated events, look for the "leftover" pieces with a search like this:

index=<You Should Always Specify Your index> AND sourcetype=<And Your sourcetype Too>
| where _time == _indextime

Or you can do

... | regex punct!="^<Your Normal Punct beginning here>"