Getting Data In

Truncate in props.conf

simon21
Path Finder

what is the expected impact of increasing the value for TRUNCATE, the log reception upper limit setting value that can be defined in the indexer props.conf.
Also, is there any problem cases with TRUNCATE increased or 0 (unlimited).
  

Further, in the customer environment, with index server directly receiving proxy log by syslog transfer, we plan to change the value to TRUNCATE = 64,000.
 
Is there a way to verify that a log (truncated log) that exceeds the TRUNCATE value has occurred?

Can I assume that the log of the truncated part cannot be recovered?

0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

If you are seeing events being truncated (regularly seeing "truncating" in splunkd.log) then increasing the value of TRUNCATE for that sourcetype should help eliminate the truncations.
Increasing the value for TRUNCATE increases indexer memory use. Likewise, for setting the value to zero.

It's not Best Practice for an indexer to receive syslog events directly. It's preferred to have a dedicated syslog server and use a forwarder to send the events to Splunk.

The new value for TRUNCATE should be a little higher than the longest message you've seen truncated in splunkd.log.

Truncated data is lost and cannot be recovered unless it is re-indexed. Of course, re-indexing probably is not possible when receiving syslog directly (which is another reason to not do it).

---
If this reply helps you, Karma would be appreciated.

View solution in original post

woodcock
Esteemed Legend

I would push VERY HARD for migrating away from syslog-direct-to-Indexer by adding a syslog-ng node and either doing either this:
http://www.georgestarcher.com/splunk-success-with-syslog/
Or this:
https://www.splunk.com/blog/2017/03/30/syslog-ng-and-hec-scalable-aggregated-data-collection-in-splu...

If you get your RegEx wrong and you have huge volume, then you can crash your Indexers using TRUNCATE=0 which is why we just keep adding 9s to the order of magintude that we expect (e.g. TRUNCATE = 999999).

To look for truncated events, look for the "leftover" pieces with a search like this:

index=<You Should Always Specify Your index> AND sourcetype=<And Your sourcetype Too>
| where _time == _indextime

Or you can do

... | regex punct!="^<Your Normal Punct beginning here>"
0 Karma

richgalloway
SplunkTrust
SplunkTrust

If you are seeing events being truncated (regularly seeing "truncating" in splunkd.log) then increasing the value of TRUNCATE for that sourcetype should help eliminate the truncations.
Increasing the value for TRUNCATE increases indexer memory use. Likewise, for setting the value to zero.

It's not Best Practice for an indexer to receive syslog events directly. It's preferred to have a dedicated syslog server and use a forwarder to send the events to Splunk.

The new value for TRUNCATE should be a little higher than the longest message you've seen truncated in splunkd.log.

Truncated data is lost and cannot be recovered unless it is re-indexed. Of course, re-indexing probably is not possible when receiving syslog directly (which is another reason to not do it).

---
If this reply helps you, Karma would be appreciated.
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...