Getting Data In

Truncate in props.conf

simon21
Path Finder

what is the expected impact of increasing the value for TRUNCATE, the log reception upper limit setting value that can be defined in the indexer props.conf.
Also, is there any problem cases with TRUNCATE increased or 0 (unlimited).
  

Further, in the customer environment, with index server directly receiving proxy log by syslog transfer, we plan to change the value to TRUNCATE = 64,000.
 
Is there a way to verify that a log (truncated log) that exceeds the TRUNCATE value has occurred?

Can I assume that the log of the truncated part cannot be recovered?

0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

If you are seeing events being truncated (regularly seeing "truncating" in splunkd.log) then increasing the value of TRUNCATE for that sourcetype should help eliminate the truncations.
Increasing the value for TRUNCATE increases indexer memory use. Likewise, for setting the value to zero.

It's not Best Practice for an indexer to receive syslog events directly. It's preferred to have a dedicated syslog server and use a forwarder to send the events to Splunk.

The new value for TRUNCATE should be a little higher than the longest message you've seen truncated in splunkd.log.

Truncated data is lost and cannot be recovered unless it is re-indexed. Of course, re-indexing probably is not possible when receiving syslog directly (which is another reason to not do it).

---
If this reply helps you, Karma would be appreciated.

View solution in original post

woodcock
Esteemed Legend

I would push VERY HARD for migrating away from syslog-direct-to-Indexer by adding a syslog-ng node and either doing either this:
http://www.georgestarcher.com/splunk-success-with-syslog/
Or this:
https://www.splunk.com/blog/2017/03/30/syslog-ng-and-hec-scalable-aggregated-data-collection-in-splu...

If you get your RegEx wrong and you have huge volume, then you can crash your Indexers using TRUNCATE=0 which is why we just keep adding 9s to the order of magintude that we expect (e.g. TRUNCATE = 999999).

To look for truncated events, look for the "leftover" pieces with a search like this:

index=<You Should Always Specify Your index> AND sourcetype=<And Your sourcetype Too>
| where _time == _indextime

Or you can do

... | regex punct!="^<Your Normal Punct beginning here>"
0 Karma

richgalloway
SplunkTrust
SplunkTrust

If you are seeing events being truncated (regularly seeing "truncating" in splunkd.log) then increasing the value of TRUNCATE for that sourcetype should help eliminate the truncations.
Increasing the value for TRUNCATE increases indexer memory use. Likewise, for setting the value to zero.

It's not Best Practice for an indexer to receive syslog events directly. It's preferred to have a dedicated syslog server and use a forwarder to send the events to Splunk.

The new value for TRUNCATE should be a little higher than the longest message you've seen truncated in splunkd.log.

Truncated data is lost and cannot be recovered unless it is re-indexed. Of course, re-indexing probably is not possible when receiving syslog directly (which is another reason to not do it).

---
If this reply helps you, Karma would be appreciated.
Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...