Getting Data In

Truncate in props.conf

simon21
Path Finder

what is the expected impact of increasing the value for TRUNCATE, the log reception upper limit setting value that can be defined in the indexer props.conf.
Also, is there any problem cases with TRUNCATE increased or 0 (unlimited).
  

Further, in the customer environment, with index server directly receiving proxy log by syslog transfer, we plan to change the value to TRUNCATE = 64,000.
 
Is there a way to verify that a log (truncated log) that exceeds the TRUNCATE value has occurred?

Can I assume that the log of the truncated part cannot be recovered?

0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

If you are seeing events being truncated (regularly seeing "truncating" in splunkd.log) then increasing the value of TRUNCATE for that sourcetype should help eliminate the truncations.
Increasing the value for TRUNCATE increases indexer memory use. Likewise, for setting the value to zero.

It's not Best Practice for an indexer to receive syslog events directly. It's preferred to have a dedicated syslog server and use a forwarder to send the events to Splunk.

The new value for TRUNCATE should be a little higher than the longest message you've seen truncated in splunkd.log.

Truncated data is lost and cannot be recovered unless it is re-indexed. Of course, re-indexing probably is not possible when receiving syslog directly (which is another reason to not do it).

---
If this reply helps you, Karma would be appreciated.

View solution in original post

woodcock
Esteemed Legend

I would push VERY HARD for migrating away from syslog-direct-to-Indexer by adding a syslog-ng node and either doing either this:
http://www.georgestarcher.com/splunk-success-with-syslog/
Or this:
https://www.splunk.com/blog/2017/03/30/syslog-ng-and-hec-scalable-aggregated-data-collection-in-splu...

If you get your RegEx wrong and you have huge volume, then you can crash your Indexers using TRUNCATE=0 which is why we just keep adding 9s to the order of magintude that we expect (e.g. TRUNCATE = 999999).

To look for truncated events, look for the "leftover" pieces with a search like this:

index=<You Should Always Specify Your index> AND sourcetype=<And Your sourcetype Too>
| where _time == _indextime

Or you can do

... | regex punct!="^<Your Normal Punct beginning here>"
0 Karma

richgalloway
SplunkTrust
SplunkTrust

If you are seeing events being truncated (regularly seeing "truncating" in splunkd.log) then increasing the value of TRUNCATE for that sourcetype should help eliminate the truncations.
Increasing the value for TRUNCATE increases indexer memory use. Likewise, for setting the value to zero.

It's not Best Practice for an indexer to receive syslog events directly. It's preferred to have a dedicated syslog server and use a forwarder to send the events to Splunk.

The new value for TRUNCATE should be a little higher than the longest message you've seen truncated in splunkd.log.

Truncated data is lost and cannot be recovered unless it is re-indexed. Of course, re-indexing probably is not possible when receiving syslog directly (which is another reason to not do it).

---
If this reply helps you, Karma would be appreciated.
Get Updates on the Splunk Community!

Happy CX Day to our Community Superheroes!

Happy 10th Birthday CX Day!What is CX Day? It’s a global celebration recognizing innovation and success in the ...

Check out This Month’s Brand new Splunk Lantern Articles

Splunk Lantern is a customer success center providing advice from Splunk experts on valuable data insights, ...

Routing Data to Different Splunk Indexes in the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...