Getting Data In

Truncate in props.conf

simon21
Path Finder

what is the expected impact of increasing the value for TRUNCATE, the log reception upper limit setting value that can be defined in the indexer props.conf.
Also, is there any problem cases with TRUNCATE increased or 0 (unlimited).
  

Further, in the customer environment, with index server directly receiving proxy log by syslog transfer, we plan to change the value to TRUNCATE = 64,000.
 
Is there a way to verify that a log (truncated log) that exceeds the TRUNCATE value has occurred?

Can I assume that the log of the truncated part cannot be recovered?

0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

If you are seeing events being truncated (regularly seeing "truncating" in splunkd.log) then increasing the value of TRUNCATE for that sourcetype should help eliminate the truncations.
Increasing the value for TRUNCATE increases indexer memory use. Likewise, for setting the value to zero.

It's not Best Practice for an indexer to receive syslog events directly. It's preferred to have a dedicated syslog server and use a forwarder to send the events to Splunk.

The new value for TRUNCATE should be a little higher than the longest message you've seen truncated in splunkd.log.

Truncated data is lost and cannot be recovered unless it is re-indexed. Of course, re-indexing probably is not possible when receiving syslog directly (which is another reason to not do it).

---
If this reply helps you, Karma would be appreciated.

View solution in original post

woodcock
Esteemed Legend

I would push VERY HARD for migrating away from syslog-direct-to-Indexer by adding a syslog-ng node and either doing either this:
http://www.georgestarcher.com/splunk-success-with-syslog/
Or this:
https://www.splunk.com/blog/2017/03/30/syslog-ng-and-hec-scalable-aggregated-data-collection-in-splu...

If you get your RegEx wrong and you have huge volume, then you can crash your Indexers using TRUNCATE=0 which is why we just keep adding 9s to the order of magintude that we expect (e.g. TRUNCATE = 999999).

To look for truncated events, look for the "leftover" pieces with a search like this:

index=<You Should Always Specify Your index> AND sourcetype=<And Your sourcetype Too>
| where _time == _indextime

Or you can do

... | regex punct!="^<Your Normal Punct beginning here>"
0 Karma

richgalloway
SplunkTrust
SplunkTrust

If you are seeing events being truncated (regularly seeing "truncating" in splunkd.log) then increasing the value of TRUNCATE for that sourcetype should help eliminate the truncations.
Increasing the value for TRUNCATE increases indexer memory use. Likewise, for setting the value to zero.

It's not Best Practice for an indexer to receive syslog events directly. It's preferred to have a dedicated syslog server and use a forwarder to send the events to Splunk.

The new value for TRUNCATE should be a little higher than the longest message you've seen truncated in splunkd.log.

Truncated data is lost and cannot be recovered unless it is re-indexed. Of course, re-indexing probably is not possible when receiving syslog directly (which is another reason to not do it).

---
If this reply helps you, Karma would be appreciated.
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...