How to fix the Truncate warning when all the parameter is set correct?

Hemnaath — Tue, 02 Feb 2021 10:24:35 GMT

Hi All,

In our splunk health dashboard panel, we could see a list of source-types having truncate issues, when digging the _internal logs, we could see the below Warning message.

02-02-2021 18:23:11.436 +1100 WARN LineBreakingProcessor - Truncating line because limit of 10000 bytes has been exceeded with a line length >= 11639 - data_source="/var/icf/logs/xxx.xxx.0/xxx_0.log", data_host="xxxxx", data_sourcetype="xxx.wps.xxx"

Followed below steps to further analysis the issue

1) Checked the actual configuration in the HF instances where the parsing is taking place by executing the btool command.

bash-4.2$ ./splunk btool --app=appname props list --debug | grep TRUNCATE ( To find the path where app is configured and its TRUNCATE value)

2) ./splunk btool --app=appname props list --debug | grep sourcetype ( To find the Truncate value specific to app and sourcetype)

3) Validated the props.conf details by using the cat /opt/splunk/etc/apps/appname/local/props.conf found the below actual configuration.

[sourceytpename]
TRUNCATE = 800000
TIME_FORMAT =
TIME_PREFIX=\[
DATETIME_CONFIG=/etc/apps/appname/local/datetime.xml
SHOULD_LINEMERGE=false
LINE_BREAKER=([\r\n]+)\[(?:\d{1,2}\/\d{1,2}\/\d{2}\s\d{1,2}:\d{2}:\d{2}\:\d{3}\s|\d{4}-\d{2}-\d{2}T\d{1,2}:\d{2}:\d{2}\.\d{3}(?:Z|[+-]\d\d?:?(?:\d\d)?))
MAX_TIMESTAMP_LOOKAHEAD=30

4) Based on the actual truncate value as reference value, Identify the maximum length and frequency of occurrence for last 7 days

sourcetype="xx.xx.xx" | eval length=len(_raw) | stats max(length) as length by sourcetype

The maximum length was more then 512273 --> But well below the actual Truncate value=800000

Frequency of occurence was only one time it had reached more then 500000

sourcetype="xx.xx.xxt" | eval length=len(_raw) | where length>=500000 | stats count by _time length

Question:

1) When the actual Truncate value=800000 is more then the maximum Truncate value=512273,in this case we should not get any warning alert.
2) By increasing the Truncate value will not solve this issue as the actual truncate value is more then the maximum truncate value.

Kindly guide me if how to fix this issue.

Re: How to fix the Truncate warning when all the parameter is set correct?

scelikok — Tue, 02 Feb 2021 12:47:32 GMT

Hi @Hemnaath,

It seems you already checked all possible configuration points.

Asking this may sound silly but sometimes one can miss an obvious point. Is the host value that you saw LineBreakingProcessor warning in _internal log the same your HF instance is the same? Maybe truncated log is on another HF or indexer?

topic How to fix the Truncate warning when all the parameter is set correct? in Getting Data In

How to fix the Truncate warning when all the parameter is set correct?

Re: How to fix the Truncate warning when all the parameter is set correct?