Hi All,
In our splunk health dashboard panel, we could see a list of source-types having truncate issues, when digging the _internal logs, we could see the below Warning message.
02-02-2021 18:23:11.436 +1100 WARN LineBreakingProcessor - Truncating line because limit of 10000 bytes has been exceeded with a line length >= 11639 - data_source="/var/icf/logs/xxx.xxx.0/xxx_0.log", data_host="xxxxx", data_sourcetype="xxx.wps.xxx"
Followed below steps to further analysis the issue
1) Checked the actual configuration in the HF instances where the parsing is taking place by executing the btool command.
bash-4.2$ ./splunk btool --app=appname props list --debug | grep TRUNCATE ( To find the path where app is configured and its TRUNCATE value)
2) ./splunk btool --app=appname props list --debug | grep sourcetype ( To find the Truncate value specific to app and sourcetype)
3) Validated the props.conf details by using the cat /opt/splunk/etc/apps/appname/local/props.conf found the below actual configuration.
[sourceytpename]
TRUNCATE = 800000
TIME_FORMAT =
TIME_PREFIX=\[
DATETIME_CONFIG=/etc/apps/appname/local/datetime.xml
SHOULD_LINEMERGE=false
LINE_BREAKER=([\r\n]+)\[(?:\d{1,2}\/\d{1,2}\/\d{2}\s\d{1,2}:\d{2}:\d{2}\:\d{3}\s|\d{4}-\d{2}-\d{2}T\d{1,2}:\d{2}:\d{2}\.\d{3}(?:Z|[+-]\d\d?:?(?:\d\d)?))
MAX_TIMESTAMP_LOOKAHEAD=30
4) Based on the actual truncate value as reference value, Identify the maximum length and frequency of occurrence for last 7 days
sourcetype="xx.xx.xx" | eval length=len(_raw) | stats max(length) as length by sourcetype
The maximum length was more then 512273 --> But well below the actual Truncate value=800000
Frequency of occurence was only one time it had reached more then 500000
sourcetype="xx.xx.xxt" | eval length=len(_raw) | where length>=500000 | stats count by _time length
Question:
1) When the actual Truncate value=800000 is more then the maximum Truncate value=512273,in this case we should not get any warning alert.
2) By increasing the Truncate value will not solve this issue as the actual truncate value is more then the maximum truncate value.
Kindly guide me if how to fix this issue.
