Getting Data In

LineBreakingProcessor - Truncating line because limit of 1000000 bytes has been exceeded with a line length >= 1003520 - data_source="lsof", data_host="gbrdcr10328n02", data_sourcetype="lsof"

john_howley
Path Finder

I am getting this error in the splunkd.log.
i've seen a previous post which talks about the Line Breaking settings within Props.conf, but I don't have that section in any of my props,conf either system or nmon (which is the element being complained about)
in the Props.conf I have for NMON in [/apps/splunk-6.2.2-255606/splunk/etc/apps/nmon/default] directory I have the nmon config as

nmon config stanza

[nmon_config]

BREAK_ONLY_BEFORE=CONFIG,
MAX_EVENTS=100000
NO_BINARY_CHECK=1
SHOULD_LINEMERGE=true
TIME_FORMAT=%d-%b-%Y:%H:%M
TIME_PREFIX=CONFIG,
TRUNCATE=0

The Truncate=0 would lead me to beleive, from what I've seen on a previous post, don't truncate, but clearly it is.

Can anyone suggest which setting might be influencing this please?

1 Solution

rphillips_splk
Splunk Employee
Splunk Employee

@john_howley : The splunkd error pertains to the sourcetype=lsof as reported in data_sourcetype=lsof. You will need a [lsof] stanza defined in props.conf to apply to these events:

example:
set in $SPLUNK_HOME/etc/system/local/props.conf on all of your indexers:
[lsof]
TRUNCATE=0

restart splunk
$SPLUNK_HOME/bin
./splunk restart

Use the following attributes to define the length of a line.

TRUNCATE =
* Change the default maximum line length (in bytes).
* Although this is in bytes, line length is rounded down when this would
otherwise land mid-character for multi-byte characters.
* Set to 0 if you never want truncation (very long lines are, however, often a sign of
garbage data).
* Defaults to 10000 bytes.

View solution in original post

rphillips_splk
Splunk Employee
Splunk Employee

@john_howley : The splunkd error pertains to the sourcetype=lsof as reported in data_sourcetype=lsof. You will need a [lsof] stanza defined in props.conf to apply to these events:

example:
set in $SPLUNK_HOME/etc/system/local/props.conf on all of your indexers:
[lsof]
TRUNCATE=0

restart splunk
$SPLUNK_HOME/bin
./splunk restart

Use the following attributes to define the length of a line.

TRUNCATE =
* Change the default maximum line length (in bytes).
* Although this is in bytes, line length is rounded down when this would
otherwise land mid-character for multi-byte characters.
* Set to 0 if you never want truncation (very long lines are, however, often a sign of
garbage data).
* Defaults to 10000 bytes.

john_howley
Path Finder

Thanks rphillips - that worked..

0 Karma

Tejkumar451
Explorer

so , should we do this change on the indexer side or splunk forwarder side?

0 Karma

john_howley
Path Finder

as an additional note there are three .conf files that do contain a =1000000 they are

indexes.conf:maxMetaEntries = 1000000
limits.conf:max_chunk_queue_size = 1000000
props.conf:TRUNCATE = 1000000

The TRUNCATE one looks hopeful, but comes from the [kvstore] stanza which I initially thought was referring to certificate, but now I see it is key values - I will try creating a local version to allow > 1000000 and see what occurs.
[kvstore]
SHOULD_LINEMERGE = false
TIMESTAMP_FIELDS = datetime
TIME_FORMAT = %m-%d-%Y %H:%M:%S.%l %z
INDEXED_EXTRACTIONS = json
KV_MODE = none
TRUNCATE = 1000000

0 Karma

john_howley
Path Finder

Adjusting that setting in ..local/props.conf and restarting had no affect - stil lget the same error.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...