Getting Data In

Splunk Indexing Acting Up

TitanAE
New Member

I'm not sure how to describe this problem. But I'm hoping someone can help me.

I have a syslog server receiving Router and Switch traffic. When it was just switch traffic everything worked perfectly. When I added router traffic things were fine for a few days. Now, in splunk web, only traffic to my router is shown.

Looking at the splunk indexing servers error logs, I see this:

WARN  LineBreakingProcessor - Truncating line because limit of 10000 bytes has been exceeded with a line length >= 13507 - data_source="/opt/splunk/var/log/splunk/remote_searches.log", data_host="servers host name", data_sourcetype="splunkd_remote_searches"

I did a bit of research on Splunk>answers and came across this:

LineBreakingProcessor - Truncating line because limit of 10000 has been exceeded

As well as similar results when searching the issue. That led me to create and update a props.conf file.

[source::tvit_syslog]
TRUNCATE = 15000
sourcetype = splunkd_remote_searches

#[sourcetype::splunkd_remote_searches]

So far I'm getting nowhere fast.

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Okay, I'm guessing you're reading the logs written by the syslog server with a forwarder? Common things to check on the forwarder / in its logs forwarded to splunk:

  • is the forwarder configured to read the files? $SPLUNK_HOME/bin/splunk btool --debug inputs list
  • any messages containing the log file names in splunkd.log? Read errors, permissions, etc.
  • is it logging thruput for those log files in metrics.log?
0 Karma

martin_mueller
SplunkTrust
SplunkTrust

The message from the LineBreakingProcessor is unrelated to your issue. It's telling you that Splunk didn't configure line breaking correctly for Splunk's own sourcetype splunkd_remote_searches.

You mentioned you have a syslog server running. Is the syslog server still receiving data from both routers and switches?

0 Karma

tentontitan
New Member

Yes. The syslog server is still receiving traffic.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...