Getting Data In

Splunk Indexing Acting Up

TitanAE
New Member

I'm not sure how to describe this problem. But I'm hoping someone can help me.

I have a syslog server receiving Router and Switch traffic. When it was just switch traffic everything worked perfectly. When I added router traffic things were fine for a few days. Now, in splunk web, only traffic to my router is shown.

Looking at the splunk indexing servers error logs, I see this:

WARN  LineBreakingProcessor - Truncating line because limit of 10000 bytes has been exceeded with a line length >= 13507 - data_source="/opt/splunk/var/log/splunk/remote_searches.log", data_host="servers host name", data_sourcetype="splunkd_remote_searches"

I did a bit of research on Splunk>answers and came across this:

LineBreakingProcessor - Truncating line because limit of 10000 has been exceeded

As well as similar results when searching the issue. That led me to create and update a props.conf file.

[source::tvit_syslog]
TRUNCATE = 15000
sourcetype = splunkd_remote_searches

#[sourcetype::splunkd_remote_searches]

So far I'm getting nowhere fast.

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Okay, I'm guessing you're reading the logs written by the syslog server with a forwarder? Common things to check on the forwarder / in its logs forwarded to splunk:

  • is the forwarder configured to read the files? $SPLUNK_HOME/bin/splunk btool --debug inputs list
  • any messages containing the log file names in splunkd.log? Read errors, permissions, etc.
  • is it logging thruput for those log files in metrics.log?
0 Karma

martin_mueller
SplunkTrust
SplunkTrust

The message from the LineBreakingProcessor is unrelated to your issue. It's telling you that Splunk didn't configure line breaking correctly for Splunk's own sourcetype splunkd_remote_searches.

You mentioned you have a syslog server running. Is the syslog server still receiving data from both routers and switches?

0 Karma

tentontitan
New Member

Yes. The syslog server is still receiving traffic.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

(view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...