Splunk Indexing Acting Up

[TitanAE]

I'm not sure how to describe this problem. But I'm hoping someone can help me.

I have a syslog server receiving Router and Switch traffic. When it was just switch traffic everything worked perfectly. When I added router traffic things were fine for a few days. Now, in splunk web, only traffic to my router is shown.

Looking at the splunk indexing servers error logs, I see this:

WARN  LineBreakingProcessor - Truncating line because limit of 10000 bytes has been exceeded with a line length >= 13507 - data_source="/opt/splunk/var/log/splunk/remote_searches.log", data_host="servers host name", data_sourcetype="splunkd_remote_searches"

I did a bit of research on Splunk>answers and came across this:

LineBreakingProcessor - Truncating line because limit of 10000 has been exceeded

As well as similar results when searching the issue. That led me to create and update a props.conf file.

[source::tvit_syslog]
TRUNCATE = 15000
sourcetype = splunkd_remote_searches

#[sourcetype::splunkd_remote_searches]

So far I'm getting nowhere fast.

[martin_mueller]

Okay, I'm guessing you're reading the logs written by the syslog server with a forwarder? Common things to check on the forwarder / in its logs forwarded to splunk:

• is the forwarder configured to read the files? $SPLUNK_HOME/bin/splunk btool --debug inputs list
• any messages containing the log file names in splunkd.log? Read errors, permissions, etc.
• is it logging thruput for those log files in metrics.log?

[martin_mueller]

The message from the LineBreakingProcessor is unrelated to your issue. It's telling you that Splunk didn't configure line breaking correctly for Splunk's own sourcetype splunkd_remote_searches.

You mentioned you have a syslog server running. Is the syslog server still receiving data from both routers and switches?

[tentontitan]

Yes. The syslog server is still receiving traffic.