topic Splunk Indexing Acting Up in Getting Data In

Splunk Indexing Acting Up

TitanAE — Sat, 24 Feb 2018 00:41:23 GMT

I'm not sure how to describe this problem. But I'm hoping someone can help me.

I have a syslog server receiving Router and Switch traffic. When it was just switch traffic everything worked perfectly. When I added router traffic things were fine for a few days. Now, in splunk web, only traffic to my router is shown.

Looking at the splunk indexing servers error logs, I see this:

WARN  LineBreakingProcessor - Truncating line because limit of 10000 bytes has been exceeded with a line length >= 13507 - data_source="/opt/splunk/var/log/splunk/remote_searches.log", data_host="servers host name", data_sourcetype="splunkd_remote_searches"

I did a bit of research on Splunk>answers and came across this:

LineBreakingProcessor - Truncating line because limit of 10000 has been exceeded

As well as similar results when searching the issue. That led me to create and update a props.conf file.

[source::tvit_syslog]
TRUNCATE = 15000
sourcetype = splunkd_remote_searches

#[sourcetype::splunkd_remote_searches]

So far I'm getting nowhere fast.

Re: Splunk Indexing Acting Up

martin_mueller — Sat, 24 Feb 2018 01:52:53 GMT

The message from the LineBreakingProcessor is unrelated to your issue. It's telling you that Splunk didn't configure line breaking correctly for Splunk's own sourcetype splunkd_remote_searches.

You mentioned you have a syslog server running. Is the syslog server still receiving data from both routers and switches?

Re: Splunk Indexing Acting Up

tentontitan — Sat, 24 Feb 2018 02:00:42 GMT

Yes. The syslog server is still receiving traffic.

Re: Splunk Indexing Acting Up

martin_mueller — Sat, 24 Feb 2018 13:22:28 GMT

Okay, I'm guessing you're reading the logs written by the syslog server with a forwarder? Common things to check on the forwarder / in its logs forwarded to splunk:

is the forwarder configured to read the files? $SPLUNK_HOME/bin/splunk btool --debug inputs list
any messages containing the log file names in splunkd.log? Read errors, permissions, etc.
is it logging thruput for those log files in metrics.log?