I have four independent indexers in a round robin, 2 are fairly old, 1 is a year old and my newest is maybe 3-4 months old.
hot/warm is on an SSD Mirror, cold is on spinning disk but currently barely used at all (thresholds not met yet)
Right after upgrading to 6.5.0, my newest indexer started filling up ALL of its indexing queues, I've taken it in/out a bunch of times and although not as often It's still randomly filling up all queues then stopping indexing of data.
All disks are healthy, no IO waits or anything.. I've watched the disks while this was happening and there are no issues with them.
Could these errors have something to do with it?
10-26-2016 13:00:08.697 -0700 WARN LineBreakingProcessor - Truncating line because limit of 10000 bytes has been exceeded with a line length >= 13431 - data_source="/opt/splunk/var/l
og/splunk/remote_searches.log", data_host="splunk08", data_sourcetype="splunkd_remote_searches"
... View more