After installing Splunk on a new node as a LightWeightForwarder and configuring for the local logs I wanted to monitor I started seeing the following messages in the splunkd.log:
TailingProcessor - Could not send data to output queue (parsingQueue), retrying…
These messages only occurred after I added the file monitors, it had run overnight with just the unix application without issue.
I realized after seeing this message that I hadn't manually inspected the directories the monitors were looking at. I had over 1000 old webserver logs that I was telling splunk to forward over to the indexer, causing it to backup with the maxKBps setting by default.
I chose to cleanout the old files to reduce what i was indexing, but altering the maxKBps would work as well to help clear out the backlog. http://answers.splunk.com/questions/7750/increase-throughput-limit-on-a-lightweight-forwarder
Chatting about it on IRC, this may be a common issue on cold installs.
I realized after seeing this message that I hadn't manually inspected the directories the monitors were looking at. I had over 1000 old webserver logs that I was telling splunk to forward over to the indexer, causing it to backup with the maxKBps setting by default.
I chose to cleanout the old files to reduce what i was indexing, but altering the maxKBps would work as well to help clear out the backlog. http://answers.splunk.com/questions/7750/increase-throughput-limit-on-a-lightweight-forwarder
Chatting about it on IRC, this may be a common issue on cold installs.