I have a forwarder on AIX with vresion 4.3.3 that probably has a problem with its parsingqueue
I see the following in metrics.log:
02-13-2013 16:47:50.219 +0100 INFO Metrics - group=queue, name=parsingqueue, max_size_kb=512, current_size_kb=449, current_size=9, largest_size=9, smallest_size=8 02-13-2013 16:48:21.226 +0100 INFO Metrics - group=queue, name=parsingqueue, max_size_kb=512, current_size_kb=449, current_size=9, largest_size=9, smallest_size=9
splunkd.log contains a lot of :
02-13-2013 17:01:37.238 +0100 INFO TailingProcessor - ...continuing. 02-13-2013 17:01:42.241 +0100 INFO TailingProcessor - Could not send data to output queue(parsingQueue), retrying...
Restarting splunk does not change the currentsizekb or current_size values so I tried to increase the queue size following this answer:
This leads to an increase of maxsizekb and currentsizekb but does not result in the forwarder sending anything to the indexer.
If current_size indicates how many events are in the queue the this number is relatively low.
Is there a way to debug what events are stuck in a queue?
Can I somehow manually force the forwarder to empty the queue and drop the events (I know, that this is ugly)?
Another strange thing is, that once in a while (every cupple of hours) the logs are suddenly indexed, but I did not find any hints in splunkd.log or metrics.log. There is an identical system with the same configuration that works fine. The indexer is not very busy it indexes about 30-40GB a day.
Thanks for your help,
If this is a forwarder, the problem is usually a step after :
Thanks for replying, the indexer queues (SOS) seem to be ok, the 256KBps is not a problem either the forwarder has a thruput close to 0 for most of the time and then from time to time indexes its data(I don't see why it behaves like this). I see a couple of WARN TcpOutputProc - Raw connection to ip=