Currently, I have 2 seperate clusters. One 'old' 6.0 cluster, and a new cluster for 6.2.
The idea is to have our forwarders forwarding to both clusters at the same time. I modified the outputs.conf on the forwarders, and can see events coming in on both clusters. So far, so good.
When I take a closer look, I can see events dropping on most forwarders:
index=_internal sourcetype=splunkd "has begun dropping events"
I can't find the root cause of this. No queues are blocked, network seems to be ok, and the indexers (both clusters) are fine too. Also, when I look closer on the local queues, I cannot see any alarming levels as well. No throtteling either (no maxkbps messages)
index=_internal source="/opt/splunkforwarder/var/log/splunk/metrics.log" group=queue current_size_kb>0
Only message that occurs frequently is "File descriptor cache is full (100), trimming". For what I could find, it should be regarderd as an informational message, not really harming anything.
Who can help me out to find the actual bottleneck?
