We are using Splunk 6.2.6.
I am using heavy forwarder at remote sites to forward data to a central indexer.
To make sure data is received we are using the useACK=true attribute.
On one of our sites, the connection is broken between the central indexer, so no forwarding can be completed.
Now the heavy forwarder, which is used locally as a search head is getting handshake timeouts, and prevents all GUI communication.
I would assume it is from failing to communicate to the central indexer.
We have tried to comment out heavy forwarder's outputs.conf file thinking that after a restart it would then be able to communicate....No such luck.
So....When a backlog of events to forward to an indexer builds up in a heavy forwarder, is there some file/directory we can delete to remove the backlog, and restore normal GUI communication?
By default the Splunk's input queues are stored in Memory, so forwarder crash/shutdown/restart will empty the queue automatically. If you're using a persistence queue, than it could be located in file sytem/disk (http://docs.splunk.com/Documentation/Splunk/6.2.6/Data/Usepersistentqueues)
The issue could be something else. Do you see any specific error in splunkd.log ?
By default the Splunk's input queues are stored in Memory, so forwarder crash/shutdown/restart will empty the queue automatically. If you're using a persistence queue, than it could be located in file sytem/disk (http://docs.splunk.com/Documentation/Splunk/6.2.6/Data/Usepersistentqueues)
The issue could be something else. Do you see any specific error in splunkd.log ?