Does anyone know where a heavy forwarder stores events to be sent to a splunk indexer when using Acknowledgement?

[kenoski]

We are using Splunk 6.2.6.

I am using heavy forwarder at remote sites to forward data to a central indexer.

To make sure data is received we are using the useACK=true attribute.

On one of our sites, the connection is broken between the central indexer, so no forwarding can be completed.

Now the heavy forwarder, which is used locally as a search head is getting handshake timeouts, and prevents all GUI communication.
I would assume it is from failing to communicate to the central indexer.

We have tried to comment out heavy forwarder's outputs.conf file thinking that after a restart it would then be able to communicate....No such luck.

So....When a backlog of events to forward to an indexer builds up in a heavy forwarder, is there some file/directory we can delete to remove the backlog, and restore normal GUI communication?

[somesoni2 · Accepted solution]

By default the Splunk's input queues are stored in Memory, so forwarder crash/shutdown/restart will empty the queue automatically. If you're using a persistence queue, than it could be located in file sytem/disk (http://docs.splunk.com/Documentation/Splunk/6.2.6/Data/Usepersistentqueues)

The issue could be something else. Do you see any specific error in splunkd.log ?