What happens when queues are blocked?

[tristanrhys]

Hi Guys,

We had a series of events that meant our SUFs were unable to forward to their respective indexers for about 10-days.

For this whole time, the queues were blocked.

Unfortunately, for those 10-days, it appears we have lost data :(.

I have read up a bit about queues, but one thing I am unsure about is what happens when the queues are blocked?

And in my situation, where communication was lost for an extended period of time, do events cache on local disk, if so, for how long? I checked limits.conf but couldn't find much, apart from maxqueuesize.

I guess I need to find the root cause, and future mitigation, of the data loss for the higher-ups. Cheers.

[Dimitri_McKay]

Unfortunately, the default local disk queue defined is incredibly small. So if the data isn't sent, the data in the queue is FIFO'd. You could increase the disk queue so you have much more local caching until the indexer comes up. But then you should be cognizant of the fact that at that point, once the indexer comes up, will the forwarder be able to catch up. Something to consider.

[abhayneilam]

Hi Dimitri,

I am also facing the similar issue . But dont know how to deal with it. Do you have stepwise procedure to go through to get the solutions OR exactly to ensure the exact reason of the problem.

[tristanrhys]

I read up on persistent queues and have a good idea of those now.

Unfortunately, we had not configured persistent queues 😞 So looks like our data is gone.