Getting Data In

What happens when queues are blocked?

tristanrhys
New Member

Hi Guys,

We had a series of events that meant our SUFs were unable to forward to their respective indexers for about 10-days.

For this whole time, the queues were blocked.

Unfortunately, for those 10-days, it appears we have lost data :(.

I have read up a bit about queues, but one thing I am unsure about is what happens when the queues are blocked?

And in my situation, where communication was lost for an extended period of time, do events cache on local disk, if so, for how long? I checked limits.conf but couldn't find much, apart from maxqueuesize.

I guess I need to find the root cause, and future mitigation, of the data loss for the higher-ups. Cheers.

0 Karma

Dimitri_McKay
Splunk Employee
Splunk Employee

Unfortunately, the default local disk queue defined is incredibly small. So if the data isn't sent, the data in the queue is FIFO'd. You could increase the disk queue so you have much more local caching until the indexer comes up. But then you should be cognizant of the fact that at that point, once the indexer comes up, will the forwarder be able to catch up. Something to consider.

0 Karma

abhayneilam
Contributor

Hi Dimitri,

I am also facing the similar issue . But dont know how to deal with it. Do you have stepwise procedure to go through to get the solutions OR exactly to ensure the exact reason of the problem.

0 Karma

tristanrhys
New Member

I read up on persistent queues and have a good idea of those now.

Unfortunately, we had not configured persistent queues 😞 So looks like our data is gone.

0 Karma
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In November, the Splunk Threat Research Team had one release of new security content via the Enterprise ...

Index This | Divide 100 by half. What do you get?

November 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this ...

Stay Connected: Your Guide to December Tech Talks, Office Hours, and Webinars!

❄️ Celebrate the season with our December lineup of Community Office Hours, Tech Talks, and Webinars! ...