Getting Data In

What happens when queues are blocked?

tristanrhys
New Member

Hi Guys,

We had a series of events that meant our SUFs were unable to forward to their respective indexers for about 10-days.

For this whole time, the queues were blocked.

Unfortunately, for those 10-days, it appears we have lost data :(.

I have read up a bit about queues, but one thing I am unsure about is what happens when the queues are blocked?

And in my situation, where communication was lost for an extended period of time, do events cache on local disk, if so, for how long? I checked limits.conf but couldn't find much, apart from maxqueuesize.

I guess I need to find the root cause, and future mitigation, of the data loss for the higher-ups. Cheers.

0 Karma

Dimitri_McKay
Splunk Employee
Splunk Employee

Unfortunately, the default local disk queue defined is incredibly small. So if the data isn't sent, the data in the queue is FIFO'd. You could increase the disk queue so you have much more local caching until the indexer comes up. But then you should be cognizant of the fact that at that point, once the indexer comes up, will the forwarder be able to catch up. Something to consider.

0 Karma

abhayneilam
Contributor

Hi Dimitri,

I am also facing the similar issue . But dont know how to deal with it. Do you have stepwise procedure to go through to get the solutions OR exactly to ensure the exact reason of the problem.

0 Karma

tristanrhys
New Member

I read up on persistent queues and have a good idea of those now.

Unfortunately, we had not configured persistent queues 😞 So looks like our data is gone.

0 Karma
Get Updates on the Splunk Community!

Splunk Observability Cloud | Unified Identity - Now Available for Existing Splunk ...

Raise your hand if you’ve already forgotten your username or password when logging into an account. (We can’t ...

Index This | How many sides does a circle have?

February 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...