Getting Data In

What happens in a distributed environment with auto load balancing after the minFreeSpace is reached on an indexer?

mrg2k8
Explorer

Hello,

Having a distributed environment with N indexers and M servers sending data in a load-balanced way (autoLB=true) towards those indexers, what happens if the minimum free space is reached for one of the indexers?

The documentation says the indexer will not take any data until space is freed up and also "If one indexer is down, the forwarder immediately switches to another".

Will the load balancing algorithm on the servers just switch to the next indexer or will the data during that time frame ("The forwarder routes data to different indexers based on a specified time interval.") be lost?

1 Solution

bmacias84
Champion

Well that depends if you have Indexer acknowledgment enabled on your forwarders. If your Forwarder do not acknowledgement enabled any data send before the forwarder received an unable to connect will be dropped the floor. If the Indexer is has reached its minimum free space forwarders will be unable to connect and will attempt to connect to the next available Indexer. When a Forwarder has indexer acknowledgement enabled the forwarder will wait until Indexer has responded with an acknowledgement before clearing the event from the TCPOUT queue.

If you are worried about data loss enable indexer acknowledgement on the Forwarder, but you many have to also increase your TCPOUT queue beyond the default to prevent data from falling to the floor if your queue back up.

View solution in original post

bmacias84
Champion

Well that depends if you have Indexer acknowledgment enabled on your forwarders. If your Forwarder do not acknowledgement enabled any data send before the forwarder received an unable to connect will be dropped the floor. If the Indexer is has reached its minimum free space forwarders will be unable to connect and will attempt to connect to the next available Indexer. When a Forwarder has indexer acknowledgement enabled the forwarder will wait until Indexer has responded with an acknowledgement before clearing the event from the TCPOUT queue.

If you are worried about data loss enable indexer acknowledgement on the Forwarder, but you many have to also increase your TCPOUT queue beyond the default to prevent data from falling to the floor if your queue back up.

Get Updates on the Splunk Community!

How to Monitor Google Kubernetes Engine (GKE)

We’ve looked at how to integrate Kubernetes environments with Splunk Observability Cloud, but what about ...

Index This | How can you make 45 using only 4?

October 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this ...

Splunk Education Goes to Washington | Splunk GovSummit 2024

If you’re in the Washington, D.C. area, this is your opportunity to take your career and Splunk skills to the ...