Getting Data In

What happens in a distributed environment with auto load balancing after the minFreeSpace is reached on an indexer?

mrg2k8
Explorer

Hello,

Having a distributed environment with N indexers and M servers sending data in a load-balanced way (autoLB=true) towards those indexers, what happens if the minimum free space is reached for one of the indexers?

The documentation says the indexer will not take any data until space is freed up and also "If one indexer is down, the forwarder immediately switches to another".

Will the load balancing algorithm on the servers just switch to the next indexer or will the data during that time frame ("The forwarder routes data to different indexers based on a specified time interval.") be lost?

1 Solution

bmacias84
Champion

Well that depends if you have Indexer acknowledgment enabled on your forwarders. If your Forwarder do not acknowledgement enabled any data send before the forwarder received an unable to connect will be dropped the floor. If the Indexer is has reached its minimum free space forwarders will be unable to connect and will attempt to connect to the next available Indexer. When a Forwarder has indexer acknowledgement enabled the forwarder will wait until Indexer has responded with an acknowledgement before clearing the event from the TCPOUT queue.

If you are worried about data loss enable indexer acknowledgement on the Forwarder, but you many have to also increase your TCPOUT queue beyond the default to prevent data from falling to the floor if your queue back up.

View solution in original post

bmacias84
Champion

Well that depends if you have Indexer acknowledgment enabled on your forwarders. If your Forwarder do not acknowledgement enabled any data send before the forwarder received an unable to connect will be dropped the floor. If the Indexer is has reached its minimum free space forwarders will be unable to connect and will attempt to connect to the next available Indexer. When a Forwarder has indexer acknowledgement enabled the forwarder will wait until Indexer has responded with an acknowledgement before clearing the event from the TCPOUT queue.

If you are worried about data loss enable indexer acknowledgement on the Forwarder, but you many have to also increase your TCPOUT queue beyond the default to prevent data from falling to the floor if your queue back up.

Get Updates on the Splunk Community!

Fueling your curiosity with new Splunk ILT and eLearning courses

At Splunk Education, we’re driven by curiosity—both ours and yours! That’s why we’re committed to delivering ...

Splunk AI Assistant for SPL 1.1.0 | Now Personalized to Your Environment for Greater ...

Splunk AI Assistant for SPL has transformed how users interact with Splunk, making it easier than ever to ...

Unleash Unified Security and Observability with Splunk Cloud Platform

     Now Available on Microsoft AzureOn Demand Now Step boldly into the AI revolution with enhanced security ...