Getting Data In

What happens in a distributed environment with auto load balancing after the minFreeSpace is reached on an indexer?

mrg2k8
Explorer

Hello,

Having a distributed environment with N indexers and M servers sending data in a load-balanced way (autoLB=true) towards those indexers, what happens if the minimum free space is reached for one of the indexers?

The documentation says the indexer will not take any data until space is freed up and also "If one indexer is down, the forwarder immediately switches to another".

Will the load balancing algorithm on the servers just switch to the next indexer or will the data during that time frame ("The forwarder routes data to different indexers based on a specified time interval.") be lost?

1 Solution

bmacias84
Champion

Well that depends if you have Indexer acknowledgment enabled on your forwarders. If your Forwarder do not acknowledgement enabled any data send before the forwarder received an unable to connect will be dropped the floor. If the Indexer is has reached its minimum free space forwarders will be unable to connect and will attempt to connect to the next available Indexer. When a Forwarder has indexer acknowledgement enabled the forwarder will wait until Indexer has responded with an acknowledgement before clearing the event from the TCPOUT queue.

If you are worried about data loss enable indexer acknowledgement on the Forwarder, but you many have to also increase your TCPOUT queue beyond the default to prevent data from falling to the floor if your queue back up.

View solution in original post

bmacias84
Champion

Well that depends if you have Indexer acknowledgment enabled on your forwarders. If your Forwarder do not acknowledgement enabled any data send before the forwarder received an unable to connect will be dropped the floor. If the Indexer is has reached its minimum free space forwarders will be unable to connect and will attempt to connect to the next available Indexer. When a Forwarder has indexer acknowledgement enabled the forwarder will wait until Indexer has responded with an acknowledgement before clearing the event from the TCPOUT queue.

If you are worried about data loss enable indexer acknowledgement on the Forwarder, but you many have to also increase your TCPOUT queue beyond the default to prevent data from falling to the floor if your queue back up.

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...