Getting Data In

What happens in a distributed environment with auto load balancing after the minFreeSpace is reached on an indexer?

mrg2k8
Explorer

Hello,

Having a distributed environment with N indexers and M servers sending data in a load-balanced way (autoLB=true) towards those indexers, what happens if the minimum free space is reached for one of the indexers?

The documentation says the indexer will not take any data until space is freed up and also "If one indexer is down, the forwarder immediately switches to another".

Will the load balancing algorithm on the servers just switch to the next indexer or will the data during that time frame ("The forwarder routes data to different indexers based on a specified time interval.") be lost?

1 Solution

bmacias84
Champion

Well that depends if you have Indexer acknowledgment enabled on your forwarders. If your Forwarder do not acknowledgement enabled any data send before the forwarder received an unable to connect will be dropped the floor. If the Indexer is has reached its minimum free space forwarders will be unable to connect and will attempt to connect to the next available Indexer. When a Forwarder has indexer acknowledgement enabled the forwarder will wait until Indexer has responded with an acknowledgement before clearing the event from the TCPOUT queue.

If you are worried about data loss enable indexer acknowledgement on the Forwarder, but you many have to also increase your TCPOUT queue beyond the default to prevent data from falling to the floor if your queue back up.

View solution in original post

bmacias84
Champion

Well that depends if you have Indexer acknowledgment enabled on your forwarders. If your Forwarder do not acknowledgement enabled any data send before the forwarder received an unable to connect will be dropped the floor. If the Indexer is has reached its minimum free space forwarders will be unable to connect and will attempt to connect to the next available Indexer. When a Forwarder has indexer acknowledgement enabled the forwarder will wait until Indexer has responded with an acknowledgement before clearing the event from the TCPOUT queue.

If you are worried about data loss enable indexer acknowledgement on the Forwarder, but you many have to also increase your TCPOUT queue beyond the default to prevent data from falling to the floor if your queue back up.

Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Laser Bananas and Edge Hubs: Exploring Operational Technology (OT) Data Through a ...

  OT is a different environment to traditional IT and can have interesting challenges when interfacing the ...

Event Series: Mastering AI Tokenomics and Splunk Agent Observability

Beyond the Black Box: Correlating AI Performance and Tokenomics with Splunk Agent Observability   As ...

span_metrics: The OpenTelemetry-Idiomatic Way to See Inside Your Services

You open a trace in Splunk Observability Cloud and everything looks fine. One root span, order-pipeline, with ...