Getting Data In

What is the impact if a Splunk forwarder loses communication with the indexer?

iurie_tuluc
Engager

Hello,

Can you please elaborate about the Forwarder/Indexer communication? As I understand the communication is realized via TCP using some Splunk proprietary mechanism.

I’d like to understand what will happen if the Forwarder will lose the connectivity with the Indexer. How will the forwarder’s host will behave on some heavy load with a big log records generated, including the situation when the Indexer is unexpectedly unreachable, like i.e. all the packets from the Forwarder will be dropped on the Indexer. Will the communication reestablish automatically when the Indexer will become reachable again?

Am curious if such tests where done and what are the results.

Thanks,

Iurie

1 Solution

somesoni2
Revered Legend

This is a very common (unwanted) scenario and you can find many discussion around that in Splunk community. Like this one

https://answers.splunk.com/answers/1114/what-happens-to-my-events-at-splunk-light-forwarder-when-the...

In summary, Forwarders have queue with limited capacity and will start to drop events if that queue is full. If you've more than one indexers then setting up LB and Indexer acknowledgement can help you avoid in-flight data loss.
http://docs.splunk.com/Documentation/Splunk/6.4.2/Forwarding/Protectagainstlossofin-flightdata

View solution in original post

somesoni2
Revered Legend

This is a very common (unwanted) scenario and you can find many discussion around that in Splunk community. Like this one

https://answers.splunk.com/answers/1114/what-happens-to-my-events-at-splunk-light-forwarder-when-the...

In summary, Forwarders have queue with limited capacity and will start to drop events if that queue is full. If you've more than one indexers then setting up LB and Indexer acknowledgement can help you avoid in-flight data loss.
http://docs.splunk.com/Documentation/Splunk/6.4.2/Forwarding/Protectagainstlossofin-flightdata

iurie_tuluc
Engager

yes, thanks. But beside the risk of losing the logs, I am analyzing a potential impact over the forwarder's host. I.e. in a brutal scenario when the indexer's server will start dropping everything coming from the client using IPTABLES.

0 Karma
Get Updates on the Splunk Community!

Observability Unlocked: Kubernetes & Cloud Monitoring with Splunk IM

Ready to master Kubernetes and cloud monitoring like the pros? Join Splunk’s Growth Engineering team on ...

Index This | What did the zero say to the eight?

June 2025 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this month’s ...

Splunk Observability Cloud's AI Assistant in Action Series: Onboarding New Hires & ...

This is the fifth post in the Splunk Observability Cloud’s AI Assistant in Action series that digs into how to ...