Getting Data In

What is the impact if a Splunk forwarder loses communication with the indexer?

iurie_tuluc
Engager

Hello,

Can you please elaborate about the Forwarder/Indexer communication? As I understand the communication is realized via TCP using some Splunk proprietary mechanism.

I’d like to understand what will happen if the Forwarder will lose the connectivity with the Indexer. How will the forwarder’s host will behave on some heavy load with a big log records generated, including the situation when the Indexer is unexpectedly unreachable, like i.e. all the packets from the Forwarder will be dropped on the Indexer. Will the communication reestablish automatically when the Indexer will become reachable again?

Am curious if such tests where done and what are the results.

Thanks,

Iurie

1 Solution

somesoni2
SplunkTrust
SplunkTrust

This is a very common (unwanted) scenario and you can find many discussion around that in Splunk community. Like this one

https://answers.splunk.com/answers/1114/what-happens-to-my-events-at-splunk-light-forwarder-when-the...

In summary, Forwarders have queue with limited capacity and will start to drop events if that queue is full. If you've more than one indexers then setting up LB and Indexer acknowledgement can help you avoid in-flight data loss.
http://docs.splunk.com/Documentation/Splunk/6.4.2/Forwarding/Protectagainstlossofin-flightdata

View solution in original post

somesoni2
SplunkTrust
SplunkTrust

This is a very common (unwanted) scenario and you can find many discussion around that in Splunk community. Like this one

https://answers.splunk.com/answers/1114/what-happens-to-my-events-at-splunk-light-forwarder-when-the...

In summary, Forwarders have queue with limited capacity and will start to drop events if that queue is full. If you've more than one indexers then setting up LB and Indexer acknowledgement can help you avoid in-flight data loss.
http://docs.splunk.com/Documentation/Splunk/6.4.2/Forwarding/Protectagainstlossofin-flightdata

iurie_tuluc
Engager

yes, thanks. But beside the risk of losing the logs, I am analyzing a potential impact over the forwarder's host. I.e. in a brutal scenario when the indexer's server will start dropping everything coming from the client using IPTABLES.

0 Karma
Get Updates on the Splunk Community!

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...

Splunkbase | Splunk Dashboard Examples App for SimpleXML End of Life

The Splunk Dashboard Examples App for SimpleXML will reach end of support on Dec 19, 2024, after which no new ...

Understanding Generative AI Techniques and Their Application in Cybersecurity

Watch On-Demand Artificial intelligence is the talk of the town nowadays, with industries of all kinds ...