Splunk streamfwd how to balance incoming data between a cluster of indexers

[Kim]

Hello, colleagues.

I am using independent streamfwd as a service installed on Linux Ubuntu 22.04.05. Streamfwd gets settings from the stream app and gets the indexers list. Everything is ok, streamfwd balancing data between all indexers, but if I made a push from the master node to the indexers cluster, and the indexers are rebooting, data balancing breaks after that streamfwd sending data just to one indexer.

I can't find how to fix this.
Please help

thanks

[Prewin27]

@Kim 

This is a known issue with independent Streamfwd when used with an indexer cluster:
After a cluster bundle push and indexer restarts, Streamfwd may lose its knowledge of all available indexers and start sending all data to a single indexer.

Workaround - Restart Streamfwd After Indexer Restarts

Solution - Use Indexer Discovery
Indexer Discovery is the recommended way for forwarders (including Streamfwd) to dynamically learn the available indexers from the master node

Regards,
Prewin
Splunk Enthusiast | Always happy to help! If this answer helped you, please consider marking it as the solution or giving a kudos. Thanks!

[Kim]

Hello! 
Solution - Use Indexer Discovery  - Could you please tell me how to configure this for streamfwd? As I understand it, streamfwd takes the indexer list from the stream app....

After restarting streamfwd the problem goes away, you are right.

But this is not a solution in my perfectionist world.) 

Now I have the following settings:
cat /opt/streamfwd/local/inputs.conf
[streamfwd://streamfwd]
splunk_stream_app_location = https://x.x.x.x:8000/en-us/custom/splunk_app_stream/
disabled = 0

cat /opt/streamfwd/local/streamfwd.conf
[streamfwd]
port = 8889
ipAddr = x.x.x.x

httpEventCollectorToken = xxxxxxxxxxxxxxxxxxxxxxxxxxxxx

netflowReceiver.0.port = 9996
netflowReceiver.0.decoder = netflow
netflowReceiver.0.ip = 0.0.0.0
netflowReceiver.0.decodingThreads = 4

Thanks.

[livehybrid]

Hi @Kim 

Please can you share details on setting up indexer discovery for an independent streamfwd installation as you suggest? This isnt something I am particularly familiar with however I didnt think this was possible, as the streamfwd.conf only takes a list of indexers and HEC token?

https://docs.splunk.com/Documentation/StreamApp/8.1.5/DeployStreamApp/InstallStreamForwarderonindepe...

Is there an associated reference for the known issue so I can look into this further?

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[Prewin27]

@livehybrid @Kim 

Yes you are right, streamfwd.conf natively only takes list of indexer and HEC only directly.

The scenario i tested was using indexer discovery on the HF itself, Streamfwd can dynamically take the list of indexers from the outputs.conf. It worked without any issues.

The key is that the streamfwd process, after parsing network data, will then attempt to forward it. If its own streamfwd.conf doesn't specify a direct S2S or HEC target, it will fall back to using the Splunk forwarding mechanism configured in outputs.conf.

[Kim]

Hello! 

If I will use streamfwd like a light forwarder, is it possible to use outputs.conf ? 
Could you provide me with your config for this scenario? 

I can't find information in the documentation... 

thanks

[livehybrid]

Hi @Kim 

Are you able to post the streamfwd logs to see if there is anything in there which might suggest why it isnt re-establishing the connection to the indexers listed? 

Does a restart of streamfwd re-instate the connection to the other indexer nodes?

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing