topic Re: How to reload changed configs at Heavy forwarder? in Getting Data In

How to reload changed configs at Heavy forwarder?

brandy81 — Thu, 04 Nov 2021 12:53:17 GMT

Hi All,

When I change some configs on HF, It seems that I need to restart HF according to the doc below.

https://docs.splunk.com/Documentation/Splunk/8.2.3/Admin/Configurationfilechangesthatrequirerestart

"If you make a configuration file change to a heavy forwarder, you must restart the forwarder, but you do not need to restart the receiving indexer."

Is it true? How to reload changed config without restart? If it is impossible, ingested data with HEC would be lost. What is the workaround?

Re: How to reload changed configs at Heavy forwarder?

somesoni2 — Thu, 04 Nov 2021 14:04:30 GMT

What changes are you making in the HWF that you think requires Splunkd restart? How many HWFs you have (If there are multiple HWF behind a F5, you can restart them serially without data loss)? Some changes can be reloaded using rest API (https://docs.splunk.com/Documentation/Splunk/8.2.3/Admin/Configurationfilechangesthatrequirerestart#How_to_reload_files).

Re: How to reload changed configs at Heavy forwarder?

PickleRick — Thu, 04 Nov 2021 14:32:29 GMT

In general, it's not a very good idea to have just one ingestion point in case of "pushed" data (like syslog or HEC).

Some sources can buffer events for a short time and re-try sending to HEC in case of failure but we don't know if yours can do that. If you had multiple forwarders behind a load-balancer as @somesoni2 suggested, you could freely restart any single one of them without noticeable impact to the whole installation.

Oh, and you don't necessarily need F5 for that. You can go cheap and do it on haproxy or any other HTTP load-balancer you can think of 🙂

Re: How to reload changed configs at Heavy forwarder?

isoutamo — Thu, 04 Nov 2021 21:43:28 GMT

And if you are using F5 then ensure that it is using FastL4 profile or otherwise you could lose some event when backend goes down.... I'm not 100% if this is still valid, but was at least couple of years ago.

r. Ismo

Re: How to reload changed configs at Heavy forwarder?

brandy81 — Fri, 05 Nov 2021 08:27:52 GMT

Hi @PickleRick Thank you for your answer. Then, is the doc saying "HF should be restarted when configs are changed" correct? if there are only one HF? I need to change props.conf for changing source type.

And when the source data is sent using HEC, dose the LB function should be implemented from source side? How can I do load balancing when I send data using HEC?

Re: How to reload changed configs at Heavy forwarder?

PickleRick — Fri, 05 Nov 2021 08:59:05 GMT

Unfortunately - most config changes indeed require restart of the HF.

And you usually do it like that:

Indexer(s) <- HFs <- HTTP load-balancer <- sources

So you point your sources at your load-balancer which in turn distributes the requests between indenticaly configured HFs

Of course you need some load balancer which is able to keep track of backends' health, not just blindly round-robins throughout all configured backends.