Getting Data In

How to reload changed configs at Heavy forwarder?

brandy81
Path Finder

Hi All,

When I change some configs on HF, It seems that I need to restart HF according to the doc below.

https://docs.splunk.com/Documentation/Splunk/8.2.3/Admin/Configurationfilechangesthatrequirerestart

"If you make a configuration file change to a heavy forwarder, you must restart the forwarder, but you do not need to restart the receiving indexer."

Is it true? How to reload changed config without restart? If it is impossible, ingested data with HEC would be lost. What is the workaround?

Labels (1)
0 Karma
1 Solution

PickleRick
Champion

In general, it's not a very good idea to have just one ingestion point in case of "pushed" data (like syslog or HEC).

Some sources can buffer events for a short time and re-try sending to HEC in case of failure but we don't know if yours can do that. If you had multiple forwarders behind a load-balancer as @somesoni2 suggested, you could freely restart any single one of them without noticeable impact to the whole installation.

Oh, and you don't necessarily need F5 for that. You can go cheap and do it on haproxy or any other HTTP load-balancer you can think of 🙂

View solution in original post

0 Karma

PickleRick
Champion

In general, it's not a very good idea to have just one ingestion point in case of "pushed" data (like syslog or HEC).

Some sources can buffer events for a short time and re-try sending to HEC in case of failure but we don't know if yours can do that. If you had multiple forwarders behind a load-balancer as @somesoni2 suggested, you could freely restart any single one of them without noticeable impact to the whole installation.

Oh, and you don't necessarily need F5 for that. You can go cheap and do it on haproxy or any other HTTP load-balancer you can think of 🙂

View solution in original post

0 Karma

brandy81
Path Finder

Hi @PickleRick Thank you for your answer. Then, is the doc saying "HF should be restarted when configs are changed" correct? if there are only one HF?  I need to change props.conf for changing source type.

And when the source data is sent using HEC, dose the LB function should be implemented from source side? How can I do load balancing when I send data using HEC? 

0 Karma

PickleRick
Champion

Unfortunately - most config changes indeed require restart of the HF.

And you usually do it like that:

Indexer(s) <- HFs <- HTTP load-balancer <- sources

So you point your sources at your load-balancer which in turn distributes the requests between indenticaly configured HFs

Of course you need some load balancer which is able to keep track of backends' health, not just blindly round-robins throughout all configured backends.

0 Karma

isoutamo
SplunkTrust
SplunkTrust

And if you are using F5 then ensure that it is using FastL4 profile or otherwise you could lose some event when backend goes down.... I'm not 100% if this is still valid, but was at least couple of years ago.

r. Ismo

0 Karma

somesoni2
Revered Legend

What changes are you making in the HWF that you think requires Splunkd restart? How many HWFs you have (If there are multiple HWF behind a F5, you can restart them serially without data loss)? Some changes can be reloaded using rest API (https://docs.splunk.com/Documentation/Splunk/8.2.3/Admin/Configurationfilechangesthatrequirerestart#...).

Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!