topic Splunk Heavy on forwarder license in Splunk Enterprise

Splunk Heavy on forwarder license

jariw — Mon, 25 Nov 2024 15:13:55 GMT

L.s.,

At our company we have multiple heavy forwarders. Normaly they talk to the central license manager, but for migrtation reason whe have to get them talking to themself. So a forwarder license is in order i think.

looks like an easy process. I did below

./splunk edit licenser-groups Forwarder -is_active 1

this will set in /opt/splunk/etc/system/local/server.conf the settings below:

[license] active_group = Forwarder

and

[lmpool:auto_generated_pool_forwarder] description = auto_generated_pool_forwarder quota = MAX slaves = * stack_id = forwarder

Have to set by myself master_uri = self

restart the server and it looks like a go.

When i do this on every heavy it will give the error in _internal

Duplicate license hash: [FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFD], also present on peer .

And

Duplicated license situation not fixed in time (72-hour grace period). Disabling peer..

Why?? Is it realy going to disable the forwarders when it uses the forwarder.license? What to do about it, where did i go wrong?

Thanks in advance

greetz

Jari

Re: Splunk Heavy on forwarder license

PaulPanther — Mon, 25 Nov 2024 14:43:29 GMT

Please remove parameter

master_uri = self

and try it again. If you get the same error please execute

splunk btool server list license --debug

and share the output.

Re: Splunk Heavy on forwarder license

jariw — Tue, 26 Nov 2024 09:04:56 GMT

Thanks for the response Paul..

I removed the master_uri. I can understand why, it is now manager_uri see below:

/opt/splunk/etc/system/local/server.conf [license]
/opt/splunk/etc/system/local/server.conf active_group = Forwarder
/opt/splunk/etc/system/default/server.conf connection_timeout = 30
/opt/splunk/etc/system/default/server.conf manager_uri = self
/opt/splunk/etc/system/default/server.conf receive_timeout = 30
/opt/splunk/etc/system/default/server.conf report_interval = 1m
/opt/splunk/etc/system/default/server.conf send_timeout = 30
/opt/splunk/etc/system/default/server.conf squash_threshold = 2000
/opt/splunk/etc/system/default/server.conf strict_pool_quota = true

I did something else, and that is remove the heavy's from the distributed search peers. Why they where there i don't know. It resolved one thing, the warning about disabling the peer...

The only thing remaining is the duplicate license hash (ffffff...) in the _internal index. I can understand the hash itself. Every forwaredr with this license has this hash. What i don't understand is why this warnimng. And it is only the warning for the heavy's which were in the distributed serach peers. Not the one's which were not in that list. It seems something remained someweher and keeps looking to the license on these heavy's and keeps reporting it is the same license...

Any idea?

Re: Splunk Heavy on forwarder license

PaulPanther — Tue, 26 Nov 2024 09:50:42 GMT

Okay, just to confirm master_uri and manager_uri is not set on the HF, right?

Could you check what files are located under etc/licenses?

Re: Splunk Heavy on forwarder license

jariw — Tue, 26 Nov 2024 10:45:42 GMT

I have a heavy's without master_uri and Manager_uri. They are luckely working okay besides the error.

In etc/licenses is only download-trial folder. No forwarder.license

Re: Splunk Heavy on forwarder license

PaulPanther — Tue, 26 Nov 2024 10:57:32 GMT

Okay. Could you check/verify if you use the Distributed Monitoring Console and if the affected HFs are configured as Indexer under Settings --> Monitoring Console --> Settings --> General Setup?

That could be the reason why the HeavyForwarder are configured as distributed search peers to monitor them in the DMC.

So if the license manager on the same instance as the DMC is check the config files for the affected HFs and may remove them.

Re: Splunk Heavy on forwarder license

jariw — Thu, 28 Nov 2024 07:56:39 GMT

I just found it in a few files inside the :

./apps/splunk_monitoring_console/lookups/hwf-list.csv

./apps/splunk_monitoring_console/lookups/dmc_forwarder_assets.csv

./apps/splunk_monitoring_console/lookups/dmc_forwarder_assets.csv.c

didn't removed them yet. The fact is we are going to rebuild the dmc/lm in a matter of weeks and wil see if these errors wil appear again. But i think they won't appear again. Until now it doesn't seem to matter, it all works great.

grts

jari