Splunk Heavy on forwarder license

[jariw]

L.s.,

At our company we have multiple heavy forwarders. Normaly they talk to the central license manager, but for migrtation reason whe have to get them talking to themself. So a forwarder license is in order i think.

looks like an easy process. I did below

./splunk edit licenser-groups Forwarder -is_active 1

this will set in /opt/splunk/etc/system/local/server.conf  the settings below:

[license]

active_group = Forwarder

and

[lmpool:auto_generated_pool_forwarder]
description = auto_generated_pool_forwarder
quota = MAX
slaves = *
stack_id = forwarder

Have to set by myself master_uri = self

restart the server and it looks like a go.

When i do this on every heavy it will give the error in _internal

Duplicate license hash: [FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFD], also present on peer .

And

Duplicated license situation not fixed in time (72-hour grace period). Disabling peer..

Why?? Is it realy going to disable the forwarders when it uses the forwarder.license? What to do about it, where did i go wrong?

Thanks in advance

greetz 

Jari

[jariw]

Thanks for the response Paul..

I removed the master_uri. I can understand why, it is now manager_uri see below:

/opt/splunk/etc/system/local/server.conf [license]
/opt/splunk/etc/system/local/server.conf active_group = Forwarder
/opt/splunk/etc/system/default/server.conf connection_timeout = 30
/opt/splunk/etc/system/default/server.conf manager_uri = self
/opt/splunk/etc/system/default/server.conf receive_timeout = 30
/opt/splunk/etc/system/default/server.conf report_interval = 1m
/opt/splunk/etc/system/default/server.conf send_timeout = 30
/opt/splunk/etc/system/default/server.conf squash_threshold = 2000
/opt/splunk/etc/system/default/server.conf strict_pool_quota = true

I did something else, and that is remove the heavy's from the distributed search peers. Why they where there i don't know. It resolved one thing, the warning about disabling the peer...

The only thing remaining is the duplicate license hash (ffffff...) in the _internal index. I can understand the hash itself. Every forwaredr with this license has this hash. What i don't understand is why this warnimng. And it is only the warning for the heavy's  which were in the distributed serach peers. Not the one's which were not in that list. It seems something remained  someweher and keeps looking to the license on these heavy's and keeps reporting it is the same license...

Any idea?

[PaulPanther]

Okay, just to confirm master_uri and manager_uri is not set on the HF, right?

Could you check what files are located under etc/licenses? 

[jariw]

I have a heavy's without master_uri and Manager_uri. They are luckely working okay besides the error.

In etc/licenses is only download-trial folder. No forwarder.license

[PaulPanther]

Okay. Could you check/verify if you use the Distributed Monitoring Console and if the affected HFs are configured as Indexer under Settings --> Monitoring Console --> Settings --> General Setup? 

That could be the reason why the HeavyForwarder are configured as distributed search peers to monitor them in the DMC.

So if the license manager on the same instance as the DMC is check the config files for the affected HFs and may remove them.

[jariw]

I just found it in a few files inside the :

./apps/splunk_monitoring_console/lookups/hwf-list.csv

./apps/splunk_monitoring_console/lookups/dmc_forwarder_assets.csv

./apps/splunk_monitoring_console/lookups/dmc_forwarder_assets.csv.c

didn't removed them yet. The fact is we are going to rebuild the dmc/lm in a matter of weeks and wil see if these errors wil appear again. But i think they won't appear again. Until now it doesn't seem to matter, it all works great.

grts

jari

[PaulPanther]

Please remove parameter 

master_uri = self

 and try it again. If you get the same error please execute 

splunk btool server list license --debug

and share the output.