We have a distributed Splunk deployment where we have deployed an indexer cluster consisting of a cluster master and 5 indexers. We recently exceeded our license quota four times. We got in touch with splunk and got license for a higher limit and have deployed the new licenses also. Even after this, one indexer out of the five is still showing license warning. When we try to run a search, we are constantly receiving errors from one indexer saying that it is has exceeded the splunk license limit.
We checked the Licensing page in the license master and it lists all the indexers in the license pool. When we checked the debug details, it was showing that 4 out of 5 indexers had 4 violations and one of them had five violations. All the other indexers are fine and we are able to search data from them.
Is this a valid scenario in case of clustered indexers? Is a reset license required to fix this issue? Please help out.
Normally, the license master is a standalone server with ALL other splunk tiers as Slaves.
Few things you can test
- Ensure that particular Indexer is NOT a license master of itself and from the master list the license elements
- Ensure it has license slave entry present for the indexer slave. Run btool on server.conf and ensure the license stanza have correct master_uri.
- Check the compatibility page for license slaves.
If above are not true, its better to raise a case with Splunk and send diag