Splunk HA is not working when Secondary License manager server is active

[malisushil119]

we have below setup

Site 1- Search Head, Indexer, Cluster master & License Master

Site 2: Search Head, Indexer, Cluster master (Splunk service is stopped) & License Master (Splunk service is stopped)

after activating the site 2 LM and CM, no clients are connecting to the site 2(active LM/LDS), it shows error message as no clients phone homed and applying quarantine to indexer

[livehybrid]

Hi @malisushil119 

This log error is saying that it cannot send the TCPOut from that host to your indexer, not that it cannot connect to the license server. This is evidenced by the fact that its attempting to connect on port 9997 which is typically a Splunk output port rather than port 8089 which is the Splunk REST API port that the Cluster Manager/License manager etc uses for communication.

During this testing is the host that its attempting to connect to online? Are you able to confirm connectivity from the host using something like telnet/netcat?

Do you have indexers in both sides (using a Multisite cluster?) - If so are these logs ultimately ending up in Site B and later replicating to Site A once back online?

Can you share logs that show that the hosts in Site B are not able to connect to the License Server/Deployment Server in Site B? Again, are you able to confirm connectivity of these from the hosts?

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[malisushil119]

@livehybrid  i can see the clients are connecting to the secondary indexer twice on port 9997, connection is established, also the license utilization is more then 100%,  Do you have indexers in both sides (using a Multisite cluster?) - If so are these logs ultimately ending up in Site B and later replicating to Site A once back online?: is this causing this issue.

how to troubleshoot furher.

[isoutamo]

This is not a LM issue as you can see on your log, it just said that it cannot send event to indexers!

How you have configured your UF's (I expecting that we are talking those) outputs.conf? Are there indexer discovery in use or just IPs of those indexers in site1 and how about site2's indexers?

Is this multisite cluster with 2 site all indexers active or is this some kind of HA setup where you have identical configuration on site1 and site2 and then you somehow do switch over when need?

[malisushil119]

We have appiled bug fix related to version 9.3.8 and managed to resolve the issue.

[malisushil119]

@kknairr Since it was previously working, what was changed in between? Like a version upgrade or something you performed.: Splunk instances upgrade from 9.3.1 to 9.3.8

Did you check splunkd.log file on affected indexer to see for any license related errors? You may try restarting one license peer node in error state to see if it re-sync with active LM and clears the quarantine?

I think i saw that indexer is unable to connect to the active LDS server, is ether any command to run on the indexer so that it can join the  LDS.

[kknairr]

@malisushil119  You can verify the licenseMasterURI setting in server.conf file under [license] stanza to verify whether the indexer points to the active LM.
The above setting connects the peers with LM node.
You may also run splunk list licenses on the LM node to confirm the license is valid and loaded.
To list all the license peers that have contacted the license manager, you can run:

splunk list licenser-peers

Review the below reference for additional commands you can run using CLI.

You can also rule out any connectivity issues on port 8089 from Indexer node to LM using curl commands.
curl -k https://<your-lm-host>:8089

Ref: 

Manage licenses from the CLI | Splunk Enterprise (last updated 2025-07-04T13:21:06.140Z)

server.conf - Splunk Documentation

>>

If this post addressed your question, you can:

• Give it karma to show appreciation 👍
• Mark it as the solution if it solved your issue ✔️
• Add a comment if you’d like more details ✏️

Acknowledging helpful answers keeps the community strong and motivates contributors to continue sharing their expertise.

>>

[malisushil119]

Single LM should be active at a time, previously t was working, all clients were able to communicate with site 2 (Active LDS/LM), uf is running on version 9.3.2 while Splunk version is 9.3.8

[kknairr]

@malisushil119 Since it was previously working, what was changed in between? Like a version upgrade or something you performed.

Did you check splunkd.log file on affected indexer to see for any license related errors? You may try restarting one license peer node in error state to see if it re-sync with active LM and clears the quarantine?

>>

If this post addressed your question, you can:

• Give it karma to show appreciation 👍
• Mark it as the solution if it solved your issue ✔️
• Add a comment if you’d like more details ✏️

Acknowledging helpful answers keeps the community strong and motivates contributors to continue sharing their expertise.

>>

[kknairr]

@malisushil119 From your query, I guess you have a multisite cluster with separate License Managers per site. Can you confirm if your goal is to run them independently, or are you trying to achieve HA with a single LM behind a failover mechanism? 

>>

If this post addressed your question, you can:

• Give it karma to show appreciation 
• Mark it as the solution if it solved your issue 
• Add a comment if you’d like more details 

Acknowledging helpful answers keeps the community strong and motivates contributors to continue sharing their expertise.

>>