Splunk licensing error

[L_Petch]

Hello,

I am getting the below error on two of my indexers. The indexers in question are on a different site (Site2) to the other two indexers & license manager in the cluster (site1). Site 1is working correctly with the same configuration as the indexers for site 2.

My guess is networking but both indexers can connect to the LM on this port and there are no issues showing on the firewall between the two. All troubleshooting I have tried shows doesn't show any connectivity issues. Anyone come across this problem and have a solution?

#######################################################################

HttpClientRequest [2156984 LMTrackerExecutorWorker-0] - Returning error HTTP/1.1 502 Error connecting: Connection reset by peer

ERROR LMTracker [2156984 LMTrackerExecutorWorker-0] - failed to send rows, reason='Unable to connect to license manager=https://****:8089 Error connecting: Connection reset by peer'

#######################################################################

[livehybrid]

Hi @L_Petch 

The "Connection reset by peer" error indicates that the TCP connection was established, but then abruptly closed by the remote side (the License Manager or a network device in between). This often happens during the SSL/TLS handshake or if the License Manager itself encounters an issue processing the request from the Site2 indexers.

1. Verify License Master URI Configuration: Ensure the master_uri in $SPLUNK_HOME/etc/system/local/server.conf on the affected indexers (Site2) correctly points to the License Manager using HTTPS and port 8089.

   [license]
   master_uri = https://<LM_hostname_or_IP>:8089

   Replace <LM_hostname_or_IP> with the actual hostname or IP address of your License Manager. Confirm this is resolvable from the Site2 indexers.

2. Check SSL/TLS Certificate and Configuration:

   • Certificate Trust: The SSL certificate used by the License Manager on port 8089 must be trusted by the Site2 indexers. If using custom certificates, ensure the CA chain is correctly installed on the indexers.
   • Test SSL Connection: From one of the problematic Site2 indexers, use openssl to test the SSL handshake directly:

     openssl s_client -connect <LM_hostname_or_IP>:8089 -servername <LM_hostname_if_SNI_used>

     This command can help identify SSL handshake failures and certificate issues. You may wish to try passing with your certificate files:

     openssl s_client -connect <LM_hostname_or_IP>:8089 -servername <LM_hostname_if_SNI_used> \
         -cert /path/to/your/client-cert.pem \
         -key /path/to/your/client-key.pem \
         -CAfile /path/to/your/ca-cert.pem​

       
3. Check yuor logs:
   1. On affected indexers (Site2): Examine splunkd.log for more detailed messages around the "failed to send rows" error. Look for messages related to SSL, TLS, or connection failures.
   2. On the License Manager (Site1): Examine splunkd.log for any errors corresponding to connection attempts from the affected indexers' IPs. Look for SSL errors, resource issues, or licensing-specific messages.

      Inspect Splunk Logs:

   3. On your SH: Search in _internal, something like this would be a good starting point

index=_internal host= sourcetype=splunkd (log_level=ERROR OR log_level=WARN) ("SSL" OR "handshake" OR "connection from" OR "LicenseManager

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[L_Petch · Accepted solution]

Hi all,

Thanks for the quick replies. They help with troubleshooting but the issue ended up being a firewall that isn't documented and i wasnt informed was part of the route whilst trying to originally diagnose the issue.

[PickleRick]

To be precise, "connection reset by peer" means that the other end sent a packet with a RST flag.

This might happen when:

1) The traffic is not filtered but the port is not open on the other side. In this case a server would simply respond with RST to the initial SYN packet and no session would be established at all.

2) The session is established (there is normal three-way handshake) but either:

2a) There is some low-level problem with the connection and the IP stack on the other end decides that it's unrecoverable and decides to close the session abruptly.

2b) There is some intermediate solution monitoring/inspecting/whatevering the traffic and breaking the connection in case it finds anything "wrong" or "suspicious". In my experience I encountered IPS solutions which would send spoofed RST both ways (to the client and server) because it was seeing unknown TLS certificates.

2c) The connection on the TCP level is working OK but there is some problem with a higher layer protocol and the protocol doesn't have signaling for that and doesn't allow for graceful shutdown and instead just closes the connection. Typical example is again TLS-oriented - when the server doesn't like client's crypto proposals (or client's certificate if you're using mTLS), it will send a TLS alert within the TLS negotiation session and then simply close the connection. It should be reflected in logs on the server's side in such case.

It's often good to get a network dump (tcpdump/wireshark) from both ends of the connection to see who's sending the RST and at which moment.

[livehybrid]

Good points @PickleRick 

So I guess further to my previous reply @L_Petch  - Is there a firewall between Site1 and Site2, and if so are you able to verify that this isnt causing an issue here!

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[kiran_panchavat]

@L_Petch 

Check this https://community.splunk.com/t5/Splunk-Enterprise-Security/Unable-to-connect-to-license-master-since... 

https://community.splunk.com/t5/Security/Getting-an-issue-where-Splunk-hosts-can-t-reach-the-License...