Forwarder Management All clients phoning home later than expected

[sdiaz5796]

We have a stand-alone splunk instance in a closed area. We had to roll back the server to a snapshot and now the clients only phone home when we restart the splunk server. I've looked at the splunk log, phonehome log, checked the outputs.conf. I've run telnet server:8089 and 9997 from the clients and the ports are open listening. Any help would be appreciated.  We are on version 9.3.1 

[PickleRick]

What do you mean by "clients phoning home only when you restart the DS"? How did you determine this? The clients phone home on schedule - it's asynchronous versus whatever the DS is doing.

[sdiaz5796]

 How did you determine this? - This is what the Forwarder Management Web UI shows us, client phone home time stamp coincides with the restart. 

[PickleRick]

OK. So this is not (or at least might not be)  about the phonehomes as such but on the info shown in the DS console.

I'd go for

1) Verifying on selected forwarders that the phonehomes are shown in the splunkd.log

2) Checking the logs on the DS itself to see if it can see the phonehomes.

3) Checking if you have the selective routing properly configured on the DS. https://help.splunk.com/en/splunk-enterprise/administer/manage-distributed-deployments/9.2/configure... (it's not about upgraded instances only; we had this issue lately on a new installation of 9.3.something).

[LAME-Creations]

It sounds like your clients (forwarders) are not consistently communicating with the Splunk instance after the snapshot rollback. Since telnet confirms ports 8089 and 9997 are open, and the issue only resolves temporarily after restarting the Splunk server, here are a few steps to troubleshoot:

1. Check Forwarder Configuration: Verify the deploymentclient.conf on the clients points to the correct Splunk server hostname/IP and port (8089 for management). Ensure the phoneHomeIntervalInSecs is set appropriately (default is 60 seconds).
2. Validate Server Rollback Impact: The rollback may have caused a mismatch in SSL certificates or server identity. Check if the server’s server.conf or certificates (in $SPLUNK_HOME/etc/auth/) were altered. Regenerate or redeploy certificates if needed.
3. Inspect Splunkd Logs on Clients: Look at $SPLUNK_HOME/var/log/splunk/splunkd.log on the clients for errors related to connection failures or authentication issues when phoning home.
4. Network Stability: Ensure there are no intermittent network issues or firewalls blocking consistent communication. Test with tcpdump or netstat on the server to confirm client connections.
5. Indexer Acknowledgment: If using indexer acknowledgment, verify the outputs.conf on clients has useACK=true and check for any backlog in the indexing queue on the server.
6. Splunk Version Compatibility: Confirm the forwarders are on a compatible version with 9.3.1. If not, upgrade them to match.

Try restarting the forwarders after checking the above. If the issue persists, share any relevant errors from the client or server logs for further assistance.

[sdiaz5796]

Still no success after attempting all the steps below. Checked splunkd log on a few fowarders as well as the Deployment server and neither indicated connection errors. One question I have is in regards to indexes. From the webui i see the _dsphonehome, _dsappevent, _dsclient, but I don't see those indexes in the indexes.conf file on the deployment server. Another note is I found this and wondering if this could help? Our Splunk instance is at version 9.3.1.

https://community.splunk.com/t5/Splunk-Enterprise/After-upgrading-my-DS-to-Enterprise-9-2-2-clients-...