Splunk Version 9.4.1

[ej87897]

I recently updated Splunk to the latest version. When I did this our Universal Forwarders and Heavy Forwarders stop showing up under forwarder management. It is showing under monitoring console and is showing data is flowing amongst the servers. I created edit deploymentclients.conf file to use FQDN , IP followed by port 8089 as well but nothing is working to allow forwarders or heavy forwarders to show up.

[livehybrid]

Hi @ej87897 

The architecture behind the Deployment Server within Splunk changed in version 9.2 and now the data on connections from clients (and which apps theyve downloaded) is stored in indexes prefixed _ds - The panels that display the clients under the Forwarder Management page rely on this information, if you have your DS configured to send all its data to an indexer tier and have not configured the selective forwarding then it will "appear" like nothing is working - when infact the clients will still be connecting and being managed by the DS as they should be. 

To fix this you need to apply a selective forwarding tweak to your outputs.conf - check out https://docs.splunk.com/Documentation/Splunk/9.4.1/Updating/Upgradepre-9.2deploymentservers

Essentially you need to configure outputs.conf as follows:

[indexAndForward]
index = true
selectiveIndexing = true 

Also - have you upgraded your indexers to at least 9.2? If not these wont have the required indexes configured on them to receive the data. 
Ensure your indexers have the following indexes:

[_dsphonehome]
[_dsclient]
[_dsappevent]

There may be other nuances depending on your architecture (such as sending via an intermediary forwarder) so check out the docs https://docs.splunk.com/Documentation/Splunk/9.4.1/Updating/Upgradepre-9.2deploymentservers page for more information 🙂

Please let me know how you get on and consider adding karma to this or any other answer if it has helped.
Regards

Will

[ej87897]

Our universal forwarders and Indexers are installed to the latest version, I have also done the edits to the conf file by adding the stanza but that didn't work either. I'm starting to think that this is an error with the 9.4.0 or 9.4.1 update.

[livehybrid · Accepted solution]

Hi @ej87897 

I have done this upgrade with the number of customers so I’m not sure if it’s a problem with 9.4.X itself but maybe a configuration somewhere which is causing the issue. 
a few more things to check:

if you do a search against your indexers against index=_ds* do you get any results?

if you do the same search from your deployment server, do you get any results?

Please let me know how you get on and we can try and work through the issue, but in the meantime you may wish to open a support case via splunk.com/support to get the ball rolling from that side. 
regards

Will

[ej87897]

I think it's a configuration issue ill open ticket I ran the command on the dm and got no results I did also go back through the default indexes.conf file on the indexer and saw that its still on version 9.2.0 and did not get updated to 9.4.0. 

[kiran_panchavat]

@ej87897I recommend raising a support ticket to troubleshoot this issue.

[kiran_panchavat]

@ej87897 

Since the forwarders are still sending data and appear in the Monitoring Console, they’re clearly functional and communicating with the Splunk infrastructure. The problem seems specific to the Deployment Server (DS) and its Forwarder Management UI.

Upgrade pre-9.2 deployment servers

This problem can occur in Splunk Enterprise 9.2 or higher if your deployment server forwards its internal logs to a standalone indexer or to the peer nodes of an indexer cluster. This issue can occur after an upgrade or in a new installation of 9.2 or higher. To rectify, add these settings to outputs.conf on the deployment server:

[indexAndForward]
index = true
selectiveIndexing = true

If you add these settings post-upgrade or post-installation, you might need to restart the deployment server.

You can see below URL:

https://docs.splunk.com/Documentation/Splunk/9.4.1/Updating/Upgradepre-9.2deploymentservers