Why are there missing clients under forwarder management after upgrading Search Head from 8.0.4 to 8.1.9?

[mello920]

Hello,

This is my first time asking a question on here, so apologies if there's some format to follow.

My work center doesn't have a Splunk Admin/Engineer, so they asked if I could try upgrading Splunk since it's hosted on Linux and I'm a RHEL admin. 

My concern is there are no clients (besides the HF) showing up under Forwarder Management on Splunk Web. Am I supposed to re-add all the clients again? Or should they have started to communicate regardless? I know the indexer is working since we can search the latest AWS logs. But any Windows/Linux box doesn't show up anymore. All apps and indexes are showing, just no "deployed clients" underneath them.

The SH is the master. 

Any help is greatly appreciated!

[Stefanie]

The clients should not have to be re-added. You're saying that the Search Head is also the Deployment Master? I.E. in Forwarder Management you can see the apps and Server Classes that typically get pushed out to the forwarders? 

Were they showing up before you upgraded Splunk? What versions of Splunk Forwarders were installed on the Windows/Linux boxes?

Are you receiving any errors in /opt/splunk/var/log/splunkd.log?

[mello920]

Hello!

Yes, the SH is the Deployment Master. I actually looked into the Apps/Server Classes in Forwarder Management and was able to click on the "Edit Clients" button. I can see the servers pre-filled in.

So, I'm guessing they just stopped forwarding. From what I gathered from the office, on the Linux boxes the UF version is 8.0.5, and on the Window boxes, it's at least 8.2. Could the problem be that the SH is on 8.1.9 but the HF is not (8.0.4)?

I realize there's a lot of work to do to get everything synced up to the same version...fun times!!

Don't see any errors in splunkd.log for the SH. I verified that I can see metric logs coming in for the HF.

[Stefanie]

I don't think the versions being different would cause an issue like that. In our environment our Splunk servers are on 8.2.5 and some forwarders are still reporting using version 7.0.x 

Do your forwarders show up in the Monitoring Console?  There is a dashboard available under the Forwarders drop-down.

It will report if they're actively reporting data to Splunk or if they're "missing"

[mello920]

So figured it out!

In my attempts to upgrade the SH (no Splunk experience lol), I thought I needed to update the Pass4SymmKey on all three servers (SH, Idx, HF). Didn't know that each UF in each Windows/Linux box has a similar configuration setup in terms of directories like the main servers. Realized that they also use the Pass4SymmKey.

So uninstalled/reinstalled the UFs. I can see them all now in Forwarder Management.

Thank you though!!!