Deployment Architecture

Why are there missing clients under forwarder management after upgrading Search Head from 8.0.4 to 8.1.9?

mello920
Path Finder

Hello,

This is my first time asking a question on here, so apologies if there's some format to follow.

My work center doesn't have a Splunk Admin/Engineer, so they asked if I could try upgrading Splunk since it's hosted on Linux and I'm a RHEL admin. 

My concern is there are no clients (besides the HF) showing up under Forwarder Management on Splunk Web. Am I supposed to re-add all the clients again? Or should they have started to communicate regardless? I know the indexer is working since we can search the latest AWS logs. But any Windows/Linux box doesn't show up anymore. All apps and indexes are showing, just no "deployed clients" underneath them.

The SH is the master. 

Any help is greatly appreciated!

Labels (2)
Tags (2)
0 Karma

Stefanie
Builder

The clients should not have to be re-added. You're saying that the Search Head is also the Deployment Master? I.E. in Forwarder Management you can see the apps and Server Classes that typically get pushed out to the forwarders? 

 

Were they showing up before you upgraded Splunk? What versions of Splunk Forwarders were installed on the Windows/Linux boxes?

 

Are you receiving any errors in /opt/splunk/var/log/splunkd.log?

0 Karma

mello920
Path Finder

Hello!

Yes, the SH is the Deployment Master. I actually looked into the Apps/Server Classes in Forwarder Management and was able to click on the "Edit Clients" button. I can see the servers pre-filled in.

So, I'm guessing they just stopped forwarding. From what I gathered from the office, on the Linux boxes the UF version is 8.0.5, and on the Window boxes, it's at least 8.2. Could the problem be that the SH is on 8.1.9 but the HF is not (8.0.4)?

I realize there's a lot of work to do to get everything synced up to the same version...fun times!!

Don't see any errors in splunkd.log for the SH. I verified that I can see metric logs coming in for the HF.

0 Karma

Stefanie
Builder

I don't think the versions being different would cause an issue like that. In our environment our Splunk servers are on 8.2.5 and some forwarders are still reporting using version 7.0.x 

 

Do your forwarders show up in the Monitoring Console?  There is a dashboard available under the Forwarders drop-down.

It will report if they're actively reporting data to Splunk or if they're "missing"

 

0 Karma

mello920
Path Finder

So figured it out!

In my attempts to upgrade the SH (no Splunk experience lol), I thought I needed to update the Pass4SymmKey on all three servers (SH, Idx, HF). Didn't know that each UF in each Windows/Linux box has a similar configuration setup in terms of directories like the main servers. Realized that they also use the Pass4SymmKey.

So uninstalled/reinstalled the UFs. I can see them all now in Forwarder Management.

Thank you though!!!

0 Karma
Get Updates on the Splunk Community!

Improve Your Security Posture

Watch NowImprove Your Security PostureCustomers are at the center of everything we do at Splunk and security ...

Maximize the Value from Microsoft Defender with Splunk

 Watch NowJoin Splunk and Sens Consulting for this Security Edition Tech TalkWho should attend:  Security ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...