Deployment Architecture

Why are there missing clients under forwarder management after upgrading Search Head from 8.0.4 to 8.1.9?

mello920
Path Finder

Hello,

This is my first time asking a question on here, so apologies if there's some format to follow.

My work center doesn't have a Splunk Admin/Engineer, so they asked if I could try upgrading Splunk since it's hosted on Linux and I'm a RHEL admin. 

My concern is there are no clients (besides the HF) showing up under Forwarder Management on Splunk Web. Am I supposed to re-add all the clients again? Or should they have started to communicate regardless? I know the indexer is working since we can search the latest AWS logs. But any Windows/Linux box doesn't show up anymore. All apps and indexes are showing, just no "deployed clients" underneath them.

The SH is the master. 

Any help is greatly appreciated!

Labels (2)
Tags (2)
0 Karma

Stefanie
Builder

The clients should not have to be re-added. You're saying that the Search Head is also the Deployment Master? I.E. in Forwarder Management you can see the apps and Server Classes that typically get pushed out to the forwarders? 

 

Were they showing up before you upgraded Splunk? What versions of Splunk Forwarders were installed on the Windows/Linux boxes?

 

Are you receiving any errors in /opt/splunk/var/log/splunkd.log?

0 Karma

mello920
Path Finder

Hello!

Yes, the SH is the Deployment Master. I actually looked into the Apps/Server Classes in Forwarder Management and was able to click on the "Edit Clients" button. I can see the servers pre-filled in.

So, I'm guessing they just stopped forwarding. From what I gathered from the office, on the Linux boxes the UF version is 8.0.5, and on the Window boxes, it's at least 8.2. Could the problem be that the SH is on 8.1.9 but the HF is not (8.0.4)?

I realize there's a lot of work to do to get everything synced up to the same version...fun times!!

Don't see any errors in splunkd.log for the SH. I verified that I can see metric logs coming in for the HF.

0 Karma

Stefanie
Builder

I don't think the versions being different would cause an issue like that. In our environment our Splunk servers are on 8.2.5 and some forwarders are still reporting using version 7.0.x 

 

Do your forwarders show up in the Monitoring Console?  There is a dashboard available under the Forwarders drop-down.

It will report if they're actively reporting data to Splunk or if they're "missing"

 

0 Karma

mello920
Path Finder

So figured it out!

In my attempts to upgrade the SH (no Splunk experience lol), I thought I needed to update the Pass4SymmKey on all three servers (SH, Idx, HF). Didn't know that each UF in each Windows/Linux box has a similar configuration setup in terms of directories like the main servers. Realized that they also use the Pass4SymmKey.

So uninstalled/reinstalled the UFs. I can see them all now in Forwarder Management.

Thank you though!!!

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

(view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...