topic Re: UF's are disappearing from ForwarderManagement in Deployment Architecture

Why are UF's are disappearing from ForwarderManagement

Gene — Mon, 14 Feb 2022 15:09:25 GMT

Dear Splunkers,

Can you please assist with following problem:

We have more 20 UF's installed on windows machines, all of them have deployment server set, and were visible in Forwarder Management. But in some time all of them disappeared from FM and are appearing from time to time there.

I have tried to delete $SPLUNK_HOME/etc/instance.cfg on several forwarders and restarted them but problem was not fixed.

Any ideas how to fix it and what can cause such strange behavior?

Regards,

Eugene

Re: UF's are disappearing from ForwarderManagement

isoutamo — Thu, 30 Dec 2021 09:31:51 GMT

Are they disappeared permanently or only time by time? If last then you must remember that when you restart or do some other configuration changes on DS side, it needs that DCs (UF's) will phone home again before you can see those there again?

Have you MC in place and if are they there under Forwarders (enable forwarder management first).

r. Ismo

Re: UF's are disappearing from ForwarderManagement

Gene — Thu, 30 Dec 2021 10:35:17 GMT

Hi, and thank you for response.

But actually situation is following:

we set up forwarders=>set DS=> they appear in console=> they disappear from console=> some of them sometimes appear, but not for a long time

I suppose that this can be connected to some firewall settings, but client assures that they can't find any connections that were blocked during that time. Also UF logs show:
INFO DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected

Re: UF's are disappearing from ForwarderManagement

isoutamo — Thu, 30 Dec 2021 10:37:30 GMT

And you have only one DS, not several behind LB?

Re: UF's are disappearing from ForwarderManagement

Gene — Thu, 30 Dec 2021 10:38:57 GMT

Yes, correct.

Re: UF's are disappearing from ForwarderManagement

Gene — Thu, 30 Dec 2021 10:41:54 GMT

I mean only one

Re: UF's are disappearing from ForwarderManagement

isoutamo — Thu, 30 Dec 2021 10:48:39 GMT

And your DS's splunk version is the highest or equal one than any other nodes UF+servers?

What "splunk btool deploymentclient list --debug" said? Are those values and places what you are expecting? And how about "splunk show deploy-poll" ?

I assume that telnet/curl from DC to DS:8089 (or what ever your mgmt port is) is working?

Those DCs are in same DC or behind VPN?

Re: UF's are disappearing from ForwarderManagement

Gene — Thu, 30 Dec 2021 11:16:01 GMT

Versions are the same.

btool and show deploy-poll show correct values.

telnet - clarifying with client, cause do not have access to endpoints where forwarders are installed.

and clients are in the same subnet, no VPN is used.

Re: UF's are disappearing from ForwarderManagement

isoutamo — Fri, 31 Dec 2021 08:40:16 GMT

You should try from UF side to DS curl/telnet. All traffic between those are initiated by DC not DS!

curl -vkI https://<Your DS fqdn>:8089

Above command show HEAD part of response with debug information.

For security reason it's good to disable 8089 (management) port on UF unless you are regularly using it from scripts etc. on UF side.

How about host based firewalls?

r. Ismo

Re: UF's are disappearing from ForwarderManagement

teunlaan — Fri, 31 Dec 2021 11:15:18 GMT

Please check if your deployment server is not restarting/crashing.

The deployment server won't show any UF clients if it just restarted. Only after de UF clients called home it will pop-up.

Re: UF's are disappearing from ForwarderManagement

Gene — Tue, 11 Jan 2022 09:51:42 GMT

DS is not restarting or crashing, the thing is that clients connect only once and after they can't reach DS. I assume that problem is with Firewall rules but for now client is checking this.

Re: UF's are disappearing from ForwarderManagement

isoutamo — Tue, 11 Jan 2022 10:10:34 GMT

If DC can connect to DS once after start, it's hard to think that the issue is in FW. Of course if you have L7/NG level FW then it's possible...
Can it be that your DC polling time is too long?

Re: UF's are disappearing from ForwarderManagement

SinghK — Tue, 11 Jan 2022 11:32:56 GMT

Not necessarily if yu ou can get a copy of the uf logs after they have connected once to DS can shed more light on this, once there was an app that caused this too. So n number of possibilities. Logs can on tell what's happening..

Re: UF's are disappearing from ForwarderManagement

Gene — Tue, 11 Jan 2022 11:38:04 GMT

Actually all settings are default, we didn't touch polling time. I will try to play with that also, thx

Re: UF's are disappearing from ForwarderManagement

Gene — Tue, 11 Jan 2022 11:41:11 GMT

Thx for the suggestion, but the strangest thing is that all forwarders are sending data as expected also to indexer, that is configured as DS. But not seen in console.

Re: UF's are disappearing from ForwarderManagement

isoutamo — Tue, 11 Jan 2022 11:47:03 GMT

Polling from DC - DS use port 8089 and sending data is using 9997 (or 9998 TLS) by default.

Re: UF's are disappearing from ForwarderManagement

Gene — Tue, 11 Jan 2022 11:55:50 GMT

Yes, correct. That is why I suspect that some firewall rules are blocking this connection, maybe some beckoning rules....

Re: UF's are disappearing from ForwarderManagement

teunlaan — Tue, 11 Jan 2022 12:30:56 GMT

So what is the internal log of the UF's telling? Do they try connect but it fails, or don't they even try.

You state they connect 1 time and than it stops., what is strange.

Check if you arn't pushing a deploymentclient.conf.

And is the config you're pushing restarting the Forwarder? if yes: try changing your serverclass so it does not restart the UF, see if it keeps the conection (maybe something failes @ the restart)

Re: UF's are disappearing from ForwarderManagement

Gene — Tue, 11 Jan 2022 15:00:12 GMT

I have checked logs and still can't find what's wrong:

01-11-2022 14:20:18.073 +0200 INFO DC:HandshakeReplyHandler [13276 HttpClientPollingThread_85591AD8-9097-47F4-B73E-4F63150ACA4D] - Handshake done

01-11-2022 14:21:30.273 +0200 INFO DS_DC_Common [5620 MainThread] - Initializing the PubSub system.
01-11-2022 14:21:30.273 +0200 INFO DS_DC_Common [5620 MainThread] - Initializing core facilities of PubSub system.
01-11-2022 14:21:30.335 +0200 WARN HTTPAuthManager [5620 MainThread] - pass4SymmKey length is too short. See pass4SymmKey_minLength under the general stanza in server.conf.
01-11-2022 14:21:30.335 +0200 INFO HttpPubSubConnection [872 HttpClientPollingThread_85591AD8-9097-47F4-B73E-4F63150ACA4D] - Initial attempt to obtain connection will try after=37.475 seconds.
01-11-2022 14:21:30.335 +0200 INFO DC:DeploymentClient [5620 MainThread] - Starting phonehome thread.
01-11-2022 14:21:30.335 +0200 INFO DS_DC_Common [5620 MainThread] - Deployment Client initialized.
01-11-2022 14:21:30.335 +0200 INFO ServerRoles [5620 MainThread] - Declared role=deployment_client.
01-11-2022 14:21:30.335 +0200 INFO DS_DC_Common [5620 MainThread] - Deployment Server not available on a dedicated forwarder.
01-11-2022 14:21:30.335 +0200 INFO DC:PhonehomeThread [6308 PhonehomeThread] - Phonehome thread start, intervals: handshakeRetry=12.0 phonehome=60.0.
01-11-2022 14:21:30.335 +0200 INFO ClusteringMgr [5620 MainThread] - initing clustering with: ht=60.000 rf=3 sf=2 ct=60.000 st=60.000 rt=60.000 rct=5.000 rst=5.000 rrt=10.000 rmst=600.000 rmrt=600.000 icps=25 sfrt=600.000 pe=1 im=0 ip=0 mob=5 mor=5 mosr=5 pb=5 rep_port= pptr=10 pptrl=100 fznb=10 Empty/Default cluster pass4symmkey=false allow Empty/Default cluster pass4symmkey=true rrt=restart dft=180 abt=600 sbs=1
01-11-2022 14:21:30.335 +0200 INFO DC:DeploymentClient [6308 PhonehomeThread] - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected

01-11-2022 14:21:42.338 +0200 INFO DC:DeploymentClient [6308 PhonehomeThread] - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected
01-11-2022 14:21:54.338 +0200 INFO DC:DeploymentClient [6308 PhonehomeThread] - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected

Re: UF's are disappearing from ForwarderManagement

SinghK — Tue, 11 Jan 2022 15:11:09 GMT

Why are logs complaining about pass4symmkey??