Really weird problem with deployment server in a heavy forwarder

[hfaz]

Hello,

I have this really weird problem I've been trying to figure out for the past 2 days without success. Basically I have a Splunk architecture where I want to put the deployment server (DS) on the heavy forwarder since I don't have a lot of clients and it's just a lab. The problem is as follows : With a fresh Splunk Enterprise instance that is going to be the heavy forwarder, when I set up the client by putting in the deploymentclient.conf  the IP address of the heavy forwarder and port, it first works as intended and I can see the client in Forwarder Management. As soon as I enable forwarding on the Heavy Forwarder and put the IP addresses of the Indexers, the client doesn't show up on the Heavy Forwarder Management panel anymore but shows up in every other instance's Forwarder Management panel (Manager node, indexers etc..) ???? It's as if the heavy forwarder is forwarding the deployment client to all instances apart the heavy forwarder itself.

Thanks in advance

[PickleRick]

In a small environment (especially lab one) you can sometimes combine several roles into one server and HF as such is nothing more than just a Splunk Enterprise instance with forwarding enabled (actually you could argue that any component not being UF and not doing local indexing is a HF). So this setup (a DS doing also HF work) should work.

In this setup you should have:

1) On your indexer(s) - inputs.conf creating input for s2s from your HF (that's kinda obvious)

2) On your HF/DS - inputs.conf, outputs.conf (again - obvious stuff), serverclass.conf

3) On your UF/client HF - deploymentclient.conf pointing to your HF/DS instance

You also need to take into account that some things changed in 9.2. So if you upgraded to 9.2, see https://docs.splunk.com/Documentation/Splunk/9.2.0/Updating/Upgradepre-9.2deploymentservers

[hfaz]

Hello,

Thank you for your answer!

I made sure that all the points you mentioned are correctly implemented and also checked the documentation you sent. I fixed the problem by enabling the indexing on the Heavy Forwarder and now the client is appearing in it's fowarder management UI aswell. However, it's still showing in the other instances (Manager Server, Indexers etc.) aswell. Also, I don't want to turn on Indexing on the Heavy Forwarder, to not index data, is there a way to avoid enabling it and still get the client showing on the UI? It's a real pain bug i hope they fix it

[esalesap]

We did the same thing you did.  DS and HF on the same internet facing server.  We disabled the web interface and manage the deployment server with .conf files only.

All of the deployment clients for the DS/HF show up on in the Settings > Forwarder Management page as you describe.  All of the deployment clients in another deployment server show up too.  My guess is that the logs in the new _dsappevent, _dsphonehome, and  _dsclient indexes that are created in 9.2 is where that page gets its information.  It's very confusing.  There should be a column for the splunk_server in the display, so that we can tell which server is serving apps to which clients.

[gcusello]

Hi @hfaz ,

when you say that enabled forwarding to the Indexers, I suppose that you're peaking of logs.

Check that you don't have the deploymentclient.conf file in the HF, eventually distributed using an add-on.

Ciao.

Giuseppe

[hfaz]

Hello,

Thanks for your answer. I don't have a deployment.conf file in the HF, only the clients. The problem is that i need to turn Indexing on the HF in order to finally get the panel showing on HF's Forwarder management. Isn't there another solution?

[gcusello]

Hi @hfaz ,

not deployment.conf but deploymentclient.conf file!

In other words, check if, for error, you conigured also the HF as client.

Ciao.

Giuseppe

[hfaz]

Hello,

Yes sorry i meant deploymentclient.conf, i didn't configure HF as a client at all. All I did was pointing the client towards the HF and turning and forwarding on in the HF aswell.