Universal forwarder is listening to the wrong port for the splunkd process

[pfabrizi]

We are rolling out the UF to our windows servers, no apps yet, just the UF. The deploymentclient.conf only has the deployment server:
targetUri = xxx.xxx.xxx.xxx:8089.

this is causing some issues with another instance of SPLUNK our business folks have running.

How do I change the port that the deployment server listens on
How do I push this change to all the UF if they are not in a server class or have applications?

Thanks!

[micahkemp · Accepted solution]

After reading your comment response to @ddrillic it seems that the conflict has nothing to do with your deployment client or deployment server port, but rather that the UFs are listening on 8089 for their splunkd process. This can be changed in web.conf

[settings]

mgmtHostPort = <IP:port>
* Location of splunkd.
* Don't include http[s]:// -- just the IP address.
* Defaults to 127.0.0.1:8089.

[pfabrizi]

ok, so I need to change the web.conf on the deployment server in etc/system/local to
mgmhostport=127.0.0.1:newport

How do I get this to the UF's? I have no serverclasses and\or apps defined yet.

[micahkemp]

You need to make that change in an app that gets pushed to the UFs. To do that you'd create an app in etc/deployment-apps on your deployment server.

You don't need to make any changes to the deployment server's ports. You only need to have it configure the UFs to use different ports.

[pfabrizi]

ok. so If I do this.
1. create an app called Travelers_Windows_UF under etc/deployment-apps
2. put a deploymentclient.conf with targetUri = deployment server:8090 in local folder

forgive my questions, I don't completely understand all this yet.

is that the correct file?
will that over ride the deploymnetclient.conf that is in etc/system/default on the UF devices?

Thanks!

[micahkemp]

You do not need to change deploymentclient.conf at all. Your deployment server will still listen on 8089, and your deployment clients (your UFs) will still point to deploymentserver:8089.

All you are intending to change is the internal splunkd port that your UFs are running on (which won't really be used by anything other than themselves).

To do this you simply need to deploy an app with the appropriate setting in web.conf (as shown above). This involves creating the app in deployment-apps and either using the UI to create the serverclass and assign the app to that serverclass, or makeing these changes it via serverclass.conf directly. There shouldn't be any need to change any other ports or configurations.

[pfabrizi]

I made a new app, put a web.conf point to 127.0.0.1:8090 and subscribed my clients to that app. I watch one client I had access to and I could see the 8090 port in the logs.

Thank You.

[micahkemp]

Where is the issue occurring: on the UFs you're deploying to, or on the Deployment Server?

It seems that you're stating the former, which wouldn't really make sense as deploymentclient.conf doesn't set a port that any service listens on. Instead it configures the port that the remote end (the deployment server) is already listening on.

Please provide detail regarding where components are installed, and which are in conflict.

[pfabrizi]

Yes, the conflict is on the UF side.
if I look at the deploymentclient.conf that was sent with the install it has targetUri = ip of deploymentserver:8089.

I need to change that so it uses another port. I am guessing I need to change my deployment server to listen on a different port then change my UF deploymentclient.conf to use that?

I apologize, I have had no training and have no idea how to resolve this.

Thanks!

[naidusadanala]

Here are my thoughts and pleast test on your test instances before proceeding

Change the mgmthost port on the DS from 8089 to available port by making the changes in system/local/web.conf

Before that deploy the new app with new target uri port to the UF's and wait till all of them are restarted .

--> Once UF's are restarted , make the splunkd port change on the DS and restart it .

With this connectivity is established ffrom UF's to DS , because as splunk precedence looks at the app/local you would be fine .

[ddrillic]

-- this is causing some issues with another instance of SPLUNK our business folks have running.

Does the other Splunk instance run on the same server?

[pfabrizi]

Yes, so it is a port conflict on the UF.
I am not sure what I need to do on the DS to change the listening port
I am also not sure how to get each UF to get a new deploymentclient.conf to pickup the change.
I have no serverclasses or applications installed.

can I create a server class, subscribe all my UF's to that class and then push the port change from the DS .

Thanks!

[ddrillic]

-- Yes, so it is a port conflict on the UF

Not sure I'm following - can you please explain?

I had some doubts recently about the 8089 port role at What can be done when port 8089 is taken on the forwarder?

[pfabrizi]

Our company has 2 separate instances of SPLUNK, 1 managed by our business folks and one managed by our security team.

There are 2 instances of UF running on a windows server, but both are configured to use port 8089 which is causing a conflict on the server (UF).

I need to change my Deployment server port to something else and then push that to all 1140 UF servers. I don't have an apps configured and\or server classes defined on my Deployment server as we ere just pre-pushing the install as we wait for our remaining SPLUNK servers.

[ddrillic]

Got it. If you read my thread, I had the same confusion ; -) port 8089 on the UF is not crucial for the forwarder.