I am running into a conflict with vCenter and using port 8089. I have the universal forwarder installed and I am deploying apps with the deployment server. Today, everything is the standard setup on 8089 and 9997. I know I can disable the port on the UF, but I do not want to do that because I want to push updates to the UF. Can I change the port on the UF and still push updates from the DS? If so, what about the rest of my environment that will still be using 8089? I need to manage those and do not want to make a system wide change for a handful of UF's.
Here is the way to change the default splunk forwarder management port:
In /opt/splunkforwarder/etc/system/local, add the following file and contents:
web.conf with
[settings]
mgmtHostPort = 127.0.0.1:8090
or whatever port that is not used, and restart splunk.
you should probably just disable the management port on UF
the UF will still fetch updates from the DS
for this , add in server.conf (locally or better use a app that push it remotely from DS)
[httpServer]
disableDefaultPort = true
If you disable the management port, what port does the UF fetch updates on? Is it not still on 8089 until the management port is changed?
The connection from UF to DS always has a local/source port using one of the Ephemeral ports. From the Wikipedia page:
The Internet Assigned Numbers Authority (IANA) suggests the range
49152 to 65535 (215+214 to 216−1) for dynamic or private ports.
Many Linux kernels use the port range 32768 to 61000.
Only the remote/destination (i.e. deployment server) side uses 8089 during this communication.
You can do (at least on Linux) a netstat -atn | grep 8089
on fthe UF and should see something like:
tcp 0 0 0.0.0.0:8089 0.0.0.0:* LISTEN
tcp 0 0 192.168.1.200:39602 192.168.1.100:8089 ESTABLISHED
while on the deployment server you get at the same time:
tcp 0 0 0.0.0.0:8089 0.0.0.0:* LISTEN
tcp 0 0 192.168.1.100:8089 192.168.1.200:39602 ESTABLISHED
In The Universal Forwarder docs I didn't find any reference that its management port is used at all. Still the UF is listening. Hence the first line of the UF's netstat output.
After some research: in fact port 8089 could be used after changing the admin password via REST calls.
The management port can be changed from 8089 (https://answers.splunk.com/answers/27394/running-splunk-forwarder-on-port-other-than-8089.html) in the web.conf mgmtHostPort setting but you would want to change that for all Splunk instances communicating with each other in the entire Splunk deployment. Send out the app with the web.conf setting from the DS to the UFs with the setting to restart and then add the web.conf setting to the DS and restart it. They will all be talking on the port that you specified for management communication including DS checkins and app pushes.
These points might help
1) The deployment server doesn't push the configurations to deployment client (forwarders here), but they connect with Deployment server and pull the new/updated configuration.
2) You can update your UF to have new port, just ensure that communication from UF server to Deployment server on Deployment server's management port is still available, you'll be just fine.
3) Since, The port for Deployment server is not changing, other UF will do just fine.