Why does UF 9.1 doesn't listen on port 8089 (default)?

[tvt0991]

I downloaded splunkforwarder-9.1.0.1-77f73c9edb85-Linux-x86_64.tgz, untared it on a fresh Ubuntu 22.04, ran ./splunk start (also tried ./splunk start --debug) then accepted license, set password....

lsof -i -P -n | grep 8089 command doesn't show anything

But ps command show UF is running:
user    8267  0.2  0.1 403216 98920 ?        Sl   14:00   0:10 splunkd -p 8089 restart --debug
user    8268  0.0  0.0 120792 15068 ?        Ss   14:00   0:00 [splunkd pid=8267] splunkd -p 8089 restart --debug [process-runner]

I tried the same steps on 9.0.5.  It worked:
splunkd    462339        user    4u  IPv4 54062151      0t0  TCP 127.0.0.1:8089 (LISTEN)

How do I get UF 9.1 to listen on port 8089?  Or UF 9.1 doesn't work the same way?  Thanks

[tvt0991]

Thanks @isoutamo  and @fatsug for your leads.  What I understand from Splunk documentation (link from @fatsug above) is that splunkd should listen to 127.0.0.1:8089 by default:

• Improvements to universal forwarder security, including the limiting of access to the UF management port to only the local machine

But as mentioned above, lsof command showed splunkd doesn't listen on to localhost,127.0.0.1:8089.  Even though when you list running processes, it shows: splunkd -p 8089.  Also, when starting it indicated in the console: Checking mgmt port [8089]: open

After more reading, it looks like mgmtMode in 9.1.0.1 doesn't work as previous version including 9.0.  To enable it, I have to force it in tcp mode.  In our case, we need to enable UF management port for REST API connection.  To enable it we added this block to $SPLUNK_HOME/etc/system/local/server.conf:

[httpServer]
mgmtMode = tcp

Depends on your setting, you might want to add "disableDefaultPort = false" to make sure it's not configured somewhere else.

What're your thoughts on this approach?

[isoutamo]

In linux UF 9.1.x+ didn't listen any TCP port by default as linux support UDS. It's disabled. See this

mgmtMode = none|auto|tcp
* Sets the transport layer protocol mode for Splunk CLI management commands.
* A value of "none" means that only Splunk CLI commands that can be run
  offline are available for use on the instance.
* A value of "auto" means that CLI commands execute over a Unix Domain Socket (UDS),
  which represents as $SPLUNK_HOME/var/run/splunk/cli.socket on the file system.
  * If the OS does not support UDS, and if 'disableDefaultPort' has a value of "false",
    the CLI executes over the splunkd management port.
  * If 'disableDefaultPort' has a value of "true", only CLI commands that can be
    run offline are available for use on the instance.
* A value of "tcp" means the CLI commands execute over the splunkd
  management port.
* This setting is only available on the universal forwarder.
* NOTE: There is a path length limit of 104-108 characters for the UDS socket file. This includes
  whatever the $SPLUNK_HOME environment variable expands to. If you exceed this length for the
  socket file, UDS does not work, and you must reinstall the universal forwarder, because you cannot
  take corrective action to fix the UDS path. Verify the UDS path length when configuring this setting.
* Default: auto

You could check it by looking splunkd process and then 

[root@infra01] /home/soutamo>
(0) # lsof -p 2952730|egrep socket
splunkd 2952730 splunk    4u     unix 0xffff9d76b64f6200      0t0  29490687 /opt/splunkforwarder/var/run/splunk/cli.socket type=STREAM (LISTEN)

If you really need remote management for that, then just add mgmtPort on web.conf and this should do it. I'm not sure if you need also that mgmtMode = tcp parameter or not.

There seems to be some other ACL etc types parameters which you can use to restrict access to splunkd. And of course you have host based firewall and SElinux on place and in use?

[tvt0991]

Agree with you on that's what the doc says but in testing (version: splunkforwarder-9.1.0.1-77f73c9edb85-Linux-x86_64.tgz), mgmtHostPort in web.conf alone doesn't work for me.  I have to add mgmtMode = tcp for it to work.  Also, the default web.conf file already has mgmtHostPort = localhost:8089.

mgmtHostPort has been working up to and including version 9.0.5 but not on 9.1.0.1.  On our instance, we only need to enable that port for localhost only.

[isoutamo]

Hi

If I recall right this has changed with 9.1 and management port is now disabled by default. Usually there is no need for that and if it is needed you should allow it only from localhost not from net.

See web.conf for doing that.

r. Ismo

[fatsug]

Seems that way:

"Improvements to universal forwarder security, including the limiting of access to the UF management port to only the local machine"

https://docs.splunk.com/Documentation/Splunk/9.1.0/Security/Updates

Don't remeber which version update this started, though alreade before 9.0 there were a number of security issues fixed. At that time there was a recommendation to disable management port on UF.