I downloaded splunkforwarder-9.1.0.1-77f73c9edb85-Linux-x86_64.tgz, untared it on a fresh Ubuntu 22.04, ran ./splunk start (also tried ./splunk start --debug) then accepted license, set password....
lsof -i -P -n | grep 8089 command doesn't show anything
But ps command show UF is running:
user 8267 0.2 0.1 403216 98920 ? Sl 14:00 0:10 splunkd -p 8089 restart --debug
user 8268 0.0 0.0 120792 15068 ? Ss 14:00 0:00 [splunkd pid=8267] splunkd -p 8089 restart --debug [process-runner]
I tried the same steps on 9.0.5. It worked:
splunkd 462339 user 4u IPv4 54062151 0t0 TCP 127.0.0.1:8089 (LISTEN)
How do I get UF 9.1 to listen on port 8089? Or UF 9.1 doesn't work the same way? Thanks
Thanks @isoutamo and @fatsug for your leads. What I understand from Splunk documentation (link from @fatsug above) is that splunkd should listen to 127.0.0.1:8089 by default:
But as mentioned above, lsof command showed splunkd doesn't listen on to localhost,127.0.0.1:8089. Even though when you list running processes, it shows: splunkd -p 8089. Also, when starting it indicated in the console: Checking mgmt port [8089]: open
After more reading, it looks like mgmtMode in 9.1.0.1 doesn't work as previous version including 9.0. To enable it, I have to force it in tcp mode. In our case, we need to enable UF management port for REST API connection. To enable it we added this block to $SPLUNK_HOME/etc/system/local/server.conf:
[httpServer]
mgmtMode = tcp
Depends on your setting, you might want to add "disableDefaultPort = false" to make sure it's not configured somewhere else.
What're your thoughts on this approach?
In linux UF 9.1.x+ didn't listen any TCP port by default as linux support UDS. It's disabled. See this
mgmtMode = none|auto|tcp
* Sets the transport layer protocol mode for Splunk CLI management commands.
* A value of "none" means that only Splunk CLI commands that can be run
offline are available for use on the instance.
* A value of "auto" means that CLI commands execute over a Unix Domain Socket (UDS),
which represents as $SPLUNK_HOME/var/run/splunk/cli.socket on the file system.
* If the OS does not support UDS, and if 'disableDefaultPort' has a value of "false",
the CLI executes over the splunkd management port.
* If 'disableDefaultPort' has a value of "true", only CLI commands that can be
run offline are available for use on the instance.
* A value of "tcp" means the CLI commands execute over the splunkd
management port.
* This setting is only available on the universal forwarder.
* NOTE: There is a path length limit of 104-108 characters for the UDS socket file. This includes
whatever the $SPLUNK_HOME environment variable expands to. If you exceed this length for the
socket file, UDS does not work, and you must reinstall the universal forwarder, because you cannot
take corrective action to fix the UDS path. Verify the UDS path length when configuring this setting.
* Default: auto
You could check it by looking splunkd process and then
[root@infra01] /home/soutamo>
(0) # lsof -p 2952730|egrep socket
splunkd 2952730 splunk 4u unix 0xffff9d76b64f6200 0t0 29490687 /opt/splunkforwarder/var/run/splunk/cli.socket type=STREAM (LISTEN)
If you really need remote management for that, then just add mgmtPort on web.conf and this should do it. I'm not sure if you need also that mgmtMode = tcp parameter or not.
There seems to be some other ACL etc types parameters which you can use to restrict access to splunkd. And of course you have host based firewall and SElinux on place and in use?
Agree with you on that's what the doc says but in testing (version: splunkforwarder-9.1.0.1-77f73c9edb85-Linux-x86_64.tgz), mgmtHostPort in web.conf alone doesn't work for me. I have to add mgmtMode = tcp for it to work. Also, the default web.conf file already has mgmtHostPort = localhost:8089.
mgmtHostPort has been working up to and including version 9.0.5 but not on 9.1.0.1. On our instance, we only need to enable that port for localhost only.
Hi
If I recall right this has changed with 9.1 and management port is now disabled by default. Usually there is no need for that and if it is needed you should allow it only from localhost not from net.
See web.conf for doing that.
r. Ismo
Seems that way:
"Improvements to universal forwarder security, including the limiting of access to the UF management port to only the local machine"
https://docs.splunk.com/Documentation/Splunk/9.1.0/Security/Updates
Don't remeber which version update this started, though alreade before 9.0 there were a number of security issues fixed. At that time there was a recommendation to disable management port on UF.