UF not installing apps with error - "splunkd is down" when it is up and running

[sylim_splunk]

I am completing the process of installing the credentials package to our universal forwarders to send data to the cloud like we did with our application service servers and I ran into an issue on the majority of the machines where I would get the following error:

/opt/splunkforwarder/bin/./splunk install app /tmp/splunkclouduf.spl
This command [POST /services/apps/local/] needs splunkd to be up, and splunkd is down.

Splunkd was definitely running. I even restarted it for good measure. Thirty percent of the machines executed the command fine, prompted me for UF credentials, and confirmed the installation was completed. Is this something you’ve run into before?

[sylim_splunk · Accepted solution]

It turned out that some UFs have mgmt port disabled due to the port scanning.  This has been worked around by enabling the port.

[httpServer]
disableDefaultPort = false (to enable mgmt)

There's a better solution for the annoying port scanner - acceptFrom = 127.0.0.1 - we deleted "disableDefaultPort = true" from "[httpServer]" stanza and added "acceptFrom" so that it only accepts connections from localhost. And we can issue splunk commands on the servers.