We have a distributed environment of Splunk, where we forward data to indexer via heavy forwarders. Also we have a deployment server which control the operation as well as the changes done on all forwarders centrally.
We came to know one strange issue, we received a couple of new fresh boxes and when we install forwarders on it, after few seconds when it sync back to deployment server, we can't able to SSH to those servers from deployment servers.
So, whenever we start slunkd service on those server, SSH login gets disabled from deployment server. We need these connectivity as to push new config changes and managing purpose.
SSH login won't allow untill we kill the splunk forwarder service.
Let me know if anyone face the issue same ? Let me know if your need anything more here...
I would recommend always running Splunk as the Splunk User. This way Splunk is controlled by a completely different user than everything else in the system. Make sure to have all files and directory to be set to splunk for the user owner and splunk for the group owner.
This should make it so the Splunk service should not affect any other service on the system.