I ran into this issue when authenticating connection (s) from the Deployment server and/or Search Head to the Indexers. While logged into Splunk Web, as Admin, I went to Settings>Distributed Search>Search Peers and it was stating... "503 service unavailable: The splunkd daemon cannot be reached by splunkweb. Check that there are no blocked network ports or that splunkd is still running." The error message itself threw me off, immediately thinking it was something to do with IPTABLES. I check that and my configs were fine. The issue was ultimately a Roles issue under the Admin account. I attempted to go into SETTINGS>ACCESS CONTROLS>ROLES>select Admin, and verified my admin user account had the appropriate capabilities, and the account did NOT.I noticed, under 'available capabilities' that 'restart_splunkd', among other admin roles I needed, we not in the 'selected capabilities' list. After trying to add the 'restart_splunkd', I would restart and it would state that the user I was logged in as, which was Admin, didn't have the rights to make the change. So I went to the command line on the Deployment Server. Go to $SPLUNK_HOME/etc/system/local. View/edit the authorize.conf. In there, I discovered that under the 'role_admin' stanza, there were quite a few capabilities that were disabled, restart_splunkd being one of them. Once I enabled those permissions and saved, chown -R user:group /opt/splunk, chmod -R o-rwx /opt/splunk, /opt/splunk/bin/splunk restart.....everything was functioning appropriately. You also might want to check your configurations under /opt/splunk/etc/deployment-apps/config_search/local/authorize.conf Hope this helps. ... View more