I installed splunk enterprise 6.2.3 on Ubuntu server 1404 with no GUI. After I remote accessed the splunk web page and click splunk apps for downloading app, the browser jumped to one page "http://<ip of the server installed Splunk>:8000/en-US/manager/search/apps/remote", and said :
503 Service Unavailable
Return to Splunk home page
The splunkd daemon cannot be reached by splunkweb. Check that there are no blocked network ports or that splunkd is still running.
View more information about your request (request ID = 55616670e27f5e10785610) in Search
I checked all configuration:
how can I solve it ??
I found there is some error log in splunk:
ERROR [55617e8e167f5e107955d0] decorators:420 - Splunkd daemon is not responding: ('Error connecting to /services/apps/remote/entries: The read operation timed out',)
Traceback (most recent call last):
File "/opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/lib/decorators.py", line 406, in handleexceptions
return fn(self, a, *kw)
File "/opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/controllers/admin.py", line 3194, in splunkbasebrowser
apps, totalresults = self.getRemoteEntries(*kwargs)
File "/opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/controllers/admin.py", line 3152, in _getRemoteEntries
entities = en.getEntities(url, *kwargs)
File "/opt/splunk/lib/python2.7/site-packages/splunk/entity.py", line 129, in getEntities
atomFeed = getEntitiesAtomFeed(entityPath, namespace, owner, search, count, offset, sortkey, sort_dir, sessionKey, uri, hostPath, **kwargs)
File "/opt/splunk/lib/python2.7/site-packages/splunk/entity.py", line 222, in _getEntitiesAtomFeed
serverResponse, serverContent = rest.simpleRequest(uri, getargs=kwargs, sessionKey=sessionKey, raiseAllErrors=True)
raise splunk.SplunkdConnectionException, 'Error connecting to %s: %s' % (path, str(e))
SplunkdConnectionException: Splunkd daemon is not responding: ('Error connecting to /services/apps/remote/entries: The read operation timed out',)
Make sure you are running splunk with splunk users and
before that from root user change files permission by
>chown -R splunk:splunk /opt/splunk/*
once done switch to splunk user
kill all the splunk and python services used by splunk,
>ps -ef|grep splunkd
>netstat -pan |grep python
>kill -9 <pid>
now restart the splunk services.
Thanks for your kindly help, i follow your instruction to run it again , but it doesn't work. I am thinking that maybe I use a wrong linux version , because the splunk download page says the package is for Linux Kernel 2.6.x. But the kernel version of Ubuntu server 14.04 is 3.1.3.
I ran into this issue when authenticating connection (s) from the Deployment server and/or Search Head to the Indexers. While logged into Splunk Web, as Admin, I went to Settings>Distributed Search>Search Peers and it was stating...
"503 service unavailable: The splunkd daemon cannot be reached by splunkweb. Check that there are no blocked network ports or that splunkd is still running."
The error message itself threw me off, immediately thinking it was something to do with IPTABLES. I check that and my configs were fine.
The issue was ultimately a Roles issue under the Admin account. I attempted to go into SETTINGS>ACCESS CONTROLS>ROLES>select Admin, and verified my admin user account had the appropriate capabilities, and the account did NOT.I noticed, under 'available capabilities' that 'restartsplunkd', among other admin roles I needed, we not in the 'selected capabilities' list. After trying to add the 'restartsplunkd', I would restart and it would state that the user I was logged in as, which was Admin, didn't have the rights to make the change. So I went to the command line on the Deployment Server.
Go to $SPLUNKHOME/etc/system/local. View/edit the authorize.conf. In there, I discovered that under the 'roleadmin' stanza, there were quite a few capabilities that were disabled, restart_splunkd being one of them. Once I enabled those permissions and saved, chown -R user:group /opt/splunk, chmod -R o-rwx /opt/splunk, /opt/splunk/bin/splunk restart.....everything was functioning appropriately.
You also might want to check your configurations under /opt/splunk/etc/deployment-apps/config_search/local/authorize.conf
Hope this helps.
thanks! -- for us the issue was that we needed to enable "editindexcluster" for our LDAP based admin group (splunk v6.5.x)