I can't browser Splunk Apps , alarm "The splunkd daemon cannot be reached by splunkweb"

[ohnewguy]

hi, Dears:

I installed splunk enterprise 6.2.3 on Ubuntu server 1404 with no GUI. After I remote accessed the splunk web page and click splunk apps for downloading app, the browser jumped to one page "http://<ip of the server installed Splunk>:8000/en-US/manager/search/apps/remote", and said :

503 Service Unavailable
Return to Splunk home page
The splunkd daemon cannot be reached by splunkweb. Check that there are no blocked network ports or that splunkd is still running.
View more information about your request (request ID = 55616670e27f5e10785610) in Search

I checked all configuration:

1. DNS is working
2. I don't install firewall on Ubuntu server
3. I try to access https://<server-ip>:8089 from my laptop , it is accessable.
4. I checked the splunkd process , it is running user1@Securitylab-opensoc:/opt/splunk/bin$ ./splunk status splunkd is running (PID: 1648). splunk helpers are running (PIDs: 1649 1662 1696 1734).

how can I solve it ??

[achurch_splunk]

I ran into this issue when authenticating connection (s) from the Deployment server and/or Search Head to the Indexers. While logged into Splunk Web, as Admin, I went to Settings>Distributed Search>Search Peers and it was stating...

"503 service unavailable: The splunkd daemon cannot be reached by splunkweb. Check that there are no blocked network ports or that splunkd is still running."

The error message itself threw me off, immediately thinking it was something to do with IPTABLES. I check that and my configs were fine.

The issue was ultimately a Roles issue under the Admin account. I attempted to go into SETTINGS>ACCESS CONTROLS>ROLES>select Admin, and verified my admin user account had the appropriate capabilities, and the account did NOT.I noticed, under 'available capabilities' that 'restart_splunkd', among other admin roles I needed, we not in the 'selected capabilities' list. After trying to add the 'restart_splunkd', I would restart and it would state that the user I was logged in as, which was Admin, didn't have the rights to make the change. So I went to the command line on the Deployment Server.

Go to $SPLUNK_HOME/etc/system/local. View/edit the authorize.conf. In there, I discovered that under the 'role_admin' stanza, there were quite a few capabilities that were disabled, restart_splunkd being one of them. Once I enabled those permissions and saved, chown -R user:group /opt/splunk, chmod -R o-rwx /opt/splunk, /opt/splunk/bin/splunk restart.....everything was functioning appropriately.

You also might want to check your configurations under /opt/splunk/etc/deployment-apps/config_search/local/authorize.conf

Hope this helps.

[t9445]

thanks! -- for us the issue was that we needed to enable "edit_index_cluster" for our LDAP based admin group (splunk v6.5.x)

[ohnewguy]

I found there is some error log in splunk:
ERROR [55617e8e167f5e107955d0] decorators:420 - Splunkd daemon is not responding: ('Error connecting to /services/apps/remote/entries: The read operation timed out',)
Traceback (most recent call last):
File "/opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/lib/decorators.py", line 406, in handle_exceptions
return fn(self, a, **kw)
File "/opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/controllers/admin.py", line 3194, in splunkbase_browser
apps, total_results = self._getRemoteEntries(*kwargs)
File "/opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/controllers/admin.py", line 3152, in _getRemoteEntries
entities = en.getEntities(url, **kwargs)
File "/opt/splunk/lib/python2.7/site-packages/splunk/entity.py", line 129, in getEntities
atomFeed = _getEntitiesAtomFeed(entityPath, namespace, owner, search, count, offset, sort_key, sort_dir, sessionKey, uri, hostPath, **kwargs)
File "/opt/splunk/lib/python2.7/site-packages/splunk/entity.py", line 222, in _getEntitiesAtomFeed
serverResponse, serverContent = rest.simpleRequest(uri, getargs=kwargs, sessionKey=sessionKey, raiseAllErrors=True)
......... omit
raise splunk.SplunkdConnectionException, 'Error connecting to %s: %s' % (path, str(e))
SplunkdConnectionException: Splunkd daemon is not responding: ('Error connecting to /services/apps/remote/entries: The read operation timed out',)

[neelamssantosh]

Make sure you are running splunk with splunk users and
before that from root user change files permission by
>chown -R splunk:splunk /opt/splunk/*

once done switch to splunk user
>su splunk

kill all the splunk and python services used by splunk,
>ps -ef|grep splunkd
>netstat -pan |grep python

>kill -9 <pid>
now restart the splunk services.

[ohnewguy]

Thanks for your kindly help, i follow your instruction to run it again , but it doesn't work. I am thinking that maybe I use a wrong linux version , because the splunk download page says the package is for Linux Kernel 2.6.x. But the kernel version of Ubuntu server 14.04 is 3.1.3.