topic After installing Splunk on Linux and I navigate to Apps in Splunk, why do I get "The splunkd daemon cannot be reached by splunkweb"? in Deployment Architecture

After installing Splunk on Linux and I navigate to Apps in Splunk, why do I get "The splunkd daemon cannot be reached by splunkweb"?

mlapilusa — Tue, 25 Aug 2015 00:23:00 GMT

I just installed Splunk on a Linux server and when I navigate to the Apps in Splunk, I get the following:

"503 Service Unavailable

Return to Splunk home page

The splunkd daemon cannot be reached by splunkweb. Check that there are no blocked network ports or that splunkd is still running.
View more information about your request (request ID = 55dbb3804a2aaaab06a2d0) in Search 


This page was linked to from http://xxxx:8000/en-US/app/launcher/home#en-US/app/launcher/home.

--------------------------------------------------------------------------------

You are logged into xxxxxxxx:8000 as admin, which is connected to splunkd @272645 at https://127.0.0.1:8089 on Mon Aug 24 17:15:26 2015."

Am I supposed to install and start a splunkweb component?

thanks,

Marcus

Re: After installing Splunk on Linux and I navigate to Apps in Splunk, why do I get "The splunkd daemon cannot be reached by splunkweb"?

MuS — Tue, 25 Aug 2015 01:48:11 GMT

Hi mlapilusa,

looks like your splunkd process either died or crashed and is no longer reachable for splunkweb. Both are installed by the install process and no need to install any additional thing.
Log on to your Splunk server and restart Splunk like this:

$SPLUNK_HOME/bin/splunk restart

Take a look at the docs http://docs.splunk.com/Documentation/Splunk/6.2.5/Installation/Splunksarchitectureandwhatgetsinstalled#Splunk_Enterprise_Processes to learn more about the Splunk processes.

Hope that helps ...

cheers, MuS

Re: After installing Splunk on Linux and I navigate to Apps in Splunk, why do I get "The splunkd daemon cannot be reached by splunkweb"?

mlapilusa — Tue, 25 Aug 2015 21:24:18 GMT

I tried restarting Splunk and it still doesn't come up:
./splunk restart
Stopping splunkd...
Shutting down. Please wait, as this may take a few minutes.
.. [ OK ]
Stopping splunk helpers...
[ OK ]
Done.

Splunk> Now with more code!

Checking prerequisites...
Checking http port [8000]: open
Checking mgmt port [8089]: open
Checking appserver port [127.0.0.1:8065]: open
Checking kvstore port [8191]: open
Checking configuration... Done.
Checking critical directories... Done
Checking indexes...
Validated: _audit _blocksignature _internal _introspection _thefishbucket history main summary
Done
Checking filesystem compatibility... Done
Checking conf files for problems...
Done
All preliminary checks passed.

Starting splunk server daemon (splunkd)...
Done
[ OK ]

Waiting for web server at http://127.0.0.1:8000 to be available... Done

If you get stuck, we're here to help.
Look for answers here: http://docs.splunk.com

The Splunk web interface is at http://edhlsbaod001.otprod.dstcorp.net:8000

[root@edhlsbaod001 bin]# ps -ef | grep splunkweb
root 23530 23135 0 14:22 pts/1 00:00:00 grep splunkweb
[root@edhlsbaod001 bin]# ps -ef | grep splun
root 23268 23190 0 14:21 pts/2 00:00:00 su - splunk
splunk 23270 23268 0 14:21 pts/2 00:00:00 -bash
splunk 23322 1 2 14:21 ? 00:00:01 splunkd -p 8089 restart
splunk 23323 23322 0 14:21 ? 00:00:00 [splunkd pid=23322] splunkd -p 8089 restart [process-runner]
splunk 23339 23323 0 14:21 ? 00:00:00 mongod --dbpath=/opt/splunk/var/lib/splunk/kvstore/mongo --port=8191 --timeStampFormat=iso8601-utc --smallfiles --oplogSize=1000 --keyFile=/opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key --setParameter=enableLocalhostAuthBypass=0 --sslMode=preferSSL --sslPEMKeyFile=/opt/splunk/etc/auth/server.pem --sslPEMKeyPassword=xxxxxxxx
splunk 23389 23323 1 14:21 ? 00:00:01 /opt/splunk/bin/python -O /opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/root.py --proxied=127.0.0.1,8065,8000
splunk 23439 23323 0 14:21 ? 00:00:00 /opt/splunk/bin/splunkd instrument-resource-usage -p 8089
splunk 23468 23270 0 14:21 pts/2 00:00:00 tail -f splunkd.log
root 23533 23135 0 14:22 pts/1 00:00:00 grep splun

Re: After installing Splunk on Linux and I navigate to Apps in Splunk, why do I get "The splunkd daemon cannot be reached by splunkweb"?

mlapilusa — Tue, 25 Aug 2015 21:29:01 GMT

I tried restarting and that didn't resolve the issue. I see this in the logs though so I think this may be related to a proxy issue from the server that is running Splunk trying to get out to the Splunk servers.

08-25-2015 14:24:53.436 -0700 INFO NetUtils - Connect timeout - waited for 10 seconds. ip=54.218.112.140 port=443
08-25-2015 14:25:03.436 -0700 INFO NetUtils - Connect timeout - waited for 10 seconds. ip=54.186.70.15 port=443
08-25-2015 14:25:13.436 -0700 INFO NetUtils - Connect timeout - waited for 10 seconds. ip=54.148.255.243 port=443
08-25-2015 14:25:13.436 -0700 WARN HTTPClient - Connect to=apps.splunk.com:443 timed out; exceeded 10sec
08-25-2015 14:25:13.436 -0700 WARN HTTPClient - Connect to=apps.splunk.com:443 timed out; exceeded 10sec
08-25-2015 14:25:13.436 -0700 ERROR ApplicationUpdater - Error checking for update, URL=/api/apps:resolve/checkforupgrade: Connect to=https://apps.splunk.com timed out; exceeded 10sec
08-25-2015 14:25:36.840 -0700 WARN HttpListener - Socket error from 127.0.0.1 while accessing /services/apps/remote/entries: Broken pipe

Re: After installing Splunk on Linux and I navigate to Apps in Splunk, why do I get "The splunkd daemon cannot be reached by splunkweb"?

swatghare — Thu, 27 Jul 2017 10:16:17 GMT

Hello
did you got this issue fixed? what was the resolution?