Deployment Architecture

Why pushing bundled apps hangs the splunkd service with error "No appservers running"?

Path Finder

I'm trying to build a clustered index in a lab environment between 3 servers (1 master, 2 slaves). The indexer clustering appears to have worked correctly. In the distributed environment/indexer clustering dashboard, both slave indexers are showing as peers.

I placed 3 applications in the master-apps directory and attempted force replication through the CLI but was given an error:

“/root/.splunk”:permission denied

Out of curiosity, I then attempted to push the apps through UI (edit, distribute configuration bundle), and this was successful, but I was then unable to access either slave indexer through port 8000. I get a "No appservers running, server had an unexpected error" message. I'm able to ssh into the servers, but when I stopped splunk (../splunk stop) and then restarted it (/etc/init.d/splunk start), the service took several minutes to start and the end result was the "No appservers running" message mentioned above.

0 Karma
1 Solution

Path Finder

To follow up, this issue appears to be related to the splunkappfor_nix.
On a rebuild of my master and 2 peers I took the following actions:

1) Connected the 3 servers as an indexer cluster
2) Created a custom app to push new indexes (that would be created by the apps I intend to upload) with the repFactor = auto switch added to each stanza.
3) Pushed the app bundle -success
4) Uploaded the SplunkTAwindows app, and pushed it -success
5) Uploaded the TA-SOS app, and pushed it -success
6) Uploaded the splunkappfornix, and pushed it -Fail, system goes into a loop and I get the "No App Server" message when trying to access the UI.
7) Turn splunkd off on all servers, delete the splunk
appfornix, restart the splunkd service and everything goes back to normal.

View solution in original post

0 Karma

Path Finder

To follow up, this issue appears to be related to the splunkappfor_nix.
On a rebuild of my master and 2 peers I took the following actions:

1) Connected the 3 servers as an indexer cluster
2) Created a custom app to push new indexes (that would be created by the apps I intend to upload) with the repFactor = auto switch added to each stanza.
3) Pushed the app bundle -success
4) Uploaded the SplunkTAwindows app, and pushed it -success
5) Uploaded the TA-SOS app, and pushed it -success
6) Uploaded the splunkappfornix, and pushed it -Fail, system goes into a loop and I get the "No App Server" message when trying to access the UI.
7) Turn splunkd off on all servers, delete the splunk
appfornix, restart the splunkd service and everything goes back to normal.

View solution in original post

0 Karma

Communicator

I have similar issue on Splunk running on Windows. Don't know which log to look into to troubleshoot the problem. Look forward to leaning how to resolve this "No appservers running" issue.

0 Karma