Deployment Architecture

Why pushing bundled apps hangs the splunkd service with error "No appservers running"?

jsmith39
Path Finder

I'm trying to build a clustered index in a lab environment between 3 servers (1 master, 2 slaves). The indexer clustering appears to have worked correctly. In the distributed environment/indexer clustering dashboard, both slave indexers are showing as peers.

I placed 3 applications in the master-apps directory and attempted force replication through the CLI but was given an error:

“/root/.splunk”:permission denied

Out of curiosity, I then attempted to push the apps through UI (edit, distribute configuration bundle), and this was successful, but I was then unable to access either slave indexer through port 8000. I get a "No appservers running, server had an unexpected error" message. I'm able to ssh into the servers, but when I stopped splunk (../splunk stop) and then restarted it (/etc/init.d/splunk start), the service took several minutes to start and the end result was the "No appservers running" message mentioned above.

0 Karma
1 Solution

jsmith39
Path Finder

To follow up, this issue appears to be related to the splunk_app_for_nix.
On a rebuild of my master and 2 peers I took the following actions:

1) Connected the 3 servers as an indexer cluster
2) Created a custom app to push new indexes (that would be created by the apps I intend to upload) with the repFactor = auto switch added to each stanza.
3) Pushed the app bundle -success
4) Uploaded the Splunk_TA_windows app, and pushed it -success
5) Uploaded the TA-SOS app, and pushed it -success
6) Uploaded the splunk_app_for_nix, and pushed it -Fail, system goes into a loop and I get the "No App Server" message when trying to access the UI.
7) Turn splunkd off on all servers, delete the splunk_app_for_nix, restart the splunkd service and everything goes back to normal.

View solution in original post

0 Karma

jsmith39
Path Finder

To follow up, this issue appears to be related to the splunk_app_for_nix.
On a rebuild of my master and 2 peers I took the following actions:

1) Connected the 3 servers as an indexer cluster
2) Created a custom app to push new indexes (that would be created by the apps I intend to upload) with the repFactor = auto switch added to each stanza.
3) Pushed the app bundle -success
4) Uploaded the Splunk_TA_windows app, and pushed it -success
5) Uploaded the TA-SOS app, and pushed it -success
6) Uploaded the splunk_app_for_nix, and pushed it -Fail, system goes into a loop and I get the "No App Server" message when trying to access the UI.
7) Turn splunkd off on all servers, delete the splunk_app_for_nix, restart the splunkd service and everything goes back to normal.

0 Karma

napomokoetle
Communicator

I have similar issue on Splunk running on Windows. Don't know which log to look into to troubleshoot the problem. Look forward to leaning how to resolve this "No appservers running" issue.

0 Karma
Get Updates on the Splunk Community!

Check out this month’s brand new Splunk Lantern articles

Splunk Lantern is a customer success center providing advice from Splunk experts on valuable data insights, ...

Happy CX Day to our Community Superheroes!

Happy 10th Birthday CX Day!What is CX Day? It’s a global celebration recognizing innovation and success in the ...

Routing Data to Different Splunk Indexes in the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...