Deployment Architecture

Why pushing bundled apps hangs the splunkd service with error "No appservers running"?

jsmith39
Path Finder

I'm trying to build a clustered index in a lab environment between 3 servers (1 master, 2 slaves). The indexer clustering appears to have worked correctly. In the distributed environment/indexer clustering dashboard, both slave indexers are showing as peers.

I placed 3 applications in the master-apps directory and attempted force replication through the CLI but was given an error:

“/root/.splunk”:permission denied

Out of curiosity, I then attempted to push the apps through UI (edit, distribute configuration bundle), and this was successful, but I was then unable to access either slave indexer through port 8000. I get a "No appservers running, server had an unexpected error" message. I'm able to ssh into the servers, but when I stopped splunk (../splunk stop) and then restarted it (/etc/init.d/splunk start), the service took several minutes to start and the end result was the "No appservers running" message mentioned above.

0 Karma
1 Solution

jsmith39
Path Finder

To follow up, this issue appears to be related to the splunk_app_for_nix.
On a rebuild of my master and 2 peers I took the following actions:

1) Connected the 3 servers as an indexer cluster
2) Created a custom app to push new indexes (that would be created by the apps I intend to upload) with the repFactor = auto switch added to each stanza.
3) Pushed the app bundle -success
4) Uploaded the Splunk_TA_windows app, and pushed it -success
5) Uploaded the TA-SOS app, and pushed it -success
6) Uploaded the splunk_app_for_nix, and pushed it -Fail, system goes into a loop and I get the "No App Server" message when trying to access the UI.
7) Turn splunkd off on all servers, delete the splunk_app_for_nix, restart the splunkd service and everything goes back to normal.

View solution in original post

0 Karma

jsmith39
Path Finder

To follow up, this issue appears to be related to the splunk_app_for_nix.
On a rebuild of my master and 2 peers I took the following actions:

1) Connected the 3 servers as an indexer cluster
2) Created a custom app to push new indexes (that would be created by the apps I intend to upload) with the repFactor = auto switch added to each stanza.
3) Pushed the app bundle -success
4) Uploaded the Splunk_TA_windows app, and pushed it -success
5) Uploaded the TA-SOS app, and pushed it -success
6) Uploaded the splunk_app_for_nix, and pushed it -Fail, system goes into a loop and I get the "No App Server" message when trying to access the UI.
7) Turn splunkd off on all servers, delete the splunk_app_for_nix, restart the splunkd service and everything goes back to normal.

0 Karma

napomokoetle
Communicator

I have similar issue on Splunk running on Windows. Don't know which log to look into to troubleshoot the problem. Look forward to leaning how to resolve this "No appservers running" issue.

0 Karma
Get Updates on the Splunk Community!

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...

Ready, Set, SOAR: How Utility Apps Can Up Level Your Playbooks!

 WATCH NOW Powering your capabilities has never been so easy with ready-made Splunk® SOAR Utility Apps. Parse ...