Unable to start splunk

[pramaswamy]

I am not able to restart my splunk and bring the app server up. I did read through similar posts in the forum to check, if a similar issue has been solved/helped before, and couldn't find one that is close enough to the problem I am facing.

Splunk was running perfectly fine for months, until I issued the command ** splunk restart **. Now, I am not able to bring it back on.

No errors reported directly in the command prompt, when executing *splunk start *. However, getting the following message in splunkd.log after a really long wait with the message "Waiting for web-server to be available..."

02-05-2018 21:24:30.692 -0500 ERROR UiPythonFallback - Appserver at http://127.0.0.1:8065 never started up!
02-05-2018 21:24:32.173 -0500 ERROR UiPythonFallback - Appserver running on port 8065 exited unexpectedly: exited with code 1
02-05-2018 21:24:32.173 -0500 ERROR UiHttpListener - An applicaiton server has exited unexpectedly, web UI cannot be used until it is restarted
02-05-2018 21:24:32.174 -0500 WARN  UiHttpListener - Web UI now stopped

Any ideas, clues or hint on what might be causing this or where I should be looking to understand further the issue ?

[christopherr_sp]

This issue occurred following an upgrade from Splunk 6.5.2 to 7.1.2.
This also included an upgrade of the Splunk Enterprise Security App to 5.1.0 which exhibited this error:

ERROR UiHttpListener - An applicaiton server has exited unexpectedly, web UI cannot be used until it is restarted

Follow the workaround steps in release notes.

http://docs.splunk.com/Documentation/ES/5.0.1/RN/KnownIssues

SOLNESS-14637
Splunk Web doesn't start after upgrading Splunk Enterprise Security to 5.0.0

Affects Enterprise Security: 5.0.0, 5.0.1, 5.1.0, 5.1.1.

Fixed in workaround below:

Known Issues for Splunk Enterprise Security
The following are issues and workarounds for this version of Splunk Enterprise Security.

Highlighted issues
Date filed Issue number Description
2018-02-20 SOLNESS-14637 Splunk Web doesn't start after upgrading Splunk Enterprise Security to 5.0.0

Workaround:
Remove Advanced XML module folder and contents from the installation.
For instance:

$SPLUNK_HOME/etc/apps/SA-Utils/appserver/modules/SOLNLookupEditor

[kearaspoor]

Is it only this one directory? or are there others that need to be identified/hunted down? The "for instance" is unhelpfully vague.

[christopherr_sp]

Yes $SPLUNK_HOME/etc/apps/SA-Utils/appserver/modules/SOLNLookupEditor is the only directory that needs to be deleted, then Splunk restarted. I can confirm this as I had encountered this twice since my original reply.

[MuS]

For me this was the only module that needed to be removed.

Thanks for this hint!

cheers, MuS

[pramaswamy · Accepted solution]

Thanks everyone for your time and inputs.

@klopez30 , that is something I was trying to avoid given I probably have some more work to bring my Splunk instance back to its original state, I have quite a few apps and indexes built.

Anyways, I tried "Repair" option, before clicking the last resort "uninstall" option and that fixed it !!!!!! Not sure what really went wrong, but things seem to be working now without issues.

[klopez30]

Have you tried re-installing the same version of Splunk, and seeing if the problem goes away. I would also check to see if any packages were updated, and maybe roll them back before trying to re-install Splunk.

[pramaswamy]

@somesoni2 @niketnilay , any ideas or thoughts that could help/guide me in this situation please ?

@richardwii, I see you have faced a similar issue in the past https://answers.splunk.com/answers/214712/why-am-i-getting-error-uipythonfallback-appserver.html

Are you able to share some of the ways you were able to get to the bottom of the problem?

[harsmarvania57]

I would suggest try to start splunk in DEBUG mode using command $SPLUNK_HOME/bin/splunk start --debug and check why appserver is not coming up, before you start splunk in DEBUG mode and if you are running linux then check whether any splunk stuck processes are running using command ps -ef | grep splunk, if there are any processes running kill those processed and then start splunk in DEBUG mode. Additionally check whether port 8065 is available or already binded with other process using command netstat -ntpul | grep 8065 before starting splunk.

[pramaswamy]

@mayurr98, tried to restart the machine. Didn't help unfortunately.

@harsmarvania57, I am running on windows machine and ---debug is not supported. However, I did change entries in log.cfg, save and tried to start. No additional insights unfortunately.

I did try find any "splunk" processes running , before starting the server and there were none. Yes, I did ensure that the port 8065 was available and no other process is using it. I got an OK message as well during splunk start command that the port was available, when it was checking the appserver port.

Thanks again for your thoughts and ideas. I just hope I am able to get to the bottom of what's happening here and fix it. Very weird and puzzling.

[mayurr98]

Try restarting the machine.

[493669]

Try :
splunk stop command first and then splunk start

[pramaswamy]

Thanks much @493669. I tried your suggestion, but still running into the same issue unfortunately. Any other ideas ?