Splunk Enterprise

Unable to start splunk

pramaswamy
Path Finder

I am not able to restart my splunk and bring the app server up. I did read through similar posts in the forum to check, if a similar issue has been solved/helped before, and couldn't find one that is close enough to the problem I am facing.

Splunk was running perfectly fine for months, until I issued the command ** splunk restart **. Now, I am not able to bring it back on.

No errors reported directly in the command prompt, when executing *splunk start *. However, getting the following message in splunkd.log after a really long wait with the message "Waiting for web-server to be available..."

02-05-2018 21:24:30.692 -0500 ERROR UiPythonFallback - Appserver at http://127.0.0.1:8065 never started up!
02-05-2018 21:24:32.173 -0500 ERROR UiPythonFallback - Appserver running on port 8065 exited unexpectedly: exited with code 1
02-05-2018 21:24:32.173 -0500 ERROR UiHttpListener - An applicaiton server has exited unexpectedly, web UI cannot be used until it is restarted
02-05-2018 21:24:32.174 -0500 WARN  UiHttpListener - Web UI now stopped

Any ideas, clues or hint on what might be causing this or where I should be looking to understand further the issue ?

Tags (1)
0 Karma
1 Solution

pramaswamy
Path Finder

Thanks everyone for your time and inputs.

@klopez30 , that is something I was trying to avoid given I probably have some more work to bring my Splunk instance back to its original state, I have quite a few apps and indexes built.

Anyways, I tried "Repair" option, before clicking the last resort "uninstall" option and that fixed it !!!!!! Not sure what really went wrong, but things seem to be working now without issues.

View solution in original post

christopherr_sp
Splunk Employee
Splunk Employee

This issue occurred following an upgrade from Splunk 6.5.2 to 7.1.2.
This also included an upgrade of the Splunk Enterprise Security App to 5.1.0 which exhibited this error:

ERROR UiHttpListener - An applicaiton server has exited unexpectedly, web UI cannot be used until it is restarted

Follow the workaround steps in release notes.

http://docs.splunk.com/Documentation/ES/5.0.1/RN/KnownIssues

SOLNESS-14637
Splunk Web doesn't start after upgrading Splunk Enterprise Security to 5.0.0

Affects Enterprise Security: 5.0.0, 5.0.1, 5.1.0, 5.1.1.

Fixed in workaround below:

Known Issues for Splunk Enterprise Security
The following are issues and workarounds for this version of Splunk Enterprise Security.

Highlighted issues
Date filed Issue number Description
2018-02-20 SOLNESS-14637 Splunk Web doesn't start after upgrading Splunk Enterprise Security to 5.0.0

Workaround:
Remove Advanced XML module folder and contents from the installation.
For instance:

$SPLUNK_HOME/etc/apps/SA-Utils/appserver/modules/SOLNLookupEditor

kearaspoor
SplunkTrust
SplunkTrust

Is it only this one directory? or are there others that need to be identified/hunted down? The "for instance" is unhelpfully vague.

0 Karma

christopherr_sp
Splunk Employee
Splunk Employee

Yes $SPLUNK_HOME/etc/apps/SA-Utils/appserver/modules/SOLNLookupEditor is the only directory that needs to be deleted, then Splunk restarted. I can confirm this as I had encountered this twice since my original reply.

0 Karma

MuS
SplunkTrust
SplunkTrust

For me this was the only module that needed to be removed.

Thanks for this hint!

cheers, MuS

0 Karma

pramaswamy
Path Finder

Thanks everyone for your time and inputs.

@klopez30 , that is something I was trying to avoid given I probably have some more work to bring my Splunk instance back to its original state, I have quite a few apps and indexes built.

Anyways, I tried "Repair" option, before clicking the last resort "uninstall" option and that fixed it !!!!!! Not sure what really went wrong, but things seem to be working now without issues.

klopez30
Explorer

Have you tried re-installing the same version of Splunk, and seeing if the problem goes away. I would also check to see if any packages were updated, and maybe roll them back before trying to re-install Splunk.

0 Karma

pramaswamy
Path Finder

@somesoni2 @niketnilay , any ideas or thoughts that could help/guide me in this situation please ?

@richardwii, I see you have faced a similar issue in the past https://answers.splunk.com/answers/214712/why-am-i-getting-error-uipythonfallback-appserver.html

Are you able to share some of the ways you were able to get to the bottom of the problem?

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

I would suggest try to start splunk in DEBUG mode using command $SPLUNK_HOME/bin/splunk start --debug and check why appserver is not coming up, before you start splunk in DEBUG mode and if you are running linux then check whether any splunk stuck processes are running using command ps -ef | grep splunk, if there are any processes running kill those processed and then start splunk in DEBUG mode. Additionally check whether port 8065 is available or already binded with other process using command netstat -ntpul | grep 8065 before starting splunk.

pramaswamy
Path Finder

@mayurr98, tried to restart the machine. Didn't help unfortunately.

@harsmarvania57, I am running on windows machine and ---debug is not supported. However, I did change entries in log.cfg, save and tried to start. No additional insights unfortunately.

I did try find any "splunk" processes running , before starting the server and there were none. Yes, I did ensure that the port 8065 was available and no other process is using it. I got an OK message as well during splunk start command that the port was available, when it was checking the appserver port.

Thanks again for your thoughts and ideas. I just hope I am able to get to the bottom of what's happening here and fix it. Very weird and puzzling.

0 Karma

mayurr98
Super Champion

Try restarting the machine.

0 Karma

493669
Super Champion

Try :
splunk stop command first and then splunk start

0 Karma

pramaswamy
Path Finder

Thanks much @493669. I tried your suggestion, but still running into the same issue unfortunately. Any other ideas ?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...