I have 2 Splunk instances. one as an indexer and search head and one just as a dedicated search head (on a VM just for testing and eventually moving to production). Everything was fine and working (except some permission issues which people couldn't save anything as private). So after awhile I was trying to log in and start investigating again but the GUI didn't come up and I found out that splunkweb service won't stay up, meaning I can't start the service but after like one second it stops! The only change that has been made recently is that the other instance which I was using as an indexer upgraded to 5.0.3 and the search head is still 5.0.2. So I tried to upgrade it but it gives this error "Splunk installer was unable to create Splunk Service"
Cheers guys this thread pointed me in the right direction and gave me enough direction to figure out what my issues was. FYI it was the Splunkd service overloading my Splunk cluster which in turn was stopping the service from starting on one of my indexers.
Thanks again, the log file was the big help 🙂
So I found my answer
In my search head event viewer I got this error"
File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\win32\win32serviceutil.py", line 785, in SvcRun self.SvcDoRun() File "C:\Program Files\Splunk\bin\SplunkWebService.py", line 40, in SvcDoRun root.run(blocking=False) File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\splunk\appserver\mrsparkle\root.py", line 539, in run i18n.init_js_cache(flush_files=True) File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\splunk\appserver\mrsparkle\lib\i18n.py", line 741, in init_js_cache os.unlink(os.path.join(CACHE_PATH, fn))
(32, 'The process cannot access the file because it is being used by another process', 'C:\Program Files\Splunk\var\run\splunk\appserver\i18n\version')
So I checked the "version" folder it was empty I just deleted the folder and started the splunkweb service again and it was regenerated and now it's working yeyyy
It should have turn on all logging channels. Splunkwebs logs are not in splunkd.log but in web_service.log. I would use the btool and do a config dump. Normally when this happens to me its when I've made a setting change to web.conf or server.conf which splunkweb does not like. Another possibility is that permission inheritance was broken (done this while converting from running services as domain account and move to LOCAL\NTAUTHORITY account..
It's running as a local system under my domain account.it was ldap integrated before and it was running fine. I did the --debug and both services (splunkd and splunkweb starts and "Done") but when i check the splunkweb it's stopped!
Your first port of call for answers would be a couple of Splunk's log files splunkd.log and web_service.log (in $SPLUNK_HOME/var/log/splunk/). You might find helpful information in there.
You don't really provide enough information to start speculating on the causes of your problem, and you don't tell us what o/s you are running on, but are you sure that a) you don't have a shortage of disk space or swap, or b) that you have not been altering file permissions?
Did you check the service under which account it's running? The folders are they having all the required access under the account where splunk is running? Always take a backup of "etc" folder before doing any changes to the instance.
It's the architecture to disable the splunk indexer's splunkweb if it's the search peer. You can't start it...
Please refer the documentation for clustered peer/Splunk indexers. It's not explicitly mentioned but it's no where mentioned that you can access indexer from UI.
Moreover you can actually search everything from your search head why do you need the indexer splunkweb. Finally don't worry that is the expected behavior when it's only a search peer and it's pooled. Thanks 🙂
Yes I added it as a search peer. How can I investigate that on the indexer side. The indexer one is being used in production and I don’t touch the config or anything I just added the search head license under its license.
Unfortunately, I cannot answer your additional questions. My expertise - such as it is - lies in running Splunk on Linux.
However - something does strike me.
"0400 ERROR ConfPathMapperManager - Error while initializing path: C:Program FilesSplunketcuserssplunk-system-usersearchhistorY"
Have you reconfigured this path? And if you did, did you use forward or backward (windows-style) slashes? Notice how the path in the log has no directory separators. Could just be a reporting issue, or it could be that Splunk has created what should be a directory as a single file.
0400 ERROR BundlesUtil - Cannot create directory: C:\Program Files\Splunk\etc\users\splunk-system-user\search\history: Cannot create a file when that file already exists.
No specific error in web-service.log,all infos.
and about file permissions, could you please explain how could I alter them? cause as far as I remember I didn't mess with the permissions.
Thank you for the response.Both instances are on windows servers. the search head is windows server 2008 R2,8 cpu cores,8G RAM,30G Cdrive and 600GB second hard drive!(this is the instance with problem)
I checked the logs:
Splunkd.log (3 common errors):
ERROR timeinvertedIndex - Raw size file="C:\Program Files\Splunk\var\lib\splunk\perfmon\db\db_1372095803_1371500475_27.rawSize" contains invalid data (\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00)
0400 ERROR ConfPathMapperManager - Error while initializing path: C:\Program Files\Splunk\etc\users\splunk-system-user\search\historY