Installation

Getting error when trying to upgrade from 5.0.2 to 5.0.3: "Splunk installer was unable to create Splunk Service".

Path Finder

I have 2 Splunk instances. one as an indexer and search head and one just as a dedicated search head (on a VM just for testing and eventually moving to production). Everything was fine and working (except some permission issues which people couldn't save anything as private). So after awhile I was trying to log in and start investigating again but the GUI didn't come up and I found out that splunkweb service won't stay up, meaning I can't start the service but after like one second it stops! The only change that has been made recently is that the other instance which I was using as an indexer upgraded to 5.0.3 and the search head is still 5.0.2. So I tried to upgrade it but it gives this error "Splunk installer was unable to create Splunk Service"
Any ideas?

Labels (3)
0 Karma

Communicator

Cheers guys this thread pointed me in the right direction and gave me enough direction to figure out what my issues was. FYI it was the Splunkd service overloading my Splunk cluster which in turn was stopping the service from starting on one of my indexers.

Thanks again, the log file was the big help 🙂

0 Karma

Path Finder

So I found my answer

In my search head event viewer I got this error"

File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\win32\win32serviceutil.py", line 785, in SvcRun self.SvcDoRun() File "C:\Program Files\Splunk\bin\SplunkWebService.py", line 40, in SvcDoRun root.run(blocking=False) File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\splunk\appserver\mrsparkle\root.py", line 539, in run i18n.init_js_cache(flush_files=True) File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\splunk\appserver\mrsparkle\lib\i18n.py", line 741, in init_js_cache os.unlink(os.path.join(CACHE_PATH, fn))

(32, 'The process cannot access the file because it is being used by another process', 'C:\Program Files\Splunk\var\run\splunk\appserver\i18n\version')

So I checked the "version" folder it was empty I just deleted the folder and started the splunkweb service again and it was regenerated and now it's working yeyyy

Path Finder

This worked for me, thanxs!

0 Karma

Champion

It should have turn on all logging channels. Splunkwebs logs are not in splunkd.log but in web_service.log. I would use the btool and do a config dump. Normally when this happens to me its when I've made a setting change to web.conf or server.conf which splunkweb does not like. Another possibility is that permission inheritance was broken (done this while converting from running services as domain account and move to LOCAL\NTAUTHORITY account..

0 Karma

Path Finder

It's running as a local system under my domain account.it was ldap integrated before and it was running fine. I did the --debug and both services (splunkd and splunkweb starts and "Done") but when i check the splunkweb it's stopped!

0 Karma

Champion

by local account do you mean Local System or a local account you created? Also trying starting splunk with ./splunk start --debug. This may give you more detail.

0 Karma

Path Finder

as a local account!

0 Karma

Champion

Question are you running Splunk as a Domain Service Account, local account, or NTAUTHORITY?

0 Karma

Motivator

Your first port of call for answers would be a couple of Splunk's log files splunkd.log and web_service.log (in $SPLUNK_HOME/var/log/splunk/). You might find helpful information in there.

You don't really provide enough information to start speculating on the causes of your problem, and you don't tell us what o/s you are running on, but are you sure that a) you don't have a shortage of disk space or swap, or b) that you have not been altering file permissions?

0 Karma

Path Finder

Yes it is the same account it was running before and working fine! I did not change a bit!

0 Karma

Champion

Did you check the service under which account it's running? The folders are they having all the required access under the account where splunk is running? Always take a backup of "etc" folder before doing any changes to the instance.

0 Karma

Path Finder

Hi Linu,
The thing is my indexer is working fine it's the search head that wouldn't come up. All mu problem is with my search head and its splunkweb service!

0 Karma

Champion

It's the architecture to disable the splunk indexer's splunkweb if it's the search peer. You can't start it...

Please refer the documentation for clustered peer/Splunk indexers. It's not explicitly mentioned but it's no where mentioned that you can access indexer from UI.

Moreover you can actually search everything from your search head why do you need the indexer splunkweb. Finally don't worry that is the expected behavior when it's only a search peer and it's pooled. Thanks 🙂

0 Karma

Path Finder

Yes I added it as a search peer. How can I investigate that on the indexer side. The indexer one is being used in production and I don’t touch the config or anything I just added the search head license under its license.

0 Karma

Champion

Hello MarMoh,
Is the indexer is added as a search peer? I think it will not allow you to start the Splunk Web on the indexer.

0 Karma

Motivator

Unfortunately, I cannot answer your additional questions. My expertise - such as it is - lies in running Splunk on Linux.

However - something does strike me.

"0400 ERROR ConfPathMapperManager - Error while initializing path: C:Program FilesSplunketcuserssplunk-system-usersearchhistorY"

Have you reconfigured this path? And if you did, did you use forward or backward (windows-style) slashes? Notice how the path in the log has no directory separators. Could just be a reporting issue, or it could be that Splunk has created what should be a directory as a single file.

0 Karma

Path Finder

0400 ERROR BundlesUtil - Cannot create directory: C:\Program Files\Splunk\etc\users\splunk-system-user\search\history: Cannot create a file when that file already exists.

No specific error in web-service.log,all infos.

and about file permissions, could you please explain how could I alter them? cause as far as I remember I didn't mess with the permissions.

Thanks,
M

0 Karma

Path Finder

Hi grijhwani,

Thank you for the response.Both instances are on windows servers. the search head is windows server 2008 R2,8 cpu cores,8G RAM,30G Cdrive and 600GB second hard drive!(this is the instance with problem)
I checked the logs:
Splunkd.log (3 common errors):
ERROR timeinvertedIndex - Raw size file="C:\Program Files\Splunk\var\lib\splunk\perfmon\db\db_1372095803_1371500475_27.rawSize" contains invalid data (\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00)

0400 ERROR ConfPathMapperManager - Error while initializing path: C:\Program Files\Splunk\etc\users\splunk-system-user\search\historY

0 Karma