App Upgrade GUI- Getting internal server error?

[pglover]

Hi all,

When I visit the Apps page on my search head server and select "Upgrade to ..." on any of my applications that require an upgrade I get the following 500 Inernal Server Error: /opt/splunk/var/log/splunk/web_service.log

2022-08-23 21:11:36,576 ERROR [63054287cc7fe2c82754d0] error:335 - Traceback (most recent call last):
File "/opt/splunk/lib/python3.7/site-packages/cherrypy/_cprequest.py", line 628, in respond
self._do_respond(path_info)
File "/opt/splunk/lib/python3.7/site-packages/cherrypy/_cprequest.py", line 687, in _do_respond
response.body = self.handler()
File "/opt/splunk/lib/python3.7/site-packages/cherrypy/lib/encoding.py", line 219, in __call__
self.body = self.oldhandler(*args, **kwargs)
File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/lib/htmlinjectiontoolfactory.py", line 75, in wrapper
resp = handler(*args, **kwargs)
File "/opt/splunk/lib/python3.7/site-packages/cherrypy/_cpdispatch.py", line 54, in __call__
return self.callable(*self.args, **self.kwargs)
File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/lib/routes.py", line 383, in default
return route.target(self, **kw)
File "</opt/splunk/lib/python3.7/site-packages/decorator.py:decorator-gen-98>", line 2, in start
File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/lib/decorators.py", line 40, in rundecs
return fn(*a, **kw)
File "</opt/splunk/lib/python3.7/site-packages/decorator.py:decorator-gen-96>", line 2, in start
File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/lib/decorators.py", line 118, in check
return fn(self, *a, **kw)
File "</opt/splunk/lib/python3.7/site-packages/decorator.py:decorator-gen-95>", line 2, in start
File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/lib/decorators.py", line 166, in validate_ip
return fn(self, *a, **kw)
File "</opt/splunk/lib/python3.7/site-packages/decorator.py:decorator-gen-94>", line 2, in start
File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/lib/decorators.py", line 245, in preform_sso_check
return fn(self, *a, **kw)
File "</opt/splunk/lib/python3.7/site-packages/decorator.py:decorator-gen-93>", line 2, in start
File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/lib/decorators.py", line 284, in check_login
return fn(self, *a, **kw)
File "</opt/splunk/lib/python3.7/site-packages/decorator.py:decorator-gen-92>", line 2, in start
File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/lib/decorators.py", line 304, in handle_exceptions
return fn(self, *a, **kw)
File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/controllers/appinstall.py", line 232, in start
remote_app = self.getRemoteAppEntry(appid);
File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/controllers/appinstall.py", line 106, in getRemoteAppEntry
return en.getEntity('/apps/remote/entriesbyid', sbAppId)
File "/opt/splunk/lib/python3.7/site-packages/splunk/entity.py", line 277, in getEntity
serverResponse, serverContent = rest.simpleRequest(uri, getargs=kwargs, sessionKey=sessionKey, raiseAllErrors=True)
File "/opt/splunk/lib/python3.7/site-packages/splunk/rest/__init__.py", line 655, in simpleRequest
raise splunk.ResourceNotFound(uri)
splunk.ResourceNotFound: [HTTP 404] https://127.0.0.1:8089/services/apps/remote/entriesbyid/treeview-viz

This happens on any application I try to upgrade via the GUI - I can upgrade them ok when doing it manually or by uploading the tgz archive.

I can use the GUI just fine on any other server (Index/Heavy Forwarder).

Does anyone know what could be causing this? Any help would be appreciated.

[dtsariapkin]

Hi All,

I will be putting this reply to the posts that I can find.  I know it's a late reply to some. But hope this will help you all.  And anyone having similar issues in the future.

The issue I will be discussing here is when Splunk update does NOT update from Splunk Web. And when you search for the error you find similar to this:
splunk.ResourceNotFound: [HTTP 404]
 

Explanation on how really it works:

When you try to update the app Splunk Web makes a call to itself 127.0.0.1 on port 8089 for SplunkD   at /services/apps/remote/entriesbyid/<your_app> e.g. ->

 https://127.0.0.1:8089/services/apps/remote/entriesbyid/Splunk_TA_windows

which you can check yourself by simple CURL:

curl -k --user "admin:changeme" https://127.0.0.1:8089/services/apps/remote/entriesbyid/Splunk_TA_windows

This call is getting proxied via SplunkD process to the internet which would end up calling  https://splunkbase.splunk.com/api/apps/entriesbyid/<your_app>
 

which you can check yourself by simple CURL:

curl -k  https://splunkbase.splunk.com/api/apps/entriesbyid/Splunk_TA_windows

Now the issues here can be numerous from here on. To give some examples:

1. Splunk has issues accessing internet from SplunkD process
2. Certificate chain was changed. By default it is configured in server.conf
   [applicationsManagement]
   sslVerifyServerCert = false
3. Proxy and/or Firewall in the middle which is changing certificates.

One of the ways you can check for networking issues for that is do a tcpdump for packet capture and check the SSL Conversation:
tcpdump -i <interface> -s 65535 port 443 -w /tmp/port443.pcap

That's for people who are familiar what packet capture looks like and can understand it's contents.

[woodcock]

Open a support ticket.

[Johnn]

I am having the same problem on all splunk instances. Any help apreciated

[markhill1]

Hi, Did you find a solution to this? I am seeing the same thing on one of my lab machines.
Thanks.

[pglover]

I haven't found a fix for this as yet unfortunately 

[dtsariapkin]

Hi,

I will be putting this reply to the posts that I can find.  I know it's a late reply to some. But hope this will help you all.  And anyone having similar issues in the future.

The issue I will be discussing here is when Splunk update does NOT update from Splunk Web. And when you search for the error you find similar to this:
splunk.ResourceNotFound: [HTTP 404]
 

Explanation on how really it works:

When you try to update the app Splunk Web makes a call to itself 127.0.0.1 on port 8089 for SplunkD   at /services/apps/remote/entriesbyid/<your_app> e.g. ->

 https://127.0.0.1:8089/services/apps/remote/entriesbyid/Splunk_TA_windows

which you can check yourself by simple CURL:

curl -k --user "admin:changeme" https://127.0.0.1:8089/services/apps/remote/entriesbyid/Splunk_TA_windows

This call is getting proxied via SplunkD process to the internet which would end up calling  https://splunkbase.splunk.com/api/apps/entriesbyid/<your_app>
 

which you can check yourself by simple CURL:

curl -k  https://splunkbase.splunk.com/api/apps/entriesbyid/Splunk_TA_windows

Now the issues here can be numerous from here on. To give some examples:

1. Splunk has issues accessing internet from SplunkD process
2. Certificate chain was changed. By default it is configured in server.conf
   [applicationsManagement]
   sslVerifyServerCert = false
3. Proxy and/or Firewall in the middle which is changing certificates.

One of the ways you can check for networking issues for that is do a tcpdump for packet capture and check the SSL Conversation:
tcpdump -i <interface> -s 65535 port 443 -w /tmp/port443.pcap

That's for people who are familiar what packet capture looks like and can understand it's contents.