Splunk Enterprise

App Upgrade GUI- Getting internal server error?

pglover
Engager

Hi all,

When I visit the Apps page on my search head server and select "Upgrade to ..." on any of my applications that require an upgrade I get the following 500 Inernal Server Error: /opt/splunk/var/log/splunk/web_service.log

2022-08-23 21:11:36,576 ERROR [63054287cc7fe2c82754d0] error:335 - Traceback (most recent call last):
File "/opt/splunk/lib/python3.7/site-packages/cherrypy/_cprequest.py", line 628, in respond
self._do_respond(path_info)
File "/opt/splunk/lib/python3.7/site-packages/cherrypy/_cprequest.py", line 687, in _do_respond
response.body = self.handler()
File "/opt/splunk/lib/python3.7/site-packages/cherrypy/lib/encoding.py", line 219, in __call__
self.body = self.oldhandler(*args, **kwargs)
File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/lib/htmlinjectiontoolfactory.py", line 75, in wrapper
resp = handler(*args, **kwargs)
File "/opt/splunk/lib/python3.7/site-packages/cherrypy/_cpdispatch.py", line 54, in __call__
return self.callable(*self.args, **self.kwargs)
File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/lib/routes.py", line 383, in default
return route.target(self, **kw)
File "</opt/splunk/lib/python3.7/site-packages/decorator.py:decorator-gen-98>", line 2, in start
File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/lib/decorators.py", line 40, in rundecs
return fn(*a, **kw)
File "</opt/splunk/lib/python3.7/site-packages/decorator.py:decorator-gen-96>", line 2, in start
File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/lib/decorators.py", line 118, in check
return fn(self, *a, **kw)
File "</opt/splunk/lib/python3.7/site-packages/decorator.py:decorator-gen-95>", line 2, in start
File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/lib/decorators.py", line 166, in validate_ip
return fn(self, *a, **kw)
File "</opt/splunk/lib/python3.7/site-packages/decorator.py:decorator-gen-94>", line 2, in start
File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/lib/decorators.py", line 245, in preform_sso_check
return fn(self, *a, **kw)
File "</opt/splunk/lib/python3.7/site-packages/decorator.py:decorator-gen-93>", line 2, in start
File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/lib/decorators.py", line 284, in check_login
return fn(self, *a, **kw)
File "</opt/splunk/lib/python3.7/site-packages/decorator.py:decorator-gen-92>", line 2, in start
File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/lib/decorators.py", line 304, in handle_exceptions
return fn(self, *a, **kw)
File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/controllers/appinstall.py", line 232, in start
remote_app = self.getRemoteAppEntry(appid);
File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/controllers/appinstall.py", line 106, in getRemoteAppEntry
return en.getEntity('/apps/remote/entriesbyid', sbAppId)
File "/opt/splunk/lib/python3.7/site-packages/splunk/entity.py", line 277, in getEntity
serverResponse, serverContent = rest.simpleRequest(uri, getargs=kwargs, sessionKey=sessionKey, raiseAllErrors=True)
File "/opt/splunk/lib/python3.7/site-packages/splunk/rest/__init__.py", line 655, in simpleRequest
raise splunk.ResourceNotFound(uri)
splunk.ResourceNotFound: [HTTP 404] https://127.0.0.1:8089/services/apps/remote/entriesbyid/treeview-viz

This happens on any application I try to upgrade via the GUI - I can upgrade them ok when doing it manually or by uploading the tgz archive.

I can use the GUI just fine on any other server (Index/Heavy Forwarder).

Does anyone know what could be causing this? Any help would be appreciated.

Labels (3)
0 Karma

dtsariapkin
Splunk Employee
Splunk Employee

Hi All,

I will be putting this reply to the posts that I can find.  I know it's a late reply to some. But hope this will help you all.  And anyone having similar issues in the future.

The issue I will be discussing here is when Splunk update does NOT update from Splunk Web. And when you search for the error you find similar to this:
splunk.ResourceNotFound: [HTTP 404]
 

Explanation on how really it works:

When you try to update the app Splunk Web makes a call to itself 127.0.0.1 on port 8089 for SplunkD   at /services/apps/remote/entriesbyid/<your_app> e.g. ->

 https://127.0.0.1:8089/services/apps/remote/entriesbyid/Splunk_TA_windows

which you can check yourself by simple CURL:

curl -k --user "admin:changeme" https://127.0.0.1:8089/services/apps/remote/entriesbyid/Splunk_TA_windows

 

This call is getting proxied via SplunkD process to the internet which would end up calling  https://splunkbase.splunk.com/api/apps/entriesbyid/<your_app>
 

which you can check yourself by simple CURL:

curl -k  https://splunkbase.splunk.com/api/apps/entriesbyid/Splunk_TA_windows

 

Now the issues here can be numerous from here on. To give some examples:

  1. Splunk has issues accessing internet from SplunkD process
  2. Certificate chain was changed. By default it is configured in server.conf
    [applicationsManagement]
    sslVerifyServerCert = false
  3. Proxy and/or Firewall in the middle which is changing certificates.

One of the ways you can check for networking issues for that is do a tcpdump for packet capture and check the SSL Conversation:
tcpdump -i <interface> -s 65535 port 443 -w /tmp/port443.pcap

That's for people who are familiar what packet capture looks like and can understand it's contents.

Dmitrii T.
0 Karma

woodcock
Esteemed Legend

Open a support ticket.

0 Karma

Johnn
Loves-to-Learn Lots

I am having the same problem on all splunk instances. Any help apreciated

0 Karma

markhill1
Path Finder

Hi, Did you find a solution to this? I am seeing the same thing on one of my lab machines.
Thanks.

0 Karma

pglover
Engager

I haven't found a fix for this as yet unfortunately 

0 Karma

dtsariapkin
Splunk Employee
Splunk Employee

Hi,

I will be putting this reply to the posts that I can find.  I know it's a late reply to some. But hope this will help you all.  And anyone having similar issues in the future.

The issue I will be discussing here is when Splunk update does NOT update from Splunk Web. And when you search for the error you find similar to this:
splunk.ResourceNotFound: [HTTP 404]
 

Explanation on how really it works:

When you try to update the app Splunk Web makes a call to itself 127.0.0.1 on port 8089 for SplunkD   at /services/apps/remote/entriesbyid/<your_app> e.g. ->

 https://127.0.0.1:8089/services/apps/remote/entriesbyid/Splunk_TA_windows

which you can check yourself by simple CURL:

curl -k --user "admin:changeme" https://127.0.0.1:8089/services/apps/remote/entriesbyid/Splunk_TA_windows

 

This call is getting proxied via SplunkD process to the internet which would end up calling  https://splunkbase.splunk.com/api/apps/entriesbyid/<your_app>
 

which you can check yourself by simple CURL:

curl -k  https://splunkbase.splunk.com/api/apps/entriesbyid/Splunk_TA_windows

 

Now the issues here can be numerous from here on. To give some examples:

  1. Splunk has issues accessing internet from SplunkD process
  2. Certificate chain was changed. By default it is configured in server.conf
    [applicationsManagement]
    sslVerifyServerCert = false
  3. Proxy and/or Firewall in the middle which is changing certificates.

One of the ways you can check for networking issues for that is do a tcpdump for packet capture and check the SSL Conversation:
tcpdump -i <interface> -s 65535 port 443 -w /tmp/port443.pcap

That's for people who are familiar what packet capture looks like and can understand it's contents.

Dmitrii T.
0 Karma
Get Updates on the Splunk Community!

How to Monitor Google Kubernetes Engine (GKE)

We’ve looked at how to integrate Kubernetes environments with Splunk Observability Cloud, but what about ...

Index This | How can you make 45 using only 4?

October 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this ...

Splunk Education Goes to Washington | Splunk GovSummit 2024

If you’re in the Washington, D.C. area, this is your opportunity to take your career and Splunk skills to the ...