Unable to update app using GUI

[Mark_H]

Hi all, when I try to update any installed apps from the GUI I receive a 500 internal error.

Checking the _internal logs I see this:
File "/opt/splunk/lib/python3.7/site-packages/splunk/rest/__init__.py", line 655, in simpleRequest raise splunk.ResourceNotFound(uri) splunk.ResourceNotFound: [HTTP 404] https://127.0.0.1:8089/services/apps/remote/entriesbyid/SplunkAdmins

I am on 9.0.3. I don't have a proxy setup. And all my file permissions are fine.
Hope someone can help on this one.

Thanks.

[dtsariapkin]

Hi,

I will be putting this reply to the posts that I can find.  I know it's a late reply to some. But hope this will help you all.  And anyone having similar issues in the future.

The issue I will be discussing here is when Splunk update does NOT update from Splunk Web. And when you search for the error you find similar to this:
splunk.ResourceNotFound: [HTTP 404]
 

Explanation on how really it works:

When you try to update the app Splunk Web makes a call to itself 127.0.0.1 on port 8089 for SplunkD   at /services/apps/remote/entriesbyid/<your_app> e.g. ->

 https://127.0.0.1:8089/services/apps/remote/entriesbyid/Splunk_TA_windows

which you can check yourself by simple CURL:

curl -k --user "admin:changeme" https://127.0.0.1:8089/services/apps/remote/entriesbyid/Splunk_TA_windows

This call is getting proxied via SplunkD process to the internet which would end up calling  https://splunkbase.splunk.com/api/apps/entriesbyid/<your_app>
 

which you can check yourself by simple CURL:

curl -k  https://splunkbase.splunk.com/api/apps/entriesbyid/Splunk_TA_windows

Now the issues here can be numerous from here on. To give some examples:

1. Splunk has issues accessing internet from SplunkD process
2. Certificate chain was changed. By default it is configured in server.conf
   [applicationsManagement]
   sslVerifyServerCert = false
3. Proxy and/or Firewall in the middle which is changing certificates.

One of the ways you can check for networking issues for that is do a tcpdump for packet capture and check the SSL Conversation:
tcpdump -i <interface> -s 65535 port 443 -w /tmp/port443.pcap

That's for people who are familiar what packet capture looks like and can understand it's contents.