topic Re: Splunk Addon for AWS configuration and Inputs keeps not loading after Splunk upgrade to 8.2.3.3? in All Apps and Add-ons

Splunk Addon for AWS configuration and Inputs keeps not loading after Splunk upgrade to 8.2.3.3?

sumeet1503 — Wed, 28 Sep 2022 14:55:53 GMT

HI All,

We are facing and issue with Splunk Addon for AWS where the Configuration and Inputs page on UI isn't loading and it keeps on loading with a circle without getting the page.

We are observing it after we did Splunk enterprise upgrade recently to 8.2.3.3 . We also updated the add-on to latest version - 5.2.1

Under Splunkd log we can see some REST errors . PFB

12-23-2021 07:40:25.606 -0500 ERROR AdminManagerExternal [11556 TcpChannelThread] - Unexpected error "<class 'splunklib.binding.HTTPError'>" from python handler: "HTTP 500 Internal Server E
rror -- Unexpected error "<class 'splunktaucclib.rest_handler.error.RestError'>" from python handler: "REST Error [500]: Internal Server Error -- Traceback (most recent call last):\n File
"/opt/splunk/hf/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/urllib3/connectionpool.py", line 667, in urlopen\n self._prepare_proxy(conn)\n File "/opt/splunk/hf/etc/apps/Splunk_TA_aws/bi
n/3rdparty/python3/urllib3/connectionpool.py", line 930, in _prepare_proxy\n conn.connect()\n File "/opt/splunk/hf/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/urllib3/connection.py", li
ne 316, in connect\n self._tunnel()\n File "/opt/splunk/hf/lib/python3.7/http/client.py", line 931, in _tunnel\n message.strip()))\nOSError: Tunnel connection failed: 403 Forbidden\n
\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "/opt/splunk/hf/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/solnlib/packa
ges/requests/adapters.py", line 449, in send\n timeout=timeout\n File "/opt/splunk/hf/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/urllib3/connectionpool.py", line 725, in urlopen\n m
ethod, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]\n File "/opt/splunk/hf/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/urllib3/util/retry.py", line 439, in increment\n raise
MaxRetryError(_pool, url, error or ResponseError(cause))\nurllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='0', port=8089): Max retries exceeded with url: /servicesNS/nobody/Splun
k_TA_aws/configs/conf-aws_sqs_tasks/_reload (Caused by ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 403 Forbidden')))\n\nDuring handling of the above exception,
another exception occurred:\n\nTraceback (most recent call last):\n File "/opt/splunk/hf/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/splunktaucclib/rest_handler/handler.

Error from python.log

Traceback (most recent call last):
File "/opt/splunk/hf/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/solnlib/splunk_rest_client.py", line 145, in request
verify=verify, proxies=proxies, cert=cert, **kwargs)
File "/opt/splunk/hf/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/solnlib/packages/requests/api.py", line 60, in request
return session.request(method=method, url=url, **kwargs)
File "/opt/splunk/hf/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/solnlib/packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/opt/splunk/hf/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/solnlib/packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/opt/splunk/hf/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/solnlib/packages/requests/adapters.py", line 510, in send
raise ProxyError(e, request=request)
solnlib.packages.requests.exceptions.ProxyError: HTTPSConnectionPool(host='0', port=8089): Max retries exceeded with url: /servicesNS/nobody/Splunk_TA_aws/configs/conf-aws_sqs_tasks/_reload (Caused by ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 403 Forbidden')))

It has just impacted the UI, the inputs continues to work in the background.

Any help with this would be appreciated.

Regards,

Sumeet

Re: Splunk Addon for AWS configuration and Inputs keeps on loading after SPlunk upgrade to 8.2.3.3

aasabatini — Thu, 23 Dec 2021 14:00:14 GMT

Hi @sumeet1503

I had the same issue last month.

by my side was a management port error, which port do you use? 8089? or another one?

Regards

Ale

Re: Splunk Addon for AWS configuration and Inputs keeps on loading after SPlunk upgrade to 8.2.3.3

sumeet1503 — Thu, 23 Dec 2021 14:31:41 GMT

Hi @aasabatini :

Its using port 8089 for internal communication .Not sure whats wrong , it was all good until the upgrade and now we cant load the UI - full of Rest errors .

What did you do to fix this. ?

Regards,

Sumeet

Re: Splunk Addon for AWS configuration and Inputs keeps on loading after SPlunk upgrade to 8.2.3.3

dm1 — Wed, 28 Sep 2022 05:20:31 GMT

were you able to fix it ? if yes, please share how ?

Re: Splunk Addon for AWS configuration and Inputs keeps on loading after SPlunk upgrade to 8.2.3.3

sumeet1503 — Wed, 28 Sep 2022 05:41:30 GMT

Yes it works now. Basically we started splunkd from root on the box which fixed this UI issue.

Earlier we were starting it from splunk and was facing same.

This was strange but at-least it works now.

We opened couple of splunk support tickets but they were not able to help.

One of the other solutions provided was to take a backup of inputs .. reinstall aws addon and reinstate the inputs .. we didnt try this as we couldnt afford to loose the data flow for a while .

hope this helps.. happy splunking

Re: Splunk Addon for AWS configuration and Inputs keeps on loading after SPlunk upgrade to 8.2.3.3

dm1 — Wed, 28 Sep 2022 05:45:35 GMT

Thanks for your prompt reply.

in that instance, what user your splunk service running as root or splunk ?

We are running splunk as "splunk" user, hence the question

Re: Splunk Addon for AWS configuration and Inputs keeps on loading after SPlunk upgrade to 8.2.3.3

isoutamo — Wed, 28 Sep 2022 07:25:07 GMT

You shouldn't run splunk as a root as it's quite huge security issue!

Here is instructions how to fix management port issue with Splunk_TA_aws.

Failed to load input and configuration page when running the Splunk software on a custom management port

r. Ismo

Re: Splunk Addon for AWS configuration and Inputs keeps on loading after SPlunk upgrade to 8.2.3.3

dm1 — Wed, 28 Sep 2022 23:08:09 GMT

but the link you shared is for when Splunk custom port is used, however, in my case, I am just using default port 8089. does that setting still apply ?

Re: Splunk Addon for AWS configuration and Inputs keeps not loading after Splunk upgrade to 8.2.3.3?

behlkush — Tue, 19 Sep 2023 20:24:40 GMT

Sometimes the fix is right there in the documentation itself:
https://docs.splunk.com/Documentation/AddOns/released/AWS/Troubleshooting

I fixed the issue by updating splunk-launch.conf file and adding the custom PORT for Management.

The latest version of Aws add on doesn't work with custom management port. It only works on 8089.