All Apps and Add-ons

Splunk Addon for AWS configuration and Inputs keeps not loading after Splunk upgrade to 8.2.3.3?

sumeet1503
Explorer

HI All,

We are facing and issue with Splunk Addon for AWS where the Configuration and Inputs page on UI isn't loading and it keeps on loading with a circle without getting the page. 

We are observing it after we did Splunk enterprise upgrade recently to 8.2.3.3 . We also updated the add-on to latest version - 5.2.1

Under Splunkd log we can see some REST errors . PFB 

12-23-2021 07:40:25.606 -0500 ERROR AdminManagerExternal [11556 TcpChannelThread] - Unexpected error "<class 'splunklib.binding.HTTPError'>" from python handler: "HTTP 500 Internal Server E
rror -- Unexpected error "<class 'splunktaucclib.rest_handler.error.RestError'>" from python handler: "REST Error [500]: Internal Server Error -- Traceback (most recent call last):\n File
"/opt/splunk/hf/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/urllib3/connectionpool.py", line 667, in urlopen\n self._prepare_proxy(conn)\n File "/opt/splunk/hf/etc/apps/Splunk_TA_aws/bi
n/3rdparty/python3/urllib3/connectionpool.py", line 930, in _prepare_proxy\n conn.connect()\n File "/opt/splunk/hf/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/urllib3/connection.py", li
ne 316, in connect\n self._tunnel()\n File "/opt/splunk/hf/lib/python3.7/http/client.py", line 931, in _tunnel\n message.strip()))\nOSError: Tunnel connection failed: 403 Forbidden\n
\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "/opt/splunk/hf/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/solnlib/packa
ges/requests/adapters.py", line 449, in send\n timeout=timeout\n File "/opt/splunk/hf/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/urllib3/connectionpool.py", line 725, in urlopen\n m
ethod, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]\n File "/opt/splunk/hf/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/urllib3/util/retry.py", line 439, in increment\n raise
MaxRetryError(_pool, url, error or ResponseError(cause))\nurllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='0', port=8089): Max retries exceeded with url: /servicesNS/nobody/Splun
k_TA_aws/configs/conf-aws_sqs_tasks/_reload (Caused by ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 403 Forbidden')))\n\nDuring handling of the above exception,
another exception occurred:\n\nTraceback (most recent call last):\n File "/opt/splunk/hf/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/splunktaucclib/rest_handler/handler.

 

Error from python.log

Traceback (most recent call last):
File "/opt/splunk/hf/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/solnlib/splunk_rest_client.py", line 145, in request
verify=verify, proxies=proxies, cert=cert, **kwargs)
File "/opt/splunk/hf/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/solnlib/packages/requests/api.py", line 60, in request
return session.request(method=method, url=url, **kwargs)
File "/opt/splunk/hf/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/solnlib/packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/opt/splunk/hf/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/solnlib/packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/opt/splunk/hf/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/solnlib/packages/requests/adapters.py", line 510, in send
raise ProxyError(e, request=request)
solnlib.packages.requests.exceptions.ProxyError: HTTPSConnectionPool(host='0', port=8089): Max retries exceeded with url: /servicesNS/nobody/Splunk_TA_aws/configs/conf-aws_sqs_tasks/_reload (Caused by ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 403 Forbidden')))

 

It has just impacted the UI, the inputs continues to work in the background. 

Any help with this would be appreciated. 

Regards,

Sumeet

 

Labels (1)
0 Karma

behlkush
Path Finder

Sometimes the fix is right there in the documentation itself:
https://docs.splunk.com/Documentation/AddOns/released/AWS/Troubleshooting

 

I fixed the issue by updating splunk-launch.conf file and adding the custom PORT for Management.

The latest version of Aws add on doesn't work with custom management port. It only works on 8089.

 

aasabatini
Motivator

Hi @sumeet1503 

I had the same issue last month.

by my side was a management port error, which port do you use? 8089? or another one?

Regards

Ale

“The answer is out there, Neo, and it’s looking for you, and it will find you if you want it to.”
0 Karma

sumeet1503
Explorer

Hi @aasabatini :

Its using port 8089 for internal communication .Not sure whats wrong , it was all good until the upgrade and now we cant load the UI - full of Rest errors .

What did you do to fix this. ?

Regards,

Sumeet

0 Karma

dm1
Contributor

were you able to fix it ? if yes, please share how ?

0 Karma

sumeet1503
Explorer

Yes it works now. Basically we started splunkd from root on the box which fixed this UI issue.

Earlier we were starting it from splunk and was facing same. 

This was strange but at-least it works now.

We opened couple of splunk support tickets but they were not able to help.

One of the other solutions provided was to take a backup of inputs .. reinstall aws addon and reinstate the inputs .. we didnt try this as we couldnt afford to loose the data flow for a while .

hope this helps.. happy splunking

isoutamo
SplunkTrust
SplunkTrust

Hi

You shouldn't run splunk as a root as it's quite huge security issue!

Here is instructions how to fix management port issue with Splunk_TA_aws.

Failed to load input and configuration page when running the Splunk software on a custom management ...

r. Ismo

0 Karma

dm1
Contributor

but the link you shared is for when Splunk custom port is used, however, in my case, I am just using default port 8089. does that setting still apply ?

0 Karma

dm1
Contributor

Thanks for your prompt reply.

in that instance, what user your splunk service running as root or splunk ?

We are running splunk as "splunk" user, hence the question

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

(view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...