Why does Splunk_TA_aws hang on configure / inputs pages after update?

[isoutamo]

Hi

I just updated splunk from 7.3.3 to 8.1.7.2 and Splunk_TA_aws from 4.6.1 to 5.2.0  (build 882) via 5.0.4. After that I cannot enter to  TA's inputs page. It open, but after that rolling "Loading" and nothing happened after that.

When I look from internal logs I found the next entries on _internal

02-24-2022 16:38:02.552 +0200 ERROR AdminManagerExternal - Stack trace from python handler:
Traceback (most recent call last):
  File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/splunklib/binding.py", line 290, in wrapper
    return request_fun(self, *args, **kwargs)
  File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/splunklib/binding.py", line 71, in new_f
    val = f(*args, **kwargs)
  File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/splunklib/binding.py", line 680, in get
    response = self.http.get(path, all_headers, **query)
  File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/splunklib/binding.py", line 1184, in get
    return self.request(url, { 'method': "GET", 'headers': headers })
  File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/splunklib/binding.py", line 1245, in request
    raise HTTPError(response)
splunklib.binding.HTTPError: HTTP 401 Unauthorized -- call not properly authenticated

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/splunk/lib/python3.7/site-packages/splunk/admin.py", line 114, in init_persistent
    hand.execute(info)
  File "/opt/splunk/lib/python3.7/site-packages/splunk/admin.py", line 637, in execute
    if self.requestedAction == ACTION_LIST:     self.handleList(confInfo)
  File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/base_input_rh.py", line 64, in handleList
    inputs = self._collection.list()
  File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/splunklib/client.py", line 1479, in list
    return list(self.iter(count=count, **kwargs))
  File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/splunklib/client.py", line 1438, in iter
    response = self.get(count=pagesize or count, offset=offset, **kwargs)
  File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/splunklib/client.py", line 1668, in get
    return super(Collection, self).get(name, owner, app, sharing, **query)
  File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/splunklib/client.py", line 766, in get
    **query)
  File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/splunklib/binding.py", line 304, in wrapper
    "Request failed: Session is not logged in.", he)
splunklib.binding.AuthenticationError: Request failed: Session is not logged in.

and after that

02-24-2022 16:38:02.552 +0200 ERROR AdminManagerExternal - Unexpected error "<class 'splunklib.binding.AuthenticationError'>" from python handler: "Request failed: Session is not logged in.".  See splunkd.log for more details.

This is an HF to get HEC and mod inputs from GCP and AWS into separate indexers. It also act as IHF for other UFs and HFs. It's on Clients own AWS environment.

I found couple of answers which has some kind of similar cases (e.g. boto.cfg) but those didn't help us.

Any ideas and hints how to solve this? We cannot update yet to 8.2.5+.

This is probably some kind of hint: HTTP 401 Unauthorized -- call not properly authenticated

All inputs are working as earlier after enabled those via conf files and restart splunkd. Before update those inputs are disabled with the same GUI (version 4.6.1).

r. Ismo

[isoutamo · Accepted solution]

FYI: The issue was a non standard management port which was used on Client's environment. After I add 

SPLUNK_MGMT_HOST_PORT=<IP>:<mgmt port>

to ${SPLUNK_HOME}/etc/splunk-launch.conf it start to work.

Currently it has documented on Splunk_TA_aws problem solving page https://docs.splunk.com/Documentation/AddOns/released/AWS/Troubleshooting#Failed_to_load_input_and_c...

r. Ismo  

[m_pham]

Hey - can you try to compare the inputs.conf from the updated TA vs the old TA? I think you can see all the settings in the inputs spec file in README directory. There may some additional configs that are needed which I think can break the GUI inputs page.

You can also just spin up a new server with the same Splunk version and AWS TA version to configure the inputs on that and then check the inputs.conf file to compare it to you existing inputs.conf file.

[isoutamo]

Thanks. I will try to install it from scratch to the another host which haven't it yet.
This is not an inputs relates issues as also configuration tab is unusable. Only working part are Health check and search tabs.

I just install it to clean machine (our MC) and it has exactly same issue. There is nothing local modifications. Only installation of this app 5.2.0 build 882.

5.0.4 was the newest version which is working as a new installation.

[duneclarke2]

When this happens to me it has always been a version issue.  Try rev. back to an earlier add-on.  Not a great answer,  but I've been there many times. Hope it helps! Dune

[isoutamo]

Then.

I just tried on clean host with the next versions.

• 5.2.0 – NOK
• 5.1.0 – NOK
• 5.0.4 – OK

Now I need to check is there anything which prevent us to go back to 5.0.4 on this node where we are needing configuration GUI or should we just use the newest one with conf files?

Probably I need to create case to Splunk Support for this.

r. Ismo

[m_pham]

Man that sucks, I tried a fresh install with Splunk v8.1.7.1 with AWS TA v5.2.0 and it worked for me. I looked at the known issues page for Splunk v8.1.7 but didn't see anything that stood out. Hopefully support can provide you with some answers.