How to delete an IAM role from the Configure section in the Splunk Add-on for AWS? Just displays "Please Wait".

[n6BXGybt]

I have a stale account with no credentials associated. But when I try to delete it from the Configure tab the status just shows "Please Wait". I let it run for over 5 minutes but no change.

Splunk Version 6.3.0
Splunk Build aa7d4b1ccb80

AWS App Version4.1.1

[pchen_splunk · Accepted solution]

Hi Rich

First of all, please update your App to version 4.2.0 or above. IAM role was only supported in AWS app since version 4.2.0. Or, you can simply update it to the latest version 4.2.1

In version 4.2.0, IAM role account is read only. IAM role account is assigned to EC2 at provision phase. You can't delete it. The IAM account is detected by AWS add-on automatically. Then, you can view it in app and do inputs configuration, but not edit or delete the IAM account.

What if an IAM role is staled? Ideally, add-on should be able to detect it. Then, in data ingestion phase, no data of that account can be fetched. I don't think you can delete that account. It's not a typical use case to stale an IAM role but still have Splunk running on that server. If you want to clean all related inputs of that IAM account, you can simply delete inputs instead of deleting account.

Peter Chen

[n6BXGybt]

Thanks for your reply.

When I look at the documentation below, I don't see any mention of using a role on the Splunk instance:

http://docs.splunk.com/Documentation/AWS/4.2.1/Installation/ConfigureyourAWSpermissions

The reason why I have this role applied to my Splunk instance is because we assign IAM roles to all instances since IAM roles cannot retroactively be added to instances. Not only is it a best practice but I do this for cases where I need to copy a file to S3 - for example the Splunk diagnostics file and it's easy to do with a temporary IAM policy applied to the IAM role.

Unfortunately when I wrote my question for some reason I thought it was a stale account but as you mentioned, Splunk automatically detected the role and I guess I didn't notice before. This IAM role is for our own general AWS infrastructure management and not related to Splunk specifically.

So it looks like I asked the wrong question. 😞 Apologies for this.

The take away is that IAM roles applied to Splunk instances will automatically be detected and displayed in the Accounts List of the Splunk App for AWS and cannot be deleted, correct?

[pchen_splunk]

Yes, correct. IAM roles applied to Splunk Add-on instances will automatically be detected and displayed in the Accounts List of the Splunk App for AWS and cannot be deleted.

[rpille_splunk]

Hi Rich. Not sure why it spins, but try going into the add-on configuration UI and deleting that account from that location instead.

[n6BXGybt]

I tried removing from the add-on configuration UI but unless I'm looking at the wrong spot, there doesn't seem to be a way to remove.

I ran a tail while trying to remove and found this log:

sudo tail -F /opt/splunk/var/log/splunk/splunk_ta_aws_util.log

Traceback (most recent call last):
  File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws_rh_settings_account.py", line 103, in handleRemove
    km.delete_accesskey(self.callerArgs.id)
  File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/taaws/aws_accesskeys.py", line 123, in delete_accesskey
    self._cred_mgr.delete(c.key_id, c.name, self.namespace, self.owner)
  File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/taaws/credentials.py", line 110, in delete
    response, content = splunk.rest.simpleRequest(cred_id, method='DELETE', raiseAllErrors=True, sessionKey=self._sessionKey)
  File "/opt/splunk/lib/python2.7/site-packages/splunk/rest/__init__.py", line 529, in simpleRequest
    raise splunk.ResourceNotFound(uri, extendedMessages=extractMessages(body))
ResourceNotFound: [HTTP 404] https://127.0.0.1:8089/servicesNS/-/Splunk_TA_aws/storage/passwords/M1-LOGGING-ROLE%3AASIAJFJEIOJ4E6T2JQVA%3A; [{'text': "\n In handler 'passwords': Delete user failed, cannot find the credential information with id : credential:M1-LOGGING-ROLE:ASIAJFJEIOJ4E6T2JQVA:", 'code': None, 'type': 'ERROR'}]

What I don't understand is that the account I'm trying to remove is actually an IAM role so it shouldn't be there in the first place.

Looking at the log above it seems that the script is looking for the secret key for the access key ASIAJFJEIOJ4E6T2JQVA. However, since this is a role, there isn't one assigned.

The EC2 role that is referenced is actually applied to the Splunk Enterprise instance and only has a policy assigned to monitor memory metrics and send to CloudWatch.