Splunk Enterprise upgrade to 9.1.0.1, all users disappeared

[tlmayes]

Upgraded several independent instances of Splunk Enterprise from various starting points, all to 9.1.0.1.   Some clustered, some standalone.

• 8.1 -> 9.1.0.1
• 9.0.1 -> 9.1.0.1

All had the same outcome:  When browsing to: Settings > Users and Authentication > Users, most but not all users are no longer visible in the 'Users' list, but the users still have access as validate by Splunk logs.  In the most severe case there were 100+ users, mostly SAML, some local.  Post upgrade there are 4 showing, yet in validation all can still login

[PickleRick · Accepted solution]

I used to have a similar problem (but at some earlier version) due to wrong entries in authorize.conf

If I remember correctly, it had something to do with a role having set edit_roles_grantable privilege but not having defined grantableRoles parameter. User with such role would not show in the users list but would still be able to authenticate to web interface and use the system normally.

[tlmayes]

PickleRick, seems you were right, and thanks for the response. 
There was a bug reported in 2019, that in my opinion is back with v9.1.0.1. Reference: https://community.splunk.com/t5/Security/Admin-can-t-see-users-with-a-certain-role-and-we-can-t-take... 

Adding all roles to 'grandableRoles' solved the problem.  Consider this a bug since the problem appeared immediately on several deployments, all unrelated to each other, that all worked fine immediately preceding upgrade.  

[VatsalJagani]

@tlmayes - I don't see any known issues, hence I would say create a Splunk support case.

I hope this helps!! Consider upvoting!!!