In my distributed environment, when I execute a search
index=_internal from the search head, I don't get any results back. When I log into an indexer directly and perform the same search, the expected results are returned. I can see from the "Indexes" pages that there are definitely events in the _internal index on each indexer, but I don't seem to be able to access them from the search head. I'm not really sure where to start to track down the issue with this. I only see the issue occur with internal (underscore) indexes, non-internal indexes work just fine from the search head. I suspect this is something simple that I am over looking.
What is the user/role capabilities you have? See if admin has allowed you to access internal indexes.
Settings>Access controls>Roles and looks for allowed indexes
Also look what role your user id is associated to.
Hope this helps,
Sorry, i meant to include that as part of the question. I have tried it both as 'admin' and a user mapped to the admin role. The admin role has "allowed indexes" set to "All Internal Indexes" and "All Non-internal Indexes". There aren't any additional restrictions set for the role.
It seems to have been an inheritance problem from another role. I created a new role with all indexes and all capabilities and no inheritance and it works now. I'll keep digging into the specific culprit.