Why are Dashboard Panels Stuck on "Waiting for data..." for Non-Admin Users?

[nextpart]

I'm facing a rather peculiar issue with dashboards. When non-admin users, or users without the admin_all_objects capability, access the dashboard, all panels display "Waiting for data..." indefinitely. However, the strangest part is that if the user clicks on the search of a panel and is redirected to the search view, the results appear immediately.

Here's what I've tried so far:

• Searched through community questions and issues, but found nothing that matches this issue exactly.
• Experimented with different capabilities, but it seems only the admin_all_objects capability solves this issue.
• Attempted to adjust the job limits similar to those set for admin users.

Assigning admin_all_objects capability to all users is not a viable solution for me due to security concerns.

Has anyone encountered this issue before? I'm running out of ideas and would appreciate any help or insights on this.

Note: Tested also on a local instance deployed via ansible-role-for-splunk to reproduce.

Thank you in advance for your time and assistance.

[bowesmana]

Is the dashboard search using tokens in the search?

[nextpart]

Yes, it is. But I just tried the same with a dashboard that's not using any type of token in a base search or somewhere else, facing the same issue.

[bowesmana]

Ah... you mention base search.

I have seen an issue with post processing searches showing Waiting for data when a search re-runs but only the post process part has a changed criteria and the base does not - there is some additional token complexity in there too.

Is this relevant to your case?

The behaviour is similar in that opening the search in a new window works.

I currently have a workaround where I ensure the post process search forces use of a field that is not actually required, but it makes the search run.

I haven't tracked this down, but suspect it's a bug as I am generally admin when I see this.

[nextpart]

Unfortunately, this is not relevant for this specific case, as it's not possible to run any simple search in the dashboards.

But when trying to run the same searches in a dashboard built with dashboard-studio I now immediately get an error message instead of infinite waiting for data:
"Search new_test_user_bmV3X3Rlc3RfdXNlcg__search__RMD5149cadac0aee6cd6_1693919913.13912 not found. The search may have been cancelled while there are still subscribers."