Dashboards & Visualizations

Why is a group of Splunk users in one location unable to view all expected results on a dashboard compared to users in another location?

Explorer

We have a dashboard, the results of which are seen perfectly correct by one usergroup (from India) and the results are distorted in the other group (from USA). Both have the same search quota and everything are the same, except that the user group in India has access to many dashboards unlike that in the USA user group. Also, both the usergroups have access to all of the indexes which are required for that particular dashboard. What might be the reason for the other usergroup who are not able to view the correct output in the dashboard? Is it something related to the location issues?

0 Karma
1 Solution

Splunk Employee
Splunk Employee

I would start by ensuring that all knowledge objects are scoped properly to the apps in which they need to be visible. This definitely sounds like your user group in the USA does not have permissions for some knowledge objects needed to properly populate the dashboard.

View solution in original post

0 Karma

Splunk Employee
Splunk Employee

I would start by ensuring that all knowledge objects are scoped properly to the apps in which they need to be visible. This definitely sounds like your user group in the USA does not have permissions for some knowledge objects needed to properly populate the dashboard.

View solution in original post

0 Karma

Explorer

Hi @ssievert. We are running splunk 6.3.4. We are planning to raise this issue with Splunk support this week.

0 Karma

Splunk Employee
Splunk Employee

Did you recently upgrade from an earlier version of Splunk to 6.3.x? There are some cases where this specific upgrade resulted in some buckets being compromised, but I cannot say for sure if this is what is happening for your as well.
Do you get consistent results when running searches repeatedly over the same time span?
Do the issues exist if you search across a very recent time period (like last 24hrs)?

I do see one internal bug report of something that sounds very similar and the issue was triggered by an unclean shutdown or crash of Splunk. This specific bug was fixed in 6.4.x, but - as I said - your issue may be sightly different. Support will be able to verify that.

0 Karma

Explorer

Sorry for the delay:
Yes we have recently move to 6.3 from 6.2,
And we get inconsistent results when running the searches repeatedly over the same time span.

We have raised for support today, should be able to get some help

0 Karma

Explorer

The issue was regarding the query optimization (ineffective usage of subsearches)

0 Karma

Explorer

@ssievert thanks for the reply. I have verified all the knowledge objects and their permissions, they are properly scoped. I took the search string for that dashboard, and ran in the search app of both user groups.
Still we are getting different results in both the Usergroups. And after reviewing from the input data, I can say that the results of the both the usergroups are incorrect.
Also, I have realized that we are getting this below warning (both in the search app and in the dashboard), which I think is one of the reasons for not getting correct output.
[subsearch]: [lgpbd022g.gso.abcd.com] Streamed search execute failed because: JournalSliceDirectory: Cannot seek to 0
Corresponding Search log:
INFO RetryManager - Peer="lgpbd0227.gso.abcd.com" at generation="1718653" was temporarily missing from the set of search peers. If the peer goes down while its information is missing, the search can return incomplete results. To remediate, run the search again.

I could not find the solution for this warning anywhere in the Splunk docs. Any help from your end will help us, as the users are facing this issue from couple of weeks. Please advise if you need anything more from my end for resolving this issue. Thanks in advance

0 Karma

Splunk Employee
Splunk Employee

What version of Splunk are you running? Did you open a case with Splunk Support for this?

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!