Dashboards & Visualizations

Why do some panels show "Waiting for data..." while loading multiple dashboards in Splunk 6.1.2?

HattrickNZ
Motivator

If i open up multiple dashboards on my machine, some panels show "waiting for data".
Basically, I am getting the below for certain panels i have in my dashboards. but there is no indication to me that there was a limit hit.

Image and video hosting by TinyPic

I am trying to understand what this is. Is it a licensing issue? if so how do I find out what it is? And is there away around it? Or at lease is there a way i can notify the user that a limit has been hit?

is there a general rule about how many panels to have in dashboards based on this restriction?

Tags (3)
0 Karma

chimell
Motivator

Hi HattrickNZ

Firstly , note that there are not a restriction about the number of panels in a dashboard . The restriction is about the number of search in your panel request , the limit is 8 searches .

When you enter the following request in one of your panel dashboard you must wait about one Day before seeing the results appear , because you specified that earliestTime = -1d@d.

look at this

 <table id="table1">
            <searchString> index=_internal | timechart span=1h count as total | appendcols [ search index=_internal (sourcetype=splunk_web_access OR sourcetype=splunkd_access) | timechart span=1h count by sourcetype ] | table _time splunk_web_access splunkd_access total </searchString>
            <earliestTime>-1d@d</earliestTime>
            <latestTime>@d</latestTime>
        </table>

In conclusion i advise you to revise your earliestTime

0 Karma

HattrickNZ
Motivator

does 8 search = 8 panels limit? or can you somehow reuse a search?
In relation to earliestTime, what should I set it to? Using the sample data below, If i have more columns, does this mean I have to reduce the number of Time datapoints I can use? Is this what you are getting at?

samle dataset

Time, networkElement, column1,column2,...columnN
1
....
0 Karma

esix_splunk
Splunk Employee
Splunk Employee

This isnt a licensing issue. Splunk licensing is based on indexed volume, not data displayed or searches run.

Most likely this has to do with the contents of the dashboard and system resources. In environments where dashboards have a lot of panels, its recommended to use populating saved searches or summary indexes to power the dashboards. This prevents a search from running everytime you load the dashboards, and eliminates the concurrency or cpu / memory related issues that can come up.

0 Karma

dmr195
Communicator

Is any effort being made to fix the "concurrency or cpu / memory related issues that can come up"? We have done some investigation and it seems to be that there is a race condition somewhere in splunkd that means there is a chance that it tells SplunkJS that there are no running searches related to a dashboard when in fact there are. Then SplunkJS never asks again about the searches that it thought were running after being told there are no running searches related to the dashboard. I can totally believe this is a concurrency issue, as we only ever see the problem when there are multiple dashboard panels. Also, it is related to timing in some way as adding a custom search command to the searches on the dashboard makes the problem far more likely to manifest itself. And finally, it's much worse on Windows than other platforms. We find that when you have a dashboard with several panels and searches containing custom search commands on Windows some of the panels always freeze up "waiting for data". It's not really a great response to just disown the bug because it's due to "concurrency or cpu / memory related issues".

0 Karma

HattrickNZ
Motivator

how do I use populated saved searches or summary indexes to power the dashboards?
If i run a search and then i want to save it i only have the option to save it as a panel or a report.
index=core host="host" userLabel=*UL* elementType=ET1 measInfoId=1693 duration=15min | timechart span=15m max(KPI1) by measObjLdn

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...