topic Re: Dashboards - concurrency in Dashboards & Visualizations

Dashboards - concurrency

serialmonkey — Sat, 31 Jul 2010 19:26:20 GMT

I've build a dashboard which contains 6 single-value queries. For admin, this is fine, but for general users they are getting caught with the limit of 3-concurrent queries at a time. This is causing the page not to fully load for a standard user.

Obviously one solution is to up the concurrency limit for the user.

BUT, is there any way I can alter the dashboard so it doesn't try and run all the queries concurrently ? I wouldn't care if it ran them 1 or 2 at a time and slowly built the page, as long as the page fully rendered in the end.

Re: Dashboards - concurrency

hexx — Sun, 01 Aug 2010 04:11:29 GMT

As you have found out, searches initiated to render dashboard panels are subject to the value of srchJobsQuota defined for the role assigned to the user running the dashboard.

I am not aware of a configuration option for dashboards in simpleXML which allows to run the panel searches sequentially.

You could possibly get around this restriction by re-thinking how you generate the search results used to build your dashboard panels. For example :

You could use "cached" results from a scheduled search for one of your panels. The search results from that panel will therefore not count agains srchJobsQuota.

http://www.splunk.com/base/Documentation/latest/Developer/DashboardIntro#Searches_and_dashboards http://www.splunk.com/base/Documentation/latest/Developer/SavedSearchesViews

If applicable, you can use one "root" search to generate results that you can then process through several post-process searches, each resulting in a different dashboard panel. There are some restrictions to the usage of searchPostProcess, but if it's applicable to your dashboard it will save your indexer(s) a lot of resources running fewer searches and the display of your dashboard will be faster.

http://www.splunk.com/base/Documentation/4.1.4/Developer/PostProcess

Re: Dashboards - concurrency

gkanapathy — Sun, 01 Aug 2010 04:32:45 GMT

I thought there was some configuration actually in the UI such that searches that could no be dispatched immediately due to quota would be held for a short period (a few seconds) to fill in a dashboard. It's possible maybe your jobs take too long for this to work, or that it doesn't apply to role quotas (only systemwide job limits).

Re: Dashboards - concurrency

gkanapathy — Sun, 01 Aug 2010 04:33:16 GMT

Sorry not that helpful, but it's a vague memory I have that this worked that way.

Re: Dashboards - concurrency

serialmonkey — Sun, 01 Aug 2010 19:05:19 GMT

Thanks for the suggestion. I've tried taking the 'HiddenPostProcess' approach, but I'm coming up stuck.

Is there any restriction in using the 'where' command in a HiddenPostProcess ?

For instance, by base search looks like

sourcetype="mytype" (with the appropriate XML-style quotes)

then my postprocess command is

|where field="value"

Doing this in a standard search works fine, but in a postprocess just always give me no results.

Re: Dashboards - concurrency

hexx — Tue, 03 Aug 2010 05:17:58 GMT

I think you are hitting into one of the limitations of PostProcess : Essentially, you shouldn't set your main search to pass a set of events to your PostProcess(es).

Check this section of the Developer Manual : http://www.splunk.com/base/Documentation/4.1.4/Developer/PostProcess#Construct_your_base_search

I recommend to restructure your base search so that it passes the information needed for your PostProcess as a table, typically by using "stats count by , , ". Make sure you include all of the fields you intend to build your statistics on.

Re: Dashboards - concurrency

serialmonkey — Tue, 03 Aug 2010 20:09:11 GMT

Unfortunatly I cannot see how todo that in my particular case. I'm trying to build a page which shows the uptime of our 6 applications. The base query is returning a list of 6 events - one for each application. Even if I were to build this as a table, how can I structure size seperate SingleValue PostProcess modules to extract each row one at a time ?

Thanks for the help.

Re: Dashboards - concurrency

hexx — Wed, 04 Aug 2010 01:42:44 GMT

It's difficult for me to answer that without looking at the structure of your events and also without knowing what you want to display with your postProcess.
If you use splunk.pastebin.com to post the raw data of your events (also indicating the fields you're interested in), I can try to comment more accurately on your use-case.

Re: Dashboards - concurrency

serialmonkey — Wed, 04 Aug 2010 20:23:47 GMT

Thanks, your pointers pushed me in the right direction. Formatted the exact same information as a table instead of an event (using some non-sensical stats commands) and everything is well. Thanks !

Re: Dashboards - concurrency

hexx — Wed, 04 Aug 2010 23:52:51 GMT

Glad to hear that 🙂
I'll try to touch up the docs a bit so that the restrictions of searchPostProcess are easier to understand and overcome.