Best Way to Maintain Live Order Status Without Expensive Searches?

[martinb]

Hi all,

I'm new to Splunk and have been thrown in at the deep end, so apologies if this is the wrong place or a basic question.

We have a dashboard that tracks the status of orders (Started, Completed, Cancelled, etc.) coming from different source types and highlights red if it hasn't updated in some time. Right now, it runs multiple heavy searches with a lot of joins on OrderID to show if an order is running late. Since this dashboard is live and viewed by many people, performance is a concern.

What I'm looking for is a way to maintain a kind of “live” or frequently updated table (keyed by OrderID) that stores the latest status values (Started, Completed, Cancelled, etc.) and maybe a calculated (on-time, late etc,) so that dashboards or other searches don’t have to recompute everything each time.

Is there a recommended approach for this in Splunk? Summary indexing, KV store, or something else?

Thanks in advance!

Note: I'm using Splunk Enterprise 8.2

[PickleRick]

Adding to what @ITWhisperer already said - what do you mean by "live" dashboards? If you by any chance mean that the dashboard is running realtime searches... that's a bad idea.

[martinb]

By Live I meant more that the dashboard is used for monitoring, so needs up to date information on the order status, if there is a delay of more than around 5 minutes then they require intervention - so running a report every 10 minutes wouldn't work in this instance.  

But also yes, they are running searches each time the dashboard is refreshed. I've managed to limit the number of users to only 1 or 2 at this stage, which is why I'm trying to optimize this now rather than letting it become a problem for later.

[gcusello]

Hi at all,

At first, as @ITWhisperer said, try to optimize your search.

In addition, you could use a summary index (https://docs.splunk.com/Documentation/Splunk/9.4.0/Knowledge/Aboutsummaryindexing) or an accelerated custom Data Model (https://docs.splunk.com/Documentation/Splunk/9.4.2/Knowledge/Acceleratedatamodels) containing the data you need.

I had a similar situation some years ago: a dashboard used by many people with 12 searches in Real Time, some of them with sobsearches: this dashboard was killing the system.

I solved scheduling 12 reports every 5 minutes and displaying the results of these reports in the dashboard: in this way the 12 reports were executed only one time every 5 minutes instead in real time for each user, for more infos see at https://docs.splunk.com/Documentation/Splunk/9.4.1/Knowledge/Manageacceleratedsearchsummaries .

In addition you could accelerate these reports if possible (there are some limitations to report accelerations: https://docs.splunk.com/Documentation/Splunk/9.4.2/Report/Acceleratereports

Ciao.

Giuseppe

[ITWhisperer]

The first thing to do is probably optimise your searches so that they don't use joins - these are heavy as you have already pointed out.

Next, would depend on the tolerances you are prepared to deal with, for example, if you use a summary index, how out of date are you prepared for it to be,  e.g. if it was updated every 10 minutes, would that satisfy your users? Same goes for KV Store. (You would have a scheduled report that updated either the KV store or the summary index.)

[martinb]

Thanks for your reply! 

What is the best alternative to using joins? 

I think I could get away with up to 5 minutes delay, but less is better - as the dashboard is highlighting when orders are delayed so that users can intervene early and find the reason for the delay.

What if I ran a summary index every 5 minutes to only identify the orderID of the orders that are still open, and then running a base search to get the details in the dashboard for the ones that aren't - that way instead of running the search on every order that happened in the last x hours, we only run the search on the ones that aren't finished?

[ITWhisperer]

Adding to a summary index every 5 minutes with the latest state of any previously unclosed orders and any new unclosed orders sounds like a good way to go, however, this is all theoretical as we don't have access to your data or even a glimpse of what you events might look like.

Same applies to removing joins, without an idea of what your events look like, and what your current search queries look like, it is not possible to give a non-theoretical answer as to how you might optimise them, but, in general, the was to remove joins (and other types of subsearches) is to include all relevant sources of data (indexes) in the initial search, and use stats commands (e.g. eventstats, streamstats and stats) to correlate data from the different sources.

If you need further guidance, please share some anonymised events of the different types, representations of your current searches demonstrating how you are correlating the data, and an non-SPL explanation of what the searches are trying to do, then perhaps we might be able to provide even more useful suggestions.

[PickleRick]

There are many techniques for improving performance. Especially dashboard performance.

Apart from making the searches more efficient (getting rid of joins is almost always a good idea) you can use base searches in your dashboard if you generate several results from the same set of data. And you can do async searching - schedule searches in background and only do loadjob in the dashboard so that each user opening a dashboard doesn't spawn a new search.