Reporting

Search optimization and caching for forms

redc
Builder

I'm building a BI analytics app and am trying to do as much caching as possible because of the huge volume of data that each report has to run against (roughly 350MB and upwards of 200,000-300,000 events per search).

Unfortunately, almost every report has at least one swappable parameter - multi-select pulldowns, single-select pulldowns, text search fields, etc., created using Sideview - that needs to swap into the searches. Example:

index="my_index" $selectedToken$

The default for "$selectedToken$" is nothing/empty/null, so the default search that returns when you load the report would be:

index="my_index"

Then you can select something from a pulldown that will rerun the search with "$selectedToken$" populated, e.g.:

index="my_index" host="my_host"

I know if I try to run the search with the Sideview token swap tag in place in the search app (not swapped out for some real value/string), I'll get zero results, so a saved search with the token swap tags will always return no results. The sheer number of possible combinations for the various tokens make it impossible to create a saved search for every combination.

Is it possible to save the default search so that the default view will load the saved search very rapidly, then revert to real-time searching when the user selects something from one of the tokens? If that's possible, how is it accomplished?

sideview
SplunkTrust
SplunkTrust

There's no simple way. There are I think a couple very strange ways.

One that springs to mind is to use a couple Gate modules and apply autoRun a little cleverly.

Gate is a relatively new Sideview module, and it has two features. The first allows it to block pushes from upstream until some set of $foo$ tokens are present. The second allows it to kind of teleport pushes from it's location the module hierarchy to another Gate module elsewhere in the hierarchy. Here we will use the second feature only.

You could have your form element modules, and then downstream from them you have a Gate module with an "id" param of "insideFormElements". Then downstream from the Gate you have your tables and charts and whatnot.

Then, completely unrelated to those modules, up at the root level of the view you could have a SavedSearch module with a Gate module downstream. That Gate module would have a "to" param set to "insideFormElements". Put an autoRun="True" on the SavedSearch module and nowhere else and you've basically implemented the thing you need here.

When the page loads, the SavedSearch gets the scheduled job, and then teleports it through the gate past all the form search elements. Then when the user makes really any change in the set of form elements, that push would clobber the old push from the saved search results and you'd get an ad-hoc search dispatched automatically.

Use cases for the Gate module are always kind of brainbending, but it's an extremely handy module if used wisely.

As always the docs and the working examples for the Gate module are embedded in the Sideview Utils app itself. Gate released in 2.4.8 but it received pretty substantial changes in later releases so make sure you're on the most current Sideview Utils (2.7.1 as of this writing).

http://sideviewapps.com/apps/sideview-utils

UPDATE: Actually the Gate module idea doesn't work here, due to a wrinkle around autoRun="True". Pulldown modules need an autoRun="True" attribute somewhere upstream from them in order to render themselves when the page loads. However once you put that autoRun="true" attribute it will push on down through the rest of the page and your scheduled search results wont be used - instead the autoRun push will dispatch an ad-hoc search. Sorry for the waste of time!

0 Karma

sideview
SplunkTrust
SplunkTrust

I dont think postProcess will help here, although it is a powerful area of Splunk and worth becoming familiar with.

Indeed the Gate module as used here in my idea does vary a bit from the examples in Sideview Utils but that's OK. Sideview modules occasionally get used in unusual ways and they can take it. 😃

I think the best thing to do is read through the "Introduction to the Advanced XML" page thoroughly. That page is very long and dense but if you understand it then you really understand the Splunk module framework well. And at that point the Gate stuff will be much more clear.

0 Karma

redc
Builder

Ugh, I think I finally found the setting to email me when people make comments in reply to my comments on answers...I just now saw that you had replied.

I'll have to come back to this later. I think what I really need (first) is post-processing, which is something I still don't understand, either. It may be that I need post-processing as well as this.

I did find the samples in the app. I'm not entirely certain, from looking over the examples, that it'll do what I need (the inputs and the dashboard panels are a one-to-many relationship, not a one-to-one relationship).

0 Karma

sideview
SplunkTrust
SplunkTrust

Weird. links to the 2 docs pages are at the top level of the Module Documentation menu, so "Module Documentation > the Gate module > Gate - Blocking Searches" and "Module Documentation > the Gate module > Gate - Complex Drilldowns" .

If you dont have that item in your docs menu, even though you're on 2.7, one possibility is that you've refreshed the views since updating but you haven't restarted splunk or refreshed the nav? Go to the Freshmaker and click "nav" (ie /en-US/app/sideview_utils/refresh_entities)

0 Karma

redc
Builder

Is it literally called "Gate"? I'm having a tough time finding the examples in the app (I'm running v2.7.0 currently). Do you know which example it's running on in the app, by any chance?

0 Karma

redc
Builder

Each filter has its own token, so you might have a search param that looks like:

index="my_index" $selectedToken$ $selectedToken2$ $selectedToken3$

And the populated search would look like:

index="my_index" host="my_host" sourcetype="my_sourcetype" my_field="my_value"

(where $selectedToken$ swaps for host="my_host", $selectedToken2$ swaps for sourcetype="my_sourcetype", and $selectedToken3$ swaps for my_field="my_value")

0 Karma

somesoni2
SplunkTrust
SplunkTrust

you said you have multiple filter controls (drop-downs, text box). How is the selectedToken generated for each of them? I mean to ask is that if I select value from 1 dropdown how's selectedToken formed and if I select from 2 dropdowns, how it's formed?

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...