How to place the Pulldown below the main search and populate it with a new search?

[greg]

I'm trying to implement the following advanced view:

This is a mockup, drop-down boxes A,B,C,D will be populated with values from 4 different sourcetypes. The main data is taken from 5-th sourcetype to be filtered out by pulldown selection on each chart.

The tricky thing here is that I want to place Pulldown modules on each panel right next to charts and use them as post-process parameters. Basically (what I have seen in examples) we place Pulldown modules one by one above the main search module and then use the values chosen by user as $foo$ parameters. In my case, I want firstly to execute the main search and then to post-process its results passed downstream to panels, displaying charts based on selection in the Pulldown.

It will be faster (no need to execute main search for the sake of some final filtering) and more user-friendly (a user can choose filters right on each panel, say, if panel count is 10 or more and you have to scroll the web-page). The drop-down boxes are filled in by new searches, each on different sourcetype. I can't combine these searches with the main one, because sourcetypes and output data are different.

The problem is that if I create a new search for each pulldown, postProcess module below no longer sees the results of upstream main search. Looks like new search cleans it up or overrides somehow.

E.g. the postprocess in code sample below

$postProcess$ | where Percentage == “$PercentA$” | table Host, Percentage

doesn't work and produces no data. I want it to inherit the results from main search.

Is there any way to rearrange the modules or use any workaround?

I'm using Sideview Utils 2.2.6.

--

The view structure:

<module name="Search" autoRun=" True">
  <param name="search">
    sourcetype="Main" … | fields Host, Percentage
  </param>
   … 
  <module name="Search" autoRun="True">
    <param name="search">sourcetype="ForPulldownA" | stats count by PercentA</param>

    <module name="Pulldown">
      <param name="name">PercentA</param>
    …
      <module name="PostProcess">
        <param name="search">
          $postProcess$ | where Percentage == “$PercentA$” | table Host, Percentage
        </param>
        <module name="HiddenChartFormatter">
    …
</module>

<module name="Search" autoRun="True">
  <param name="search">sourcetype="ForPulldownB" | stats count by PercentB</param>
  <module name="Pulldown">
    <param name="name">PercentB</param>
  … 
    <module name="PostProcess">
    <param name="search">
      $postProcess$ | where Percentage == “$PercentB$” | table Host, Percentage
    </param>
    <module name="HiddenChartFormatter">
  …
</module>

[sideview · Accepted solution]

If I understand correctly, what you're doing with each panel looks like this, in sort-of pseudoXml:

Main Search
  Search for Pulldown 
    Pulldown 
      PostProcess
        Chart to render main search. 

But you have a catch-22., Obviously you need the "Search for Pulldown" search to be upstream from the Pulldown or else the Pulldown cant render it's options, but you also want the MainSearch to be upstream from the Pulldown because you dont want changing the Pulldown selection to redispatch the main search.

As you've guessed, the Splunk UI only allows there to be one search at any given point in the hierarchy, and if another one is set, the preexisting one inherited from upstream is forgotten. (Sideview Utils actually makes this a bit stronger and throws away the postProcess search at the same points but that's another story. )

There are some ways out of this problem.

-- it's probably not an easy option, and maybe not an option at all, but you can combine the searchterms of the two searches in a big OR expression, then get clever with stats, then use postProcess for BOTH the Pulldown and the chart results. In effect then both the Pulldown and the Chart are rendered from the same base search result. Even if the two datasets are wildly different from eachother, this is usually possible, although I'm not going to say it's intuitive. 😃 Email me with the two searches and I can show you how to combine them into one 'datacube' search and then carve them back apart into two search results with postprocess.

-- write a Sideview CustomBehavior that actually squirrels away the search object above our little paradox, puts it back into the context as some other key like "secretSearch". Then have a second customBehavior that does the reverse. This would effectively create a little wormhole whereby your upstream search could survive the journey through the Pulldown's search.

The bad news is that CustomBehavior is a relatively advanced piece of Sideview Utils and it requires you write a little javascript. the good news is that the particular JS in question is simple enough that I've written it for you here:

Sideview.utils.declareCustomBehavior("stashMySearch", function(customBehaviorModule) {
    customBehaviorModule.getModifiedContext= function() {
        var context = this.getContext();
        // this is actually an object, not just a string, which is 
        // incidentally why we can't just use the ValueSetter module 
        // to do this trick...
        var search  = context.get("search");
        context.set("stashedSearch",search);
        return context;
    }
});


Sideview.utils.declareCustomBehavior("unstashMySearch", function(customBehaviorModule) {
    customBehaviorModule.getModifiedContext= function() {
        var context = this.getContext();
        var search  = context.get("stashedSearch");
        context.set("search",search);
        return context;
    }
});

Then in your XML you'd have

<module name="CustomBehavior">
  <param name="customBehavior">stashMySearch</param>

at the point where you want to hide the search results from upstream.

and below the Pulldown you'd have:

<module name="CustomBehavior">
  <param name="customBehavior">unstashMySearch</param>

which would "unstash" the previously hidden search results. Oh and there are docs for CustomBehavior but because it's a bit of a loaded gun I made them hard to find. Look in the "Tools" page and you'll find a link to them.

3) You could lobby Sideview to write one or more modules to provide this mechanism. It could either automatically stash the successive layers, and then have some module that can pop one or more layers off the stack and get you back to where you were, or there can be two modules one that stashes and one that unstashes. It certainly would need a lot more thought to be done correctly, but feel free to lobby me. 😃

UPDATE: There is technically another way. Sideview Utils makes a lot of new and useful $foo$ keys available (and there's an overview of all of these in the Sideview Utils docs at "Key Techniques > Other > Overview of all the $foo$ tokens"). One of these is $results.sid$ which is the search id of the current search results. You could use the ValueSetter module to stash this little identifier into a key called like 'secretSearchId', and then further down the page you could use the loadjob command, and pass that search command, this search id. There can be some sort of 'frankenstein' aspects to this - in that a search result set reconstituted from loadjob does not always act the same as the original search result set, but it can often work well.

[sideview]

You should also be careful of autoRun="True". Nesting autoRun's like that can cause dispatch-> cancel -> dispatch sequences, or can cause little storms of HTTP requests that are immediately aborted and then re-requested. I can easily imagine that IE would react poorly to such a thing, and at any rate it's an easy thing to test. Never nest an autoRun="True" inside another one.

[sideview]

With most cases of a slow running browser suspicion falls on the UI but ends up on less sexy causes. Sometimes it's just too many expensive searches running, and/or underspec hardware such that SplunkWeb itself becomes almost unresponsive during page load. A lot of the time postProcess searches are set up that are just extraordinarily expensive such that they take many seconds to run. However if we're just talking IE, I have seen some terrible slowdown issues when rendering thousands of elements into Pulldowns. Why the limit is so low only sometimes on IE I haven't found out yet.

[greg]

Well, this slowness is only a problem in Internet Explorer, checked this out for many times today. In Opera and Chrome the page has been loaded for a couple seconds. In addition, search+postprocess in a separate search window work quite fast. Looks like a browser-specific issue in JavaScript or flash, have you ever seen it in advanced XML views?

[sideview]

Sorry about the typo - it has been fixed. If there's slowness I think it will be from the characteristics of the postprocess and the base search. Postprocess is great but if you don't carefully step around the pitfalls it can be very inefficient. Maybe post one of the searches and the postprocesses as a separate question?

[greg]

Noticed one side effect - my view now loads visually "slower", web-page behaves weird during processing (it hangs, no job progress, no UI responding - and after a while it instantly comes up with fully rendered charts). This one is with Internet Explorer only (in Chrome/Opera I see progress gear is rolling), guess it's because flash loading specific in IE.

[greg]

Thank you for such a detailed reply, appreciate it!
First thing I tried was approach #2 since I'm already familiar with "old trick #1" to suppress some of splunk warnings. It works! but please correct a misprint in your code snippet (declareCustomBehavior("stashMySearch", customBehaviorModule) -> ...function(customBehaviorModule), and for "unstashMySearch" as well, am I right?)
I will also try to compose search #1 (guess I got your 'datacube' idea), maybe things be simpler this way (I will email you if I get stuck).

[sideview]

btw, this was a known problem before we released 4.0, and the general solution space was loosely labeled the "multicontext" idea. However we weren't sure if it was going to be a big issue, or if it was going to be something that really didn't matter that much. We guessed the latter. Since that was many years ago and it's only come up a few times, I'm going to say we guessed right. 😃