Dynamic syncing of timechart zoom and time picker in dashboards?

[Graham_Hanningt]

Background to this question

I've developed some dashboards in Kibana (Elastic Stack 5.5). I've been asked to reproduce those dashboards in Splunk (6.6.3). Here, I'm using the term "reproduce" loosely: the Kibana and Splunk dashboards do not need to be identical. I can exploit unique features of each platform.

Kibana dashboards have the following default, out-of-the-box behavior: if you marquee-select (drag your mouse over) an area of a time-based chart, the time range of the entire dashboard—the time picker, and every visualization in the dashboard—changes ("zooms in") to match that selection. To zoom out to the previous time range, you click the browser Back button. For a demonstration of that behavior, watch my YouTube video "Splunk feature request: zooming one chart zooms entire dashboard".

I like that behavior. I want to reproduce it in Splunk. So do other users. I'm flexible about the method for zooming out, but I really want this "zoom one, zoom all; time picker in sync" behavior.

What I've tried

I've studied the "Pan and Zoom Chart Controls" example, but that doesn't update the time picker.

I'm using a global time picker that defaults to "All time":

<input type="time" searchWhenChanged="true">
  <label>Time range</label>
  <default>
    <earliest>0</earliest>
    <latest></latest>
  </default>
</input>

(Perhaps using a global time picker is one of my problems, but why? I want a "global" time range; one that applies to the entire dashboard. Do I really need to use a "local" time picker, and then set earliest and latest in every search to that particular token value?)

I tried adding the following to every timechart:

<selection>
  <set token="earliest">$start$</set>
  <set token="latest">$end$</set>
</selection>

but that resulted in "invalid earliest_time" errors: the earliest and latest tokens were being set to the literal string values "$start$" and "$end$", because the start and end tokens had not yet been defined; the <selection> element was being used (the selection event was firing) before the user did anything with the chart. This issue is described in more detailed in another question.

Then I tried replacing the <set> elements with <eval> (I wasn't sure from reading the documentation that this would be valid, or would work):

<selection>
  <eval token="earliest">if($start$="$start$",$earliest$,$start$)</eval>
  <eval token="latest">if($end$="$end$",$latest$,$end$)</eval>
</selection>

(That is, if the start and end tokens have not been defined—if the selection event has been fired without the user actually selecting an area of the chart—then set earliest and latest to their current values.)

That avoids the issue of setting earliest and latest to bogus values, but uncovers new problems:

• Clicking "Reset" in the chart that I "zoomed" does nothing. I guess I can understand that, because I've now set the global time to that zoomed selection.
• After zooming, selecting "All time" from the time picker, or removing earliest and latest from the URL parameters and reloading the page, produces unpredictable results. The time range can change to something that is neither all time, nor the previously zoomed time, but some other (unpredictable, to me) time range.

I've tried using a "local" time picker, and referring to the resulting $form.local.earliest$ and $form.local.latest$ tokens in the <earliest> and <latest>elements of all searches, and in the <selection> elements of time charts, but that doesn't have any effect on these problems.

Details of my use case

These particular dashboards show log data from "high-volume" transaction processing systems; systems that process many transactions per second. I have deliberately designed the dashboards, in Kibana and Splunk, to "work" regardless of the duration specified by the time picker: the X-axis of time-based charts automatically adjusts to match the duration. The same dashboards can be used to analyze logs across a wide variety of durations. In practice, users might initially be interested in a time range of several minutes, and then zoom in to analyze progressively narrower time ranges.

Summary

I'm bewildered that this behavior appears to be so hard to achieve in Splunk. I feel I must be missing something, but what? Are the typical use cases for Kibana (Elastic Stack) and Splunk so different that this out-of-the-box Kibana behavior is not also desirable in Splunk?

[rjthibod · Accepted solution]

OK, I have tried to give you a generic JavaScript solution.

You must copy the following JavaScript into a JS file (e.g., kibanareload.js), copy the file into the appserver/static folder of your Splunk app, restart Splunk, reference the file in the top of your SimpleXML file, and then add the custom selection handler.

What this JavaScript does is look for any pan & zoom handlers that edit specific tokens, kibana_pan_zoom.earliest and kibana_pan_zoom.latest . Here is what you need to include on any chart that should trigger the Kibana-like behavior for pan & zoom.

  <selection>
    <set token="kibana_pan_zoom.earliest">$start$</set>
    <set token="kibana_pan_zoom.latest">$end$</set>
  </selection>

Here is the full JavaScript file.

(function() {
  require([
    "underscore",
    "jquery",
    "splunkjs/mvc",
    "splunkjs/ready!",
    "splunkjs/mvc/simplexml/ready!",
  ], function(_, $, mvc) {

    function reloadPageWithNewTime(earliest, latest) {
      var k = location.href;
      var s = location.search.substring(1);
      var i = (s.length < 1 ? k.length : k.indexOf(location.search))

      var x = s?JSON.parse('{"' + s.replace(/&/g, '","').replace(/=/g,'":"') + '"}',
        function(key, value) { return key===""?value:decodeURIComponent(value) }):{}

      if (typeof earliest !== "undefined") {
        x.earliest = earliest;  
      }
      if (typeof latest !== "undefined") {
        x.latest = latest;  
      } 

      newSearch = Object.keys(x).map(function(k) {
        return encodeURIComponent(k) + '=' + encodeURIComponent(x[k])
      }).join('&')

      newUrl=k.substring(0, i) + "?" + newSearch;

      window.location = newUrl;
    };

    function setPanZoomTimer() {
      if (!pan_zoom_timer_is_on) {
        myVar = setTimeout(checkPanZoomBoundaries, 100);
        pan_zoom_timer_is_on = 1;
      }
    }

    function checkPanZoomBoundaries() {
      var defaultTokenModel   = mvc.Components.get('default');

      if (pan_zoom_timer_is_on) {
        pan_zoom_timer_is_on = 0;
      }

      var ts_earliest = defaultTokenModel.get("kibana_pan_zoom.earliest");
      var ts_latest = defaultTokenModel.get("kibana_pan_zoom.latest");

      if ($('[class*="btn-reset"]').length || $('button').find('[class^="resetZoom"]').length && ts_earliest > 0 && ts_latest > 0) {
        reloadPageWithNewTime(ts_earliest, ts_latest);
      }
    }


    /////////////////////////////////////////
    ///  Start Main Code Here
    /////////////////////////////////////////

    var defaultTokenModel   = mvc.Components.get('default');
    var pan_zoom_timer_is_on = 0;

    defaultTokenModel.on("change:kibana_pan_zoom.earliest", function(model, value, options) {
      setPanZoomTimer();
    });
    defaultTokenModel.on("change:kibana_pan_zoom.latest", function(model, value, options) {
      setPanZoomTimer();
    });
  });
}).call(this);

Here is the modified SimpleXML dashboard I shared earlier. It has been updated to use this new script. You can see in the top line how to reference the JavaScript file that has been copied into the app's appserver/static folder.

<form script="kibanareload.js">
  <label>Pan &amp; Zoom Test</label>
  <fieldset submitButton="false">
    <input type="time" searchWhenChanged="true">
      <label></label>
      <default>
        <earliest>-7d@h</earliest>
        <latest>now</latest>
      </default>
    </input>
  </fieldset>
  <row>
    <panel>
      <chart>
        <search>
          <query>
            | tstats prestats=t count where index=* by index _time span=1h
            | timechart span=1d count by index
          </query>
        </search>
        <option name="charting.chart">line</option>
        <selection>
          <set token="kibana_pan_zoom.earliest">$start$</set>
          <set token="kibana_pan_zoom.latest">$end$</set>
        </selection>
      </chart>
    </panel>
  </row>
  <row>
    <panel>
      <chart>
        <search>
          <query>
            | tstats prestats=t count where index=* by host _time span=1h
            | timechart span=2h count by host
          </query>
        </search>
        <option name="charting.chart">line</option>
        <selection>
          <set token="kibana_pan_zoom.earliest">$start$</set>
          <set token="kibana_pan_zoom.latest">$end$</set>
        </selection>        
      </chart>
    </panel>
    <panel>
      <chart>
        <search>
          <query>
            | tstats prestats=t count where index=* by sourcetype _time span=1h
            | timechart span=2h count by sourcetype
          </query>
        </search>
        <option name="charting.chart">line</option>
        <selection>
          <set token="kibana_pan_zoom.earliest">$start$</set>
          <set token="kibana_pan_zoom.latest">$end$</set>
        </selection>        
      </chart>
    </panel>
  </row>  
</form>

[Graham_Hanningt]

@rjthibod,

More niggles with narrow time ranges: if the time range is less than 2 seconds—if the difference between the values of the earliest and latest URL query string parameters is less than 2—then marquee-selecting an area of a timechart fails to trigger the custom "reload" behavior.

What I'm doing: you've given me the ability (thank you!) to progressively zoom in on timecharts. That ability led me to realize that the granularity of auto-span timecharts "bottoms out" at a minimum bucket duration of 1 second, hence my latest question. Finding a solution to that question led me to discover these niggles with your solution to this question.

[rjthibod]

@Graham_Hannington, did you get to try out the JS solution I posted above?

[Graham_Hanningt]

@rjthibod,

I've hit a problem when marquee-selecting a sub-second time range: the earliest and latest parameter values in the resulting query string don't accurately reflect the time range I marquee-selected in the timechart.

For example, if I select a half-a-second (0.5s) time range in a timechart—I know I'm selecting that time range, because of the tooltips on the data points within the area that I've selected—I end up with the following URL parameters:

earliest=1502692231&latest=1502692231.05

and no results in the newly loaded dashboard, even though the area that I marquee-selected contained multiple data points.

I discovered this problem while researching an answer to my recent question
Subsecond minspan in auto-span timechart?.

Your thoughts?

(I wish Splunk Answers showed a preview for comments!)

[Graham_Hanningt]

@rjthibod,

With apologies for the belated reply (I felt bad not getting back to you, but I'm under the hammer at work)...

Yes, I've tried the solution you posted above, and...

It works perfectly! Thank you so much!

Thank you especially for providing code that plays nicely with URLs that contain query string parameters in addition to earliest and latest.

I am happy to accept your answer as-is, but before I do that, I thought I'd ask: is it possible to apply this behavior to all time-based charts in a dashboard without explicitly specifying:

<selection>
    <set token="kibana_pan_zoom.earliest">$start$</set>
    <set token="kibana_pan_zoom.latest">$end$</set>
</selection>   

For example, is it possible in JavaScript to detect whether the query for a chart includes a timechart command, and then dynamically perform the equivalent of specifying that <selection> element in the Simple XML?

If so, then simply adding a script attribute to the dashboard Simple XML would confer this "Kibana-like" zooming behavior, rather than having to add that <selection> element to every "time-based" chart.

One reason I'm asking: some of my dashboards contain numerous timecharts. For example, one of my dashboards has 9 instances of Simple XML similar to this:

<chart>
    <search base="base_timechart">
        <query>fields _time avg(response)</query>
    </search>

(That is, the dashboard contains numerous timecharts, all based on a common base search.)

If it's not possible, or it's too difficult, or you disagree that it's worthwhile, or you feel—justifiably!—that you have already spent enough time answering this question, then I will happily accept your answer without further ado. Or I'll accept it as-is anyway, if you're prepared to add comments or edit the code in your answer later, after I've accepted your answer. (Is such "post-acceptance" editing of an answer allowed? I don't know.)

I'm wondering how to "spread the word" about your solution, over and above accepting your answer, because I think it will be useful to a lot of people. Certainly, I think that anyone who has developed dashboards in Kibana, and then moves to Splunk, will be interested in this.

Thank you so much for taking the time to consider my question and provide such a clear answer, with working code.

[rjthibod]

I will see if I can get it to work. It is probably going to result in a separate JS file because there are probably situations where my original version is wanted in addition to the version you want.

[Graham_Hanningt]

@rjthibod,

Even nicer would be if you could suppress the display of the now-unnecessary slider controls and Reset button, and reload immediately after the user finishes marquee selection. But I imagine that might require significantly deeper customization of the selection behavior; best left to the Splunk dev team deciding to support this behavior without custom JavaScript (I can hope). And it's not going to stop me accepting your current answer.

[rjthibod]

Suppressing those is definitely a much larger undertaking that would require overwriting internal Splunk objects. So, probably not going to happen.

[Graham_Hanningt]

@rjthibod,

Re:

Suppressing those is definitely a much larger undertaking that would require overwriting internal Splunk objects.

Yep. Prompted by @lfedak_splunk (thank you!), I've sent a feature request to Splunk via email.

[Graham_Hanningt]

@rjthibod,

Do you have any preference, or suggestion, for a JavaScript file name that does not explicitly include the term kibana? Yes, this JavaScript enables Kibana-like behavior—that is, after all, why I asked for it—but the fact that @lfedak_splunk redacted the term "Kibana" from my original question has sensitized me to the possibility that some people might not appreciate seeing that term in relation to Splunk. I don't want to offend anyone.

One idea:

reload_on_zoom.js

On the other hand, if users start including your code as-is, and the Splunk dev team starts noticing this code with its "kibana" qualifier cropping up frequently, that might prompt them to support this behavior out of the box.

(Splunk devs, I'm curious: are you monitoring the activity on this question?)

[lfedak_splunk]

Hey @Graham_Hannington, That wasn't redacted for sensitivity. Most people only look at the title to see if they might tackle a question and/or subscribe to tags, so I made it focus on the Splunk aspects of this question as this is a Splunk Answers forum. You'll notice that Kibana is still in your post several times. I looked into our forum and only saw one mention of Kibana, where it said Kibana could not ingest Splunk data, so I omitted it from the title. If you ever have a question about edits to posts you can comment (as you did), and, as @rjthibod points out, anyone is able to edit their own posts.

[rjthibod]

You can rename the files and token names to whatever you want. The moderators of the forum can bug me if they want.

[Graham_Hanningt]

@lfedak_splunk, I notice that you redacted "Kibana-like" from my original question. No worries about that; I don't want to annoy anyone by mentioning products-that-must-not-be-named. Still, I do wonder whether the Splunk dev team has any thoughts on this question. I'm not trying to stir up trouble; I really would like to see this behavior in Splunk, and I do wonder whether I'm missing something; for example, some reason why the Splunk devs have deliberately chosen not to offer this behavior (specifically, via Simple XML).

[lfedak_splunk]

Hey @graham_hannington, You can send feature requests to support@splunk.com 🙂

[Graham_Hanningt]

Hi @lfedak_splunk,

I sent an email to support@splunk.com. Here's what happened next...

I received a "do-not-reply" acknowledgement email:

The support mailbox received the following Community Support case
Priority: P3
Case: 541832
Subject: Splunk feature requests: 1. Zooming one chart zooms entire dashboard, 2. Auto-spanning for timecharts with sub-second time ranges.

followed by another (without the "do not reply" text):

As our priority is with entitled customers, we can only provide our best effort and delay is expected. We suggest that you post your issue in Splunk Answers

I replied to that, and immediately received a reply with this:

Your email is associated to the CLOSED Splunk Support CASE [541832]

For what it's worth, here's my second email to Splunk Support (the one that resulted in the "associated to (sic) the CLOSED..." message)...

---

RE: CASE [541832] 1. chart zooms entire dashboard, 2. Auto-spanning with sub-second time ranges

Hi Splunk Support,

Thanks for the response.

Re:

our priority is with entitled customers

No worries at all, I completely understand. I expect the support I pay for ;-).

Re:

We suggest that you post your issue in Splunk Answers

Already done, long before emailing you. In fact, I was prompted to email you by a Splunker commenting on a question that I had asked in Splunk Answers. I guess your suggestion means that you haven't yet had the chance to read down to the fourth line in my original email to you. Again, no worries.

To recap:

• I already have dashboards with the behavior that I want. In Kibana.
• I've been asked to reproduce those dashboards in Splunk.
• It's likely these Kibana and Splunk dashboards will be demo'd alongside each other, or at roughly the same time, to customers. (For brevity, I'm not going in detail about who I mean by "customers", but I can expand on that term, if asked.)
• The customers will see—or experience for themselves, first-hand—Kibana dashboards dynamically zooming down to millisecond granularity, by marquee-selecting within any time-based chart.
• Thanks to the help of a Splunk user who provided an answer on Splunk Answers, these customers might get to see something like that in Splunk, too; although, as I've noted in comments on Splunk Answers, the user experience will not be as slick, and the current answer is not usable at time ranges under 2 seconds (so I've not yet implemented the code provided in that answer to all of my dashboards).

Perhaps my experience is atypical, but for what it's worth: when I show someone a Splunk dashboard—or I chat to someone about their first experience developing and then using a Splunk dashboard—it's common for them to attempt to zoom in on a narrower time range by marquee-selecting in a time-based chart. And they're taken aback when it doesn't "work"; when it doesn't do what Kibana does, out-of-the-box. And then I have to explain—or the developer has had to independently find out for themselves—that reproducing that behavior in Splunk is not straightforward; in fact, to my knowledge, there is currently no good solution. Hence my feature requests. I can tell that many users flat-out don't believe me when I try to explain this. So, in a way, my experience on Splunk Answers has been reassuring.

I regard sending you these feature requests as performing "due diligence", following through on a request from my management to reproduce Kibana dashboards in Splunk.

Please don't worry at all about considering these feature requests just to satisfy me, a non-paying user. I wouldn't expect you to. 🙂

Instead, think about a user who has seen a Kibana dashboard progressively zooming in from a time range of, say, an hour, down to several milliseconds. Then think about them attempting to reproduce that experience in Splunk.

Is this really such an atypical use case for Splunk?

[Graham_Hanningt]

Hi @lfedak_splunk,

Uncanny timing. I was just about to go looking for a way to send feature requests when I saw your comment. I'll send an email now. Thanks for the tip!

[niketn]

@Graham_Hannington, following is the scenario with a run-anywhere dashboard based on Splunk's _internal index, with zoom in selection and zoom out, I was trying to mention.

You can actually try out Splunk Search Timeline as well for similar kind of selection behavior (check out Splunk Docs: http://docs.splunk.com/Documentation/Splunk/latest/Search/Usethetimeline). PS: For exact behavior it is better to handle using Simple XML JavaScript Extension or HTML Panel.

<form>
  <label>Timechart selection event handler override</label>
  <!-- Dummy Search to Set Earliest Latest Epoch time based on Time Input Picker -->
  <search>
    <query>| makeresults
    </query>
    <earliest>$tokTime.earliest$</earliest>
    <latest>$tokTime.latest$</latest>
    <done>
      <eval token="tokEarliest">strptime($job.earliestTime$,"%Y/%m/%d %H:%M:%S.3N %Z")</eval>
      <eval token="tokLatest">strptime($job.latestTime$,"%Y/%m/%d %H:%M:%S.3N %Z")</eval>
      <set token="tokEarliestString">$job.earliestTime$</set>
      <set token="tokLatestString">$job.latestTime$</set>
    </done>
  </search>
  <fieldset submitButton="false"></fieldset>
  <row>
    <panel>
      <input type="time" token="tokTime" searchWhenChanged="true">
        <label></label>
        <default>
          <earliest>-24h@h</earliest>
          <latest>now</latest>
        </default>
      </input>
      <input type="link" token="tokZoom" searchWhenChanged="true">
        <label></label>
        <choice value="zoomin">+ Zoom in</choice>
        <choice value="zoomout">- Zoom out</choice>
        <change>
          <condition value="zoomin">
            <set token="form.tokTime.earliest">$selectionStart$</set>
            <set token="form.tokTime.latest">$selectionEnd$</set>
            <set token="tmpTokEarliest">$tokEarliest$</set>
            <set token="tmpTokLatest">$tokLatest$</set>
          </condition>
          <condition value="zoomout">
            <set token="form.tokTime.earliest">$tmpTokEarliest$</set>
            <set token="form.tokTime.latest">$tmpTokLatest$</set>
          </condition>
        </change>
      </input>
      <chart>
        <title>Use Pan and Zoom to Select Time Ranges Used By Other Visualizations</title>
        <search>
          <query>index=_internal sourcetype=splunkd log_level=* component=* 
          | timechart count</query>
          <earliest>$tokTime.earliest$</earliest>
          <latest>$tokTime.latest$</latest>
        </search>
        <option name="charting.chart">line</option>
        <option name="charting.legend.placement">none</option>
        <option name="charting.legend.masterLegend">null</option>
        <option name="charting.seriesColors">[0x1D2F3E]</option>
        <option name="charting.axisTitleX.visibility">collapsed</option>
        <option name="charting.axisTitleY.visibility">collapsed</option>
        <option name="charting.axisTitleY2.visibility">visible</option>
        <option name="height">200</option>
        <selection>
          <set token="selection.earliest">$start$</set>
          <set token="selection.latest">$end$</set>
          <set token="start.count">$start.count$</set>
          <set token="end.count">$end.count$</set>
          <set token="selectionStart">$start$</set>
          <set token="selectionEnd">$end$</set>
          <set token="form.tokZoom=zoomout"></set>
        </selection>
      </chart>
    </panel>
  </row>
  <row>
    <panel>
      <title>Count of Components split by Log Levels - Pie Chart</title>
      <chart>
        <search>
          <query>index=_internal sourcetype=splunkd log_level=* component=*
          | stats count by log_level,component</query>
          <earliest>$tokTime.earliest$</earliest>
          <latest>$tokTime.latest$</latest>
        </search>
        <option name="charting.chart">pie</option>
        <option name="charting.drilldown">all</option>
        <option name="trellis.enabled">1</option>
        <option name="height">200</option>        
      </chart>
    </panel>
    <panel>
      <title>Log Levels - Info over Error Warn or Fatal - Timechart</title>
      <chart>
        <search>
          <query>index=_internal sourcetype=splunkd log_level=* component=*
          | timechart count by log_level</query>
          <earliest>$tokTime.earliest$</earliest>
          <latest>$tokTime.latest$</latest>
        </search>
        <option name="charting.axisTitleX.visibility">collapsed</option>
        <option name="charting.axisTitleY.visibility">collapsed</option>
        <option name="charting.axisTitleY2.visibility">visible</option>
        <option name="charting.axisY2.enabled">1</option>
        <option name="charting.chart">column</option>
        <option name="charting.chart.overlayFields">INFO</option>
        <option name="charting.drilldown">all</option>
        <option name="charting.legend.masterLegend">null</option>
        <option name="charting.legend.placement">none</option>
        <option name="height">200</option>
      </chart>
    </panel>
  </row>
</form>

[Graham_Hanningt]

Thanks very much for taking the time to read and respond to my question.

I've pasted the source you provided into a new dashboard and played with it.

Unfortunately, it's not what I'm after. While I appreciate the ingenuity of your approach—the zoom in/out inputs—I think you'll agree that the user experience is overcomplicated compared to the Kibana behavior demonstrated in that YouTube video.

I've read the Splunk docs topic "Use the timeline to investigate events", but—with apologies if I'm being obtuse—I cannot see how to apply the information in that topic to achieve the result I'm after.

[niketn]

@Graham_Hannington, the reason why I had asked you to refer to try out "Timeline" topic was because the Zoom in and Zoom out approach I adopted was from Splunk's Timeline behavior. One of the things with Zoom in Zoom out approach is that entire dashboard does not refresh (only required panels refresh) as opposed to Kibana.

However, if you need the page to submit the selection value and refresh the dashboard you would need to use Simple XML JS Extension, where you can combine Splunk JS Stack and jQuery to implement the UI behavior you need.

[Graham_Hanningt]

@niketnilay,

Re:

you would need to use Simple XML JS Extension, where you can combine Splunk JS Stack and jQuery to implement the UI behavior you need.

Yes, you are absolutely correct: @rjthibod has provided an answer with working code that matches this description.