Dynamic syncing of timechart zoom and time picker in dashboards?

[Graham_Hanningt]

Background to this question

I've developed some dashboards in Kibana (Elastic Stack 5.5). I've been asked to reproduce those dashboards in Splunk (6.6.3). Here, I'm using the term "reproduce" loosely: the Kibana and Splunk dashboards do not need to be identical. I can exploit unique features of each platform.

Kibana dashboards have the following default, out-of-the-box behavior: if you marquee-select (drag your mouse over) an area of a time-based chart, the time range of the entire dashboard—the time picker, and every visualization in the dashboard—changes ("zooms in") to match that selection. To zoom out to the previous time range, you click the browser Back button. For a demonstration of that behavior, watch my YouTube video "Splunk feature request: zooming one chart zooms entire dashboard".

I like that behavior. I want to reproduce it in Splunk. So do other users. I'm flexible about the method for zooming out, but I really want this "zoom one, zoom all; time picker in sync" behavior.

What I've tried

I've studied the "Pan and Zoom Chart Controls" example, but that doesn't update the time picker.

I'm using a global time picker that defaults to "All time":

<input type="time" searchWhenChanged="true">
  <label>Time range</label>
  <default>
    <earliest>0</earliest>
    <latest></latest>
  </default>
</input>

(Perhaps using a global time picker is one of my problems, but why? I want a "global" time range; one that applies to the entire dashboard. Do I really need to use a "local" time picker, and then set earliest and latest in every search to that particular token value?)

I tried adding the following to every timechart:

<selection>
  <set token="earliest">$start$</set>
  <set token="latest">$end$</set>
</selection>

but that resulted in "invalid earliest_time" errors: the earliest and latest tokens were being set to the literal string values "$start$" and "$end$", because the start and end tokens had not yet been defined; the <selection> element was being used (the selection event was firing) before the user did anything with the chart. This issue is described in more detailed in another question.

Then I tried replacing the <set> elements with <eval> (I wasn't sure from reading the documentation that this would be valid, or would work):

<selection>
  <eval token="earliest">if($start$="$start$",$earliest$,$start$)</eval>
  <eval token="latest">if($end$="$end$",$latest$,$end$)</eval>
</selection>

(That is, if the start and end tokens have not been defined—if the selection event has been fired without the user actually selecting an area of the chart—then set earliest and latest to their current values.)

That avoids the issue of setting earliest and latest to bogus values, but uncovers new problems:

• Clicking "Reset" in the chart that I "zoomed" does nothing. I guess I can understand that, because I've now set the global time to that zoomed selection.
• After zooming, selecting "All time" from the time picker, or removing earliest and latest from the URL parameters and reloading the page, produces unpredictable results. The time range can change to something that is neither all time, nor the previously zoomed time, but some other (unpredictable, to me) time range.

I've tried using a "local" time picker, and referring to the resulting $form.local.earliest$ and $form.local.latest$ tokens in the <earliest> and <latest>elements of all searches, and in the <selection> elements of time charts, but that doesn't have any effect on these problems.

Details of my use case

These particular dashboards show log data from "high-volume" transaction processing systems; systems that process many transactions per second. I have deliberately designed the dashboards, in Kibana and Splunk, to "work" regardless of the duration specified by the time picker: the X-axis of time-based charts automatically adjusts to match the duration. The same dashboards can be used to analyze logs across a wide variety of durations. In practice, users might initially be interested in a time range of several minutes, and then zoom in to analyze progressively narrower time ranges.

Summary

I'm bewildered that this behavior appears to be so hard to achieve in Splunk. I feel I must be missing something, but what? Are the typical use cases for Kibana (Elastic Stack) and Splunk so different that this out-of-the-box Kibana behavior is not also desirable in Splunk?

[rjthibod]

Thanks for the youtube video, that is helpful.

In my estimation, you cannot get exactly what you show in the video with existing versions of Splunk unless you add custom JavaScript. The custom JavaScript would spy on you using the pan & zoom time picker on the marquee timechart and then reload the dashboard with the new time value. It wouldn't be a huge amount of javascript, but it is extra work.

As @niketnilay alluded to, this is impossible because Splunk designed the pan & zoom and time pickers to be very flexible. There is no real pre-defined behavior, and you can have multiple charts, multiple time pickers, and lots of dynamic action without reloading the page.

So, now for an alternative.

With current Splunk, you should be able to get an approximation of what you want using just Splunk SimpleXML. There are two things that will be slightly different.

1. there will have to be a second time token but no input to go with it (i.e., the users can't edit it)
2. you would have to live with that activating the pan & zoom on the marquee time chart does not actually zoom in on the chart

If this alternative is acceptable to you or you want to play with it, I have copied an example dashboard at the bottom that does what I am suggesting.

Note that the bottom panels are populated based on the time token dd_time that is set by the <selection> value in the top-chart. There is not actual second time picker input, just an intermediate value dd_time.

If this is not acceptable, then JavaScript is your way forward. I could show you how to do that, but it is more of a time investment - I will wait to see what this alternative gives you.

<form>
  <label>Pan &amp; Zoom Test</label>
  <fieldset submitButton="false">
    <input type="time" searchWhenChanged="true">
      <label></label>
      <default>
        <earliest>-7d@h</earliest>
        <latest>now</latest>
      </default>
    </input>
  </fieldset>
  <row>
    <panel>
      <chart>
        <search>
          <query>
            | tstats prestats=t count where index=* by index _time span=1h
            | timechart span=1d count by index
          </query>
        </search>
        <option name="charting.chart">line</option>
        <selection>
          <set token="dd_time.earliest">$start$</set>
          <set token="dd_time.latest">$end$</set>
        </selection>
      </chart>
    </panel>
  </row>
  <row>
    <panel>
      <chart>
        <search>
          <query>
            | tstats prestats=t count where index=* by host _time span=1h
            | timechart span=2h count by host
          </query>
          <earliest>$dd_time.earliest$</earliest>
          <latest>$dd_time.latest$</latest>
        </search>
        <option name="charting.chart">line</option>
      </chart>
    </panel>
    <panel>
      <chart>
        <search>
          <query>
            | tstats prestats=t count where index=* by sourcetype _time span=1h
            | timechart span=2h count by sourcetype
          </query>
          <earliest>$dd_time.earliest$</earliest>
          <latest>$dd_time.latest$</latest>          
        </search>
        <option name="charting.chart">line</option>
      </chart>
    </panel>
  </row>  
</form>

[Graham_Hanningt]

First, thanks very much for taking the time to read my question.

Thanks also for the dashboard source. I've created a dashboard from that source and played with it.

Unfortunately—and here, I doubt I'm telling you anything you don't already know—it's pretty far from the "zoom one, zoom all; time picker in sync" behavior I'm after.

I could show you how to do that, but it is more of a time investment

I can't ask you to invest that time, but I would be grateful if you did, and I doubt I'd be alone. If you can show me how to reproduce in Splunk the behavior demonstrated in that YouTube video, I'll definitely accept your answer, and you'll get the points, for what that's worth.

I'm probably stating the bleeding obvious here, but just in case: one reason I like the behavior I demonstrate in that video is its simplicity. The user doesn't have to think about which time-based chart they can drag over to zoom on, and which other charts that will affect, or understand that zooming in won't affect the time range shown in the time picker. The entire dashboard uses a single time range.

[rjthibod]

See my new answer that includes JavaScript for a more generic solution.

[niketn]

@Graham_Hannington, can you add a screenshot of Kibana like features you are trying to build in Splunk? You can upload image to image sharing website and then share link here Image <img> button here on Splunk Answers.

One more question, when the chart selection is made you wanted it to populate the earliest and latest values of the time input based on the start and end time selected in the chart. When the form loads/reloads what provides the time range for the earliest and latest time for the chart? Is it going to be the same time input?

Time input can either be linked to query for the chart or chart selection change. If it gets linked to both, events will be conflicting. Hence if you need this you will need to have a button or intermediate event to pick the value of chart selection and apply back to the time input. This is the behavior of Search Timeline selection when you are looking at raw events in the Splunk Search screen.

[Graham_Hanningt]

There are also other Kibana features that I would like to see in Splunk, but I'll leave them for other questions, if I get time.

It's not all one-way: there are features of Splunk that I would like to see in Elastic/Kibana. I have the "luxury" of being able to forward exactly the same data, in the same format, using the same method—streaming JSON Lines over TCP—to either Elastic or Splunk.

[Graham_Hanningt]

Is it going to be the same time input?

Yes. Watch my demo video: one time picker for the entire dashboard. Zooming on a chart refreshes the entire dashboard, including the single time picker. At least, that's the behavior I want, without considering implementation issues in Splunk.

[Graham_Hanningt]

Hi @niketnilay,

Thanks for your interest. I have uploaded a video to YouTube: "Splunk feature request: zooming one chart zooms entire dashboard".