Splunk Search

Real-time search, URL Loader and inputlookup issue

anjafischer
Path Finder

Hello there,

I am facing a fairly difficult problem with Splunk...

Let me quickly explain my current scenario:
I have created a dashboard (advanced XML, in combination with Sideview utils) where I offer some PullDowns and a TimeRangePicker. based on the combination of these modules that a user clicked a Chart is displayed.

Now, the users of this dashboard are interested in monitoring performance data. This is why this chart supports real-time searches, so it gets automatically updated as soon as ne data arrives into Splunk. This works fine for a while, but, unfortunately, after a while the real-time search gets cancelled by Splunk.

I assume, that this "timeout" can be configured somewhere in Splunk, but I also assume, that no matter what value I choose, it will eventually get cancelled 😞

So, as a workaround I had the idea to auto-refresh the whole dashboard in bigger intervals, like, every half an hour or so. However, auto-refresh would undo all the user set configuration of TimeRangePicker and PullDown modules...

Thats why I implemented another "feature" within the dashboard. At the bottom of the dashboard I provide the user with a link to the chosesn configuration of the dashboard. So I use the Redirector module here and "store" all pullDown and TimerangePicker values in the link.

If the user configures his dashboard once, then clicks the link, it would take him to the same side but auto-refresh would now reload all the PullDowns and so on from the URL parameters. this works perfectly, UNLESS I define a real-time search via the TimeRangePicker.

Then i get an error message that real-time and inputlookup command are not compatible.
The problem is, I cannot get rid of the inputlookup, because I am populating my PullDown modules from values that are stores within lookup tables...

Besides, the TimeRangePicker is way downstream from those PullDowns, so there shouldn?t even be any problem related to real-time and inputlookup..

Is there a way to somehow pass the earliest/latest URL paramters directly to the TimeRangepicker and not through the PullDown setup?

I cannot put the URLLoader downstream from those Pulldowns either, because they are also set to the user chosen values via the URLLoader...

Unfortunately, at this point i am running out of ideas and I am hoping, that i am not the only person with this kind of problem/issue 😉

UPDATE:
this is my current (compressed/truncated) XML:






*
2
False

<module name="URLLoader" layoutPanel="viewHeader" autoRun="True" keepURLUpdated="True">

    <!-- search to retrieve all environment options from lookup table -->
    <module name="Search">
        <param name="search">| inputlookup mtc_environment_lookup.csv</param>

        <!-- Environment PullDown -->
        <module name="Pulldown">
            <param name="name">selectedEnvironment</param>

            <!-- lookup metrics in CSV lookup table -->
            <module name="Search">
                <param name="search">| inputlookup mtc_metrics_lookup.csv</param>

                <!-- Pulldown left metric -->
                <module name="Pulldown">
                    <param name="name">metricLeftAxis</param>

                    <!-- Pulldown right metric -->
                    <module name="Pulldown">
                        <param name="name">metricRightAxis</param>

                        <!-- bild search strings for later inclusion in main (real-time enabled) search -->
                        <module name="Search">
                            <param name="search">| inputlookup cics_trans_id_lookup.csv | dedup cics_trans_area_name | stats count as n 
                                <!-- calculate some static values that are needed for the HiddenchartFormatter later on -->
                                | table calcMetricRightAxis, appendRightTotal, removeMetricRightAxisParts, renameTotal, modifiedTotal, rightAxisTitle, rightAxisLabel, leftColumns, rightColumns</param>

                            <!-- make search string parts available to downstream modules -->
                            <module name="ResultsValueSetter">
                                <param name="fields">calcMetricRightAxis, appendRightTotal, removeMetricRightAxisParts, renameTotal, modifiedTotal, rightAxisTitle, rightAxisLabel, leftColumns, rightColumns</param>

                                <!-- TimeRangePicker for chart -->
                                <module name="TimeRangePicker">
                                    <param name="label">Choose your time range: </param>
                                    <param name="searchWhenChanged">True</param>
                                    <param name="selected">Today</param>

                                    <module name="Search">
                                        <param name="search">system=cics dc="$selectedEnvironment$" | lookup cics_trans_id_lookup.csv cics_trans_id as tran OUTPUT cics_trans_area_name
                                            | timechart minspan=5m nullstr="Other" limit=0 $metricLeftAxis$ $calcMetricRightAxis$ by cics_trans_area_name
                                            $appendRightTotal$
                                            $removeMetricRightAxisParts$
                                            $renameTotal$
                                            | fields _time, $modifiedTotal$, *</param>

                                        <!-- make some chart characteristics dynamicly configurable -->
                                        <module name="ValueSetter">
                                            <param name="arg.charting.axisTitleY.text">$metricLeftAxis.label$</param>
                                            <param name="arg.charting.axisTitleY2.text">$metricRightAxis.label$</param>
                                            <param name="arg.charting.data1.columns">[$leftColumns$]</param>
                                            <param name="arg.charting.data2.columns">[$rightColumns$]</param>
                                            <param name="arg.charting.layout.axisTitles">[@axisTitleX,@axisTitleY$rightAxisTitle$]</param>
                                            <param name="arg.charting.layout.axisLabels">[@axisLabelsX,@axisLabelsY$rightAxisLabel$]</param>

                                            <module name="HiddenChartFormatter">
                                                <!-- set some chart formatting values here -->
                                                <module name="FlashChart">
                                                    <param name="height">500px</param>
                                                </module>
                                            </module>
                                        </module>
                                    </module>

                                    <!-- generate Link to user selected configuration -->
                                    <module name="HTML">
                                        <param name="html">
                                            <![CDATA[
                                              Link to this page (your current configuration: 
                                              <a href="MTC_Projection_Overview?selectedEnvironment=$selectedEnvironment$&earliest=$search.timeRange.earliest$&latest=$search.timeRange.latest$&metricLeftAxis=$metricLeftAxis$&metricRightAxis=$metricRightAxis$">
                                                  MTC_Projection_Overview?selectedEnvironment=$selectedEnvironment$&earliest=$search.timeRange.earliest$&latest=$search.timeRange.latest$&metricLeftAxis=$metricLeftAxis$&metricRightAxis=$metricRightAxis$
                                              </a>              
                                            ]]>
                                        </param>
                                    </module>
                                </module>
                            </module>
                        </module>
                    </module>
                </module>
            </module>
        </module>
    </module>
</module>

0 Karma
1 Solution

anjafischer
Path Finder

I think I found a solution.

Adding an "append=true" to all my inputlookup command did the trick.

This way the inputlookup command suddenly becomes compatible with real-time searches and I can click on the link that I built at the bottom of my dashboard and the URLLoader sets all the interactive elements (PullDowns and TimeRangePicker) to the correct values, that were listed in the URL. I can now auto-refresh without problems, however, I am still interested in the whole remote cancellation of real-time searches...

Is there a way to keep Splunk from cancelling my jobs?

View solution in original post

anjafischer
Path Finder

I think I found a solution.

Adding an "append=true" to all my inputlookup command did the trick.

This way the inputlookup command suddenly becomes compatible with real-time searches and I can click on the link that I built at the bottom of my dashboard and the URLLoader sets all the interactive elements (PullDowns and TimeRangePicker) to the correct values, that were listed in the URL. I can now auto-refresh without problems, however, I am still interested in the whole remote cancellation of real-time searches...

Is there a way to keep Splunk from cancelling my jobs?

View solution in original post

martin_mueller
SplunkTrust
SplunkTrust

Does this boil down to a page refresh that maintains settings?

I've solved that by letting the URLLoader update the URL whenever changes are made, and by implementing a small JavaScript refresh timer. That will then refresh the page based on the most up-to-date URL.

Edit: Maybe not, missed the realtime vs inputlookup part... Do post your condensed XML tree though so we get a better picture.

Update: You can work your way around this using the ValueSetter module. Here's an example:

<module name="URLLoader" layoutPanel="panel_row1_col1" autoRun="True">
  <param name="keepURLUpdated">True</param>
  <module name="ValueSetter">
    <param name="if.$earliest$=*">$earliest$</param>
    <param name="name">url_earliest</param>
    <param name="default">rt-30m</param>
    <param name="name">url_earliest</param>
    <module name="ValueSetter">
      <param name="if.$latest$=*">$latest$</param>
      <param name="default">rt</param>
      <param name="name">url_latest</param>
      <param name="name">url_latest</param>
      <module name="Search">
        <param name="search">
| inputlookup price_lookup
</param>
        <param name="latest">now</param>
        <param name="earliest">-1m</param>
        <module name="Pulldown">
          <param name="template">$name$="$value$"</param>
          <param name="labelField">product_name</param>
          <param name="name">productId</param>
          <param name="valueField">$name$</param>
          <param name="staticOptions"/>
          <module name="ValueSetter">
            <param name="value">$url_earliest$</param>
            <param name="name">search.timeRange.earliest</param>
            <module name="ValueSetter">
              <param name="value">$url_latest$</param>
              <param name="name">search.timeRange.latest</param>
              <module name="TimeRangePicker">
                <module name="Search">
                  <param name="search">
index=_internal | eval $productId$ | table _time sourcetype productId
</param>
                  <module name="Table" />
                </module>
              </module>
            </module>
          </module>
        </module>
      </module>
    </module>
  </module>
</module>

Basically you stash away the parameters from the URL, do the inputlookup with a fixed timerange, and then restore your URL parameters.

martin_mueller
SplunkTrust
SplunkTrust

I've added a proper answer, stashing values away around the inputlookup.

0 Karma

anjafischer
Path Finder

Yes, unfortunately it is not only the auto-refresh and maintaing settings.
Also, I am not using the Redirector to link to the user-set configuration, I am just using a plain HTML module, that shouldn't be the reason of the issue.

Anyways, i have updated my initial post and included a compressed version of my current XML. I hope it helps 🙂

0 Karma
.conf21 CFS Extended through 5/20!

Don't miss your chance
to share your Splunk
wisdom in-person or
virtually at .conf21!

Call for Speakers has
been extended through
Thursday, 5/20!