Building a view with 2 dynamic drop down menus

[lukeh]

Hi, I am building a view that has 2 drop down menus that auto-populate themselves. The first drop down menu displays hostnames, and the second drop down menu displays file systems relevant to the hostname selected in the previous drop down menu. Both menus are populated by Search listers which build their lists from the ingested Nagios logs.

The selected file system in the second drop down menu then populates an area chart below with a file system usage graph (used space vs total space) over time.

The graph is generated by a master search that includes a regex to extract the used space (fs_used) and the total space (fs_total), but there are two ConvertToIntention's that both do an 'addterm' for the hostname and the file system... the problem is that the terms are being added in the wrong position in the master search; so the graph fails and says "No results found."

When I click on More info... I receive the following error:

Search Job Inspector This search has completed, but did not match any events. The terms specified in the highlighted portion of the search:

search index=nagiosdev sourcetype="nagiosserviceperf" plugin="Disk" | rex max_match=1 field=_raw "\Q$filesystem$=\E(?[^M]*)MB;[0-9]+;[0-9]+;[0-9]+;(?[0-9]+)" | search hostname="dev1web07" filesystem="/" | timechart span=5m avg(fs_total) as "Total $filesystem$",avg(fs_used) as "Used $filesystem$"

As can been seen in the search above, one of the problems is that the following is being appended after the regular expression that does the multiple field extractions: search hostname="dev1web07" filesystem="/"

What should happen is that hostname="dev1web07" should be added after plugin="Disk" but before the pipe and the subsequent rex command, and filesystem="/" should not be appended, instead the variable $filesystem$ should be substituted after the timechart.

i.e. the search should look like this: search index=nagiosdev sourcetype="nagiosserviceperf" plugin="Disk" hostname="dev1web07" | rex max_match=1 field=_raw "\Q/=\E(?[^M]*)MB;[0-9]+;[0-9]+;[0-9]+;(?[0-9]+)" | timechart span=5m avg(fs_total) as "Total /",avg(fs_used) as "Used /"

I am running Splunk 4.1.7.

Any help/guidance would be greatly appreciated 🙂

Luke 🙂

My code:

<?xml version="1.0"?>
<view onunloadCancelJobs="False" autoCancelInterval="100">
  <!--  autoCancelInterval is set here to 100  -->
  <label>Nagios Filesystem Usage Graphs</label>
  <module name="AccountBar" layoutPanel="appHeader"/>
  <module name="AppBar" layoutPanel="navigationHeader"/>
  <module name="Message" layoutPanel="messaging">
    <param name="filter">*</param>
    <param name="clearOnJobDispatch">False</param>
    <param name="maxSize">1</param>
  </module>
  <module name="TitleBar" layoutPanel="viewHeader">
    <param name="actionsMenuFilter">dashboard</param>
  </module>

  <module name="HiddenSearch" layoutPanel="mainSearchControls" autoRun="True">
    <param name="search">index=nagiosdev sourcetype="nagiosserviceperf" plugin="Disk" | timechart span=5m avg(fs_total) as "Total $filesystem$",avg(fs_used) as "Used $filesystem$" </param>  
    <module name="SearchSelectLister">
      <param name="search">index="nagiosdev" nagiosevent="CURRENT HOST STATE" | rex ".+CURRENT HOST STATE\: (?P&lt;device&gt;[^;]*)(?=;)"| stats count by device</param>
      <param name="earliest">-24h</param>
      <param name="selected">dev1web07</param>
      <param name="label">Select a Hostname:</param>
      <param name="settingToCreate">hostname</param>
      <param name="searchFieldsToDisplay">
        <list>
          <param name="value">device</param>
          <param name="label">device</param>
        </list>
      </param>
      <module name="ConvertToIntention">
        <param name="settingToConvert">hostname</param>
        <param name="intention">
          <param name="name">addterm</param>
          <param name="arg">
            <param name="hostname">$target$</param>
          </param>
        </param>

        <module name="SearchSelectLister">
          <param name="searchWhenChanged">True</param>

          <param name="settingToCreate">series_setting</param>
          <param name="label">Select a Filesystem:</param>
          <param name="applyOuterIntentionsToInternalSearch">True</param>
          <param name="search">index=nagiosdev sourcetype="nagiosserviceperf" plugin="Disk" | rex max_match=1000 field=_raw "(?&lt;filesystem&gt;\/[^= ]*)=" | stats count by filesystem</param>
          <param name="earliest">-1h</param>
          <param name="searchFieldsToDisplay">
            <list>
              <param name="label">filesystem</param>
              <param name="value">filesystem</param>
            </list>
          </param>
          <module name="ConvertToIntention">
            <param name="settingToConvert">series_setting</param>
            <param name="intention">
              <param name="name">addterm</param>
              <param name="arg">
                <param name="filesystem">$target$</param>
              </param>
            </param>



            <module name="TimeRangePicker">
              <param name="selected">Last 4 hours</param>
              <param name="searchWhenChanged">True</param>
              <module name="SubmitButton">
                <param name="allowSoftSubmit">True</param>
                <param name="label">Search</param>


                <module name="Message" layoutPanel="graphArea">
                  <param name="filter">splunk.search.job</param>
                  <param name="clearOnJobDispatch">True</param>
                  <param name="maxSize">2</param>
                  <module name="GenericHeader" layoutPanel="resultsAreaLeft">
                    <param name="label">Filesystem Usage over time</param>
                  </module>
                  <module name="HiddenSearch" layoutPanel="resultsAreaLeft">
                    <param name="search">index=nagiosdev sourcetype="nagiosserviceperf" plugin="Disk" | rex max_match=1 field=_raw "\Q$filesystem$=\E(?&lt;fs_used&gt;[^M]*)MB;[0-9]+;[0-9]+;[0-9]+;(?&lt;fs_total&gt;[0-9]+)" | timechart span=5m avg(fs_total) as "Total $filesystem$",avg(fs_used) as "Used $filesystem$"</param>
                    <module name="HiddenChartFormatter">
                      <param name="chart">area</param>
                      <param name="primaryAxisTitle.text">Time</param>
                      <param name="secondaryAxisTitle.text">MB</param>
                      <module name="JobProgressIndicator"/>
                      <module name="FlashChart">
                        <param name="width">100%</param>
                        <param name="height">200px</param>
                      </module>
                    </module>
                  </module>
                </module>
              </module>
            </module>
          </module>
        </module>
      </module>
    </module>
  </module>

</view>

Sample log data:

1297934993  dev1web07   SERVICEPERFDATA Disk    WARNING 3   HARD    0.889   0.531   DISK WARNING - free space: /opt 190 MB (8% inode=76%):  /=1987MB;2173;2294;0;2415 /boot=23MB;170;179;0;189 /home=276MB;445;470;0;495 /opt=2038MB;2090;2206;0;2323 /tmp=36MB;891;941;0;991 /var=1965MB;2231;2355;0;2479 /var/crash=18MB;445;470;0;495 /srv/cpan=9196MB;11781;12436;0;13091
1297935293  dev1web07   SERVICEPERFDATA Disk    WARNING 3   HARD    0.155   0.663   DISK WARNING - free space: /opt 190 MB (8% inode=76%):  /=1987MB;2173;2294;0;2415 /boot=23MB;170;179;0;189 /home=276MB;445;470;0;495 /opt=2038MB;2090;2206;0;2323 /tmp=36MB;891;941;0;991 /var=1965MB;2231;2355;0;2479 /var/crash=18MB;445;470;0;495 /srv/cpan=9196MB;11781;12436;0;13091
1297935593  dev1web07   SERVICEPERFDATA Disk    WARNING 3   HARD    0.363   0.833   DISK WARNING - free space: /opt 190 MB (8% inode=76%):  /=1987MB;2173;2294;0;2415 /boot=23MB;170;179;0;189 /home=276MB;445;470;0;495 /opt=2038MB;2090;2206;0;2323 /tmp=36MB;891;941;0;991 /var=1965MB;2231;2355;0;2479 /var/crash=18MB;445;470;0;495 /srv/cpan=9196MB;11781;12436;0;13091
1297935893  dev1web07   SERVICEPERFDATA Disk    WARNING 3   HARD    0.105   0.678   DISK WARNING - free space: /opt 185 MB (8% inode=76%):  /=1987MB;2173;2294;0;2415 /boot=23MB;170;179;0;189 /home=276MB;445;470;0;495 /opt=2043MB;2090;2206;0;2323 /tmp=36MB;891;941;0;991 /var=1965MB;2231;2355;0;2479 /var/crash=18MB;445;470;0;495 /srv/cpan=9196MB;11781;12436;0;13091
1297936192  dev1web07   SERVICEPERFDATA Disk    WARNING 3   HARD    0.189   0.850   DISK WARNING - free space: /opt 191 MB (8% inode=76%):  /=1987MB;2173;2294;0;2415 /boot=23MB;170;179;0;189 /home=276MB;445;470;0;495 /opt=2038MB;2090;2206;0;2323 /tmp=36MB;891;941;0;991 /var=1965MB;2231;2355;0;2479 /var/crash=18MB;445;470;0;495 /srv/cpan=9196MB;11781;12436;0;13091
1297936492  dev1web07   SERVICEPERFDATA Disk    WARNING 3   HARD    0.278   0.615   DISK WARNING - free space: /opt 190 MB (8% inode=76%):  /=1987MB;2173;2294;0;2415 /boot=23MB;170;179;0;189 /home=276MB;445;470;0;495 /opt=2038MB;2090;2206;0;2323 /tmp=36MB;891;941;0;991 /var=1965MB;2231;2355;0;2479 /var/crash=18MB;445;470;0;495 /srv/cpan=9196MB;11781;12436;0;13091
1297936793  dev1web07   SERVICEPERFDATA Disk    WARNING 3   HARD    0.291   0.459   DISK WARNING - free space: /opt 191 MB (8% inode=76%):  /=1987MB;2173;2294;0;2415 /boot=23MB;170;179;0;189 /home=276MB;445;470;0;495 /opt=2037MB;2090;2206;0;2323 /tmp=36MB;891;941;0;991 /var=1965MB;2231;2355;0;2479 /var/crash=18MB;445;470;0;495 /srv/cpan=9196MB;11781;12436;0;13091
1297937095  dev1web07   SERVICEPERFDATA Disk    WARNING 3   HARD    0.293   2.193   DISK WARNING - free space: /opt 190 MB (8% inode=76%):  /=1987MB;2173;2294;0;2415 /boot=23MB;170;179;0;189 /home=276MB;445;470;0;495 /opt=2038MB;2090;2206;0;2323 /tmp=36MB;891;941;0;991 /var=1965MB;2231;2355;0;2479 /var/crash=18MB;445;470;0;495 /srv/cpan=9196MB;11781;12436;0;13091
1297937390  dev1web07   SERVICEPERFDATA Disk    WARNING 3   HARD    0.343   0.201   DISK WARNING - free space: /opt 190 MB (8% inode=76%):  /=1987MB;2173;2294;0;2415 /boot=23MB;170;179;0;189 /home=276MB;445;470;0;495 /opt=2038MB;2090;2206;0;2323 /tmp=36MB;891;941;0;991 /var=1965MB;2231;2355;0;2479 /var/crash=18MB;445;470;0;495 /srv/cpan=9196MB;11781;12436;0;13091
1297937690  dev1web07   SERVICEPERFDATA Disk    WARNING 3   HARD    0.153   0.182   DISK WARNING - free space: /opt 191 MB (8% inode=76%):  /=1987MB;2173;2294;0;2415 /boot=23MB;170;179;0;189 /home=276MB;445;470;0;495 /opt=2037MB;2090;2206;0;2323 /tmp=36MB;891;941;0;991 /var=1965MB;2231;2355;0;2479 /var/crash=18MB;445;470;0;495 /srv/cpan=9196MB;11781;12436;0;13091
1297937992  dev1web07   SERVICEPERFDATA Disk    WARNING 3   HARD    0.196   0.047   DISK WARNING - free space: /opt 190 MB (8% inode=76%):  /=1987MB;2173;2294;0;2415 /boot=23MB;170;179;0;189 /home=276MB;445;470;0;495 /opt=2038MB;2090;2206;0;2323 /tmp=36MB;891;941;0;991 /var=1965MB;2231;2355;0;2479 /var/crash=18MB;445;470;0;495 /srv/cpan=9196MB;11781;12436;0;13091
1297938292  dev1web07   SERVICEPERFDATA Disk    WARNING 3   HARD    0.355   0.067   DISK WARNING - free space: /opt 190 MB (8% inode=76%):  /=1987MB;2173;2294;0;2415 /boot=23MB;170;179;0;189 /home=276MB;445;470;0;495 /opt=2038MB;2090;2206;0;2323 /tmp=36MB;891;941;0;991 /var=1965MB;2231;2355;0;2479 /var/crash=18MB;445;470;0;495 /srv/cpan=9196MB;11781;12436;0;13091

[hazekamp · Accepted solution]

Luke,

The following code should work. Basically I converted your addterm intentions to stringreplace. The searches may not be exactly correct depending on how you have fields extracted. I would recommend extracting fields via props.conf so that you have key=value pairs extracted automatically at search time. The bottom line is that you have $deviceToken$ and $filesystemToken$ at your disposal:

<?xml version="1.0"?>
<view onunloadCancelJobs="False" autoCancelInterval="100">
  <!--  autoCancelInterval is set here to 100  -->
  <label>Nagios Filesystem Usage Graphs</label>
  <module name="AccountBar" layoutPanel="appHeader"/>
  <module name="AppBar" layoutPanel="navigationHeader"/>
  <module name="Message" layoutPanel="messaging">
    <param name="filter">*</param>
    <param name="clearOnJobDispatch">False</param>
    <param name="maxSize">1</param>
  </module>
  <module name="TitleBar" layoutPanel="viewHeader">
    <param name="actionsMenuFilter">dashboard</param>
  </module>

  <module name="SearchSelectLister" layoutPanel="mainSearchControls" autoRun="True">
    <param name="search">index="nagiosdev" nagiosevent="CURRENT HOST STATE" | rex ".+CURRENT HOST STATE\: (?P&lt;device&gt;[^;]*)(?=;)"| stats count by device</param>
    <param name="earliest">-24h</param>
    <param name="selected">dev1web07</param>
    <param name="label">Select a Hostname:</param>
    <param name="settingToCreate">device_setting</param>
    <param name="searchFieldsToDisplay">
      <list>
        <param name="value">device</param>
        <param name="label">device</param>
      </list>
    </param>
    <module name="ConvertToIntention">
      <param name="settingToConvert">device_setting</param>
      <param name="intention">
        <param name="name">stringreplace</param>
        <param name="arg">
          <param name="deviceToken">
            <param name="value">$target$</param>
          </param>
        </param>
      </param>
      <module name="SearchSelectLister">
        <param name="searchWhenChanged">True</param>
         <param name="settingToCreate">filesystem_setting</param>
        <param name="label">Select a Filesystem:</param>
        <param name="applyOuterIntentionsToInternalSearch">True</param>
        <param name="search">index=nagiosdev sourcetype="nagiosserviceperf" plugin="Disk" device=$deviceToken$ | rex max_match=1000 field=_raw "(?&lt;filesystem&gt;\/[^= ]*)=" | stats count by filesystem</param>
        <param name="earliest">-1h</param>
        <param name="searchFieldsToDisplay">
          <list>
            <param name="label">filesystem</param>
            <param name="value">filesystem</param>
          </list>
        </param>
        <module name="ConvertToIntention">
          <param name="settingToConvert">filesystem_setting</param>
          <param name="intention">
            <param name="name">stringreplace</param>
            <param name="arg">
              <param name="filesystemToken">
                <param name="value">$target$</param>
              </param>
            </param>
          </param>
          <module name="TimeRangePicker">
            <param name="selected">Last 4 hours</param>
            <param name="searchWhenChanged">True</param>
            <module name="SubmitButton">
              <param name="allowSoftSubmit">True</param>
              <param name="label">Search</param>
              <module name="Message" layoutPanel="graphArea">
                <param name="filter">splunk.search.job</param>
                <param name="clearOnJobDispatch">True</param>
                <param name="maxSize">2</param>
                <module name="GenericHeader" layoutPanel="resultsAreaLeft">
                  <param name="label">Filesystem Usage over time</param>
                </module>
                <module name="HiddenSearch" layoutPanel="resultsAreaLeft">
                  <param name="search">index=nagiosdev sourcetype="nagiosserviceperf" plugin="Disk" device=$deviceToken$ filesystem=$filesystemToken$ | rex max_match=1 field=_raw "\Q$filesystemToken$=\E(?&lt;fs_used&gt;[^M]*)MB;[0-9]+;[0-9]+;[0-9]+;(?&lt;fs_total&gt;[0-9]+)" | timechart span=5m avg(fs_total) as "Total $filesystemToken$",avg(fs_used) as "Used $filesystemToken$"</param>
                  <module name="HiddenChartFormatter">
                    <param name="chart">area</param>
                    <param name="primaryAxisTitle.text">Time</param>
                    <param name="secondaryAxisTitle.text">MB</param>
                    <module name="JobProgressIndicator"/>
                    <module name="FlashChart">
                      <param name="width">100%</param>
                      <param name="height">200px</param>
                    </module>
                  </module>
                </module>
              </module>
            </module>
          </module>
        </module>
      </module>
    </module>
  </module>

</view>

[lukeh]

Thanks hazedav !!!

[araitz]

There appear to be encoded terms in some of your rex entries - not sure how to interpret those.

I'm not sure I understand why you have two HiddenSearches. Just use one, and place it at the top of your hierarchy. You can use HiddenPostProcess to append terms to the search, but you shouldn't need to do that here.

If you want the $filesystem$ token to be replaced, use the 'stringreplace' intention rather than 'addterm'.

Speaking of 'addterm', the way that intention works is that if there is a pipe in the base search, it will append using the '| search' convention. If you want it in the base search, I suggest using 'stringreplace' as well.