Using HiddenPostProcess to Build FlashCharts Based on User Selection

[snowmizer]

I have created a dashboard that has a flashchart showing protocol/port information using the HiddenSearch/HiddenPostProcess combination. I would like to be able to build additional flashcharts based on the protocol/port combination the user clicked on in my chart.

The basic data doesn't have the protocol/port combined so I used a eval statement to build this combination. When I run the code below my initial flashchart displays fine. However, when the user clicks on a port combination (i.e. TCP 80) my second flashchart doesn't display.

I'm attaching my code to this post. Can anyone tell me what I'm doing wrong?

<view template="dashboard.html">
  <label>HLC Botnet Dashboard</label>
  <module name="AccountBar" layoutPanel="navigationHeader" />
  <module name="AppBar" layoutPanel="navigationHeader" />
  <module name="Message" layoutPanel="messaging">
    <param name="filter">*</param>
    <param name="clearOnJobDispatch">False</param>
    <param name="maxSize">1</param>
  </module>
  <module name="TitleBar" layoutPanel="navigationHeader">
    <param name="showActionsMenu">False</param>
  </module>
  <module name="TimeRangePicker" layoutPanel="splSearchControls-inline">
    <param name="label">Select Time Range</param>
    <param name="default">Last 7 days</param>
    <param name="selected">Last 7 days</param>
    <param name="searchWhenChanged">true</param>
    <module name="HiddenSearch" autoRun="True" layoutPanel="panel_row1_col1" group="Active Ports">
      <param name="search">sourcetype="cisco_asa" (error_code="338002" OR error_code="338001" OR error_code="338003" OR error_code="338004" OR error_code="338202" OR error_code="338201" OR error_code="338203" OR error_code="338204") | eval proto_port = protocol." ".dest_port | fields proto_port, dest_ip</param>
      <module name="JobProgressIndicator"></module>
      <module name="HiddenPostProcess">
        <param name="autoRun">true</param>
        <param name="search">stats count by proto_port | sort -count</param>
        <module name="HiddenChartFormatter">
          <param name="chart">bar</param>
          <param name="legend.placement">none</param>
          <param name="primaryAxisTitle.text">Port</param>
          <param name="secondaryAxisTitle.text">Number of Connections</param>
          <module name="FlashChart">
            <module name="ConvertToDrilldownSearch">
              <module name="ConvertToIntention">
                <param name="intention">
                  <param name="name">stringreplace</param>
                  <param name="arg">
                    <param name="selected_port">
                      <param name="value">$click.value$</param>
                    </param>
                  </param>
                </param>
                <module name="HiddenPostProcess">
                  <param name="search">search proto_port= "$selected_port$" | fields dest_ip, count</param>
                  <module name="JobProgressIndicator"></module>
                  <module name="HiddenChartFormatter">
                    <param name="chart">bar</param>
                    <param name="legend.placement">none</param>
                    <param name="primaryAxisTitle.text">Malware Site</param>
                    <param name="secondaryAxisTitle.text">Number of Connections</param>
                    <module name="FlashChart">
                      <param name="width">100%</param>
                      <param name="height">300px</param>
                      <module name="ConvertToDrilldownSearch">
                        <module name="ViewRedirector">
                          <param name="viewTarget">flashtimeline</param>
                        </module>
                      </module>
                    </module>
                  </module>
                </module>
              </module>
            </module>
          </module>
        </module>
      </module>
    </module>
  </module>
</view>

[sideview · Accepted solution]

There's a couple things to address here.

Most fundamentally there is currently no mechanism to take user interactions with drilldowns and form-search modules and drive changes in the postProcess search. In view-speak, the HiddenPostProcess and PostProcessBar modules pay no attention to stringreplace intentions or any other intentions. Which means that your HiddenPostProcess is literally searching for proto_port="$selected_port$".

It's a very natural to expect this config to work, and I totally understand - because some params in some modules do dynamic replacement of $foo$, you expect the same in other sensible places. However in this case Im afraid you're ahead of us.

(It's probably in your best interests to email support to start an enhancement request - although this is tracked internally as a limitation, afaik no work is slated yet to address it)

So moving forward, if it's OK for the second FlashChart to actually run its own search, this is totally possible. The second search in a drilldown pair is often a considerably faster cheaper search so hopefully that's feasible here.

Anyway, the ConvertToDrilldownSearch module that you have there will take the base search, factor in the postProcess search it sees, factor in the clicked upon combination, and it will all by itself yield the following search string for downstream modules:

<your search> | search proto_port=<clicked on protocol-port-pair>

This isnt what you want though, since you want to turn the first click into a second report about the clicked-upon combination, not just a second search for the corresponding events.

What you need to do is what we call 'custom drilldown wiring'. SIDE NOTE: If you havent seen the ui_examples app yet, get it from splunkbase because it has a lot of drilldown examples and inline explanation. (You can install apps from splunkbase from your own UI, by using the 'launcher' app.)

So take out your ConvertToDrilldownSearch as well as the ConvertToIntention and HiddenPostProcess below it. you'll need to have

1) a single HiddenSearch underneath the first FlashChart, that gives you the report with the # of connections, with the search protoport="$proto_port$" syntax to express the clicked-upon combination.

2) a ConvertToIntention that turns $click.value$ into a 'stringreplace' intention that references the '$proto_port$' value. I know its annoyingly convoluted and verbose config and I apologize. (i'd welcome the incoming enhancement request to simplify the intention config because again its tracked but not actually scheduled).

The bottom line is that you've come pretty far. I think if you check out the drilldown examples and the lister examples in the ui_examples app, you've nearly gotten it figured out.

(One huge tip for debugging inline drilldown configurations, is to install the firebug extension for firefox if you havent already, and in the 'net' tab you can watch the POST's the UI is making to the jobs endpoint. Open those up and you'll see the searches that it is actually running. Likewise watch the requests that the FlashChart makes and you can see the postProcess searches it's sending.)

[sideview]

Ah of course. I should have thought of that. Yea there's a bug currently filed where the postProcess from upper layers is still being applied to various things in lower layers. Another workaround would be to put in a second HiddenPostProcess to null the postProcess search out to " ". Glad to hear you got it working though!

[snowmizer]

I think I got it figured out. Actually I think I stumbled on the answer. 🙂 Turns out that part of my problem was that my first chart was still using HiddenPostProcess. Once I changed my first search to HiddenSearch my second chart displayed. Then it was a matter of tweaking my second search to display as I wanted.

Thanks for all of the help. I will be submitting an enhancement request for the drilldown/postprocess.

[sideview]

Sounds like i need to see your modified XML too. Can you put it in something like pastebin and post a link?

[snowmizer]

Did this but now I don't see any chart. I've tried modifying the HiddenSearch so that it does just a basic search for the top 10 dest ips and creates a bar chart but it doesn't display anything.

[sideview]

that's right. If it feels like there's an extra step you're right. it'd be nice if there was a simpler mode where you dont need that ConvertToIntention and you can just put $click.value$ right into the search.

[snowmizer]

So you're replacing the HiddenPostProcess search proto_port... with a hiddensearch that has the | search proto_port="$selected_port$" and then inside the HiddenSearch placing a ConvertToIntention that converts $click.value$ into selected_port?