Chart Overlay and Different Graph Type

[aaronkorn]

Hello,

We are working on a dashboard panel right now that is going to show overall CPU utilization for user/system but want to overlay 2 processes that we want to monitor for utilization as well. The data is coming back fine but ideally we would like the User/System graph to be one graph ie. stacked area and the process utilization to be in the form of a line chart. Any ideas?

Here is the search we are using:
index=os source=cpu host=x38dev10 | multikv fields | timechart span=60s max(pctUser) as User, avg(pctSystem) as System by host | appendcols [search index="os" sourcetype="ps" host=x38dev10 | multikv fields pctCPU, COMMAND, USER | eval process=USER."/".COMMAND | search USER=tomcat OR redis | timechart span=60s avg(pctCPU) by process useother="f"]

[jonuwz]

Here's an example :

<?xml version="1.0" encoding="utf-8"?>
<dashboard>
  <label>SuperChart</label>
  <row>
    <chart>
      <searchName>SuperChartSearch</searchName>
      <title>SuperChart/title>

      <!-- set up alternative palatte for stacked chart, transparancy makes the line chart more visible -->
      <option name="charting.transparentBrushPalette">solidFill</option>
      <option name="charting.transparentBrushPalette.colorPalette">@colorPalette</option>
      <option name="charting.transparentBrushPalette.alpha">0.7</option>

      <!-- set chart to use columns 0,1,2,3 where @data is the original "cube", and 0 is the time-->
      <option name="charting.data1">view</option>
      <option name="charting.data1.table">@data</option>
      <option name="charting.data1.columns">[0,1,2,3]</option>
      <option name="charting.chart.data">@data1</option>

      <!-- set chart2 to use columns 0,4,5, where 0 is time, 4 + 5 are data -->
      <option name="charting.data2">view</option>
      <option name="charting.data2.table">@data</option>
      <option name="charting.data2.columns">[0,4,5]</option>
      <option name="charting.chart2.data">@data2</option>

      <!-- splunk doesnt suppory 2 Y axis, so we have to clone -->
      <option name="charting.axisY2">#axisY</option>                 <!-- clone the Yaxis into axisY2 -->
      <option name="charting.axisLabelsY2">#axisLabelsY</option>     <!-- clone the Yaxis Labels -->
      <option name="charting.axisLabelsY2.axis">@axisY2</option>     <!-- associate the new Y2 axis-labels with the Y2 axis -->

      <!--create the 1st chart -->
      <option name="charting.chart">column</option>
      <option name="charting.chart.columnBrushPalette">@transparentBrushPalette</option>
      <option name="charting.chart.stackMode">stacked</option>
      <option name="charting.chart.useAbsoluteSpacing">True</option>
      <option name="charting.chart.columnSpacing">20</option>
      <option name="charting.chart.columnAlignment">.5</option>
      <option name="charting.axisTitleY.text">CPU</option>
      <option name="charting.axisLabelsY.placement">left</option>

      <!--create the 2nd chart -->
      <option name="charting.chart2">line</option>
      <option name="charting.chart2.nullValueMode">zero</option>
      <option name="charting.chart2.axisY">@axisY2</option>          <!-- the Yaxis of chart 2 is a link to charting.axisY2 -->
      <option name="charting.axisTitleY2">axisTitle</option>
      <option name="charting.axisTitleY2.text">Memory</option>
      <option name="charting.axisLabelsY2.placement">right</option>

      <!-- throw the structure to layout -->
      <option name="charting.layout.charts">[@chart,@chart2]</option>
      <option name="charting.layout.axisLabels">[@axisLabelsX,@axisLabelsY,@axisLabelsY2]</option>
      <option name="charting.layout.axisTitles">[@axisTitleX,@axisTitleY,@axisTitleY2]</option>
    </chart>
  </row>
</dashboard>

This assumes saved search called "SuperChartSearch" returns data like this :

... | bin _time span=1h | stats avg(User) as User avg(Sys) as Sys avg(Wait) as Wait avg(mem_free) as mem_free avg(swap_free) as swap by _time

And will display like this :

Update

If you want a realtime chart, replace the <searchName>.....</searchName>

with (for example)

  <searchString>index=_internal | bin _time span=1m | stats min(eps) avg(eps) max(eps) sum(eps) perc95(eps) by _time</searchString>
    <earliestTime>rt-15m</earliestTime>
    <latestTime>rt</latestTime>

[lguinn2]

This is documented as a migration issue in moving from Splunk 5 to Splunk 6. In Splunk 6, simple XML uses JavaScript, not Flash. The technique used here requires Flash.

http://docs.splunk.com/Documentation/Splunk/6.0.1/AdvancedDev/Migration#Flash_charting_not_available...

[spammenot66]

@iguinn are you saying this won't work anymore in the latest 6.3? If so how would we replicate this in 6.3? I know this is still possible because I've seen a couple folks at the conf use a similiar chart

[watsm10]

This works brilliantly. Only problem I have is that when I try to generate a PDF, it doesn't render correctly. Anyone else having this issue?

[jonuwz]

that is surprising : )

feel free to mark the answer as accepted if its working for you.

[aaronkorn]

worked like a charm. Thanks!

[jonuwz]

try something like this - I have no idea if it works - I dont have the app for *nix :

index=os host=x38dev10 source=cpu OR sourcetype=ps
| multikv
| search USER=tomcat OR USER=redis OR source=cpu
| eval metric=(if(sourcetype=="ps",USER."/".COMMAND,"CPU"))
| timechart max(pctUser) as User, avg(pctSystem) as System avg(pctCPU) as proc by metric

[aaronkorn]

Thanks for the update. For some reason, I think how I have the subsearch setup with appendcols does not like realtime searches. Do I have the search setup properly for this or is there a better way to go about pulling in both the searches?

Thanks!

[jonuwz]

updated answer

[aaronkorn]

This is excellent! Thank you very much for the help!

Last question though.. How would I make this into a realtime search without having to use a saved search? How would I specify in the XML to make it a realtime search if I use this as the search?

index=os source=cpu host=x38dev10 | multikv fields | timechart span=60s max(pctUser) as User, avg(pctSystem) as System by host | appendcols [search index="os" sourcetype="ps" host=x38dev10 | multikv fields pctCPU, COMMAND, USER | eval process=USER."/".COMMAND | search USER=tomcat OR redis | timechart span=60s avg(pctCPU) by process useother="f"]