Timechart grouping

[kmahamkali]

I am trying to analyze patterns of heap usage by Java Virtual Machine (JVM) level and 5 jvms grouped as a host. Now I want to timechart the heap by JVM and output it by host. I output all of the data by JVM, I am getting an unreadable graph.

Here is my search:
search |timechart span=10min avg(heap) by JVM.
With this search i am getting all the JVMS in graph(which is not readable), but I want a separate graph for each host with 4 jvms trending data.

[niketn]

@kmahamkali If you are on Splunk Enterprise 6.6. or higher you can use Trellis Layout to split the timechart by hosts and show the four JVMs in each of the host timecharts.

However, if you are using any previous versions of Splunk Enterprise you will have following two options:
Option 1) Aggregate field as JVM + host and then use the same new field to split your timechart.
Option 2) Create Post-processing to plot one Timechart at a time for one host. This will create view same as Trellis. Please let me know if this is possible option and you are not able to figure this out, as I have listed only option 1 below.

Following is run anywhere dashboard based on Splunk's _internal index where you can think component as your host field and log_level as your JVM field. Kindly adjust your search accordingly once you try out the run anywhere dashboard and it matches your expectations:

<dashboard>
  <label>Two Groups in Timechart</label>
  <row>
    <panel>
      <title>Without Trellis</title>
      <chart>
        <search>
          <query>index=_internal (sourcetype=splunkd log_level="ERROR" OR log_level="WARN") (component="AutoRegressionProcessor" OR component="EvalCommand" OR component="ExecProcessor" OR component="SearchParser" OR component="UiAuth")
| timechart count(eval(log_level="ERROR")) as ERROR count(eval(log_level="WARN")) as WARN by component limit=0 useother=f usenull=f</query>
          <earliest>-7d@h</earliest>
          <latest>now</latest>
          <sampleRatio>1</sampleRatio>
        </search>
        <option name="charting.axisLabelsX.majorLabelStyle.overflowMode">ellipsisNone</option>
        <option name="charting.axisLabelsX.majorLabelStyle.rotation">0</option>
        <option name="charting.axisTitleX.visibility">visible</option>
        <option name="charting.axisTitleY.visibility">visible</option>
        <option name="charting.axisTitleY2.visibility">visible</option>
        <option name="charting.axisX.abbreviation">none</option>
        <option name="charting.axisX.scale">linear</option>
        <option name="charting.axisY.abbreviation">none</option>
        <option name="charting.axisY.scale">linear</option>
        <option name="charting.axisY2.abbreviation">none</option>
        <option name="charting.axisY2.enabled">0</option>
        <option name="charting.axisY2.scale">inherit</option>
        <option name="charting.chart">column</option>
        <option name="charting.chart.bubbleMaximumSize">50</option>
        <option name="charting.chart.bubbleMinimumSize">10</option>
        <option name="charting.chart.bubbleSizeBy">area</option>
        <option name="charting.chart.nullValueMode">gaps</option>
        <option name="charting.chart.showDataLabels">none</option>
        <option name="charting.chart.sliceCollapsingThreshold">0.01</option>
        <option name="charting.chart.stackMode">default</option>
        <option name="charting.chart.style">shiny</option>
        <option name="charting.drilldown">none</option>
        <option name="charting.layout.splitSeries">0</option>
        <option name="charting.layout.splitSeries.allowIndependentYRanges">0</option>
        <option name="charting.legend.labelStyle.overflowMode">ellipsisMiddle</option>
        <option name="charting.legend.mode">standard</option>
        <option name="charting.legend.placement">right</option>
        <option name="charting.lineWidth">2</option>
        <option name="refresh.display">progressbar</option>
        <option name="trellis.enabled">0</option>
        <option name="trellis.scales.shared">0</option>
        <option name="trellis.size">medium</option>
        <option name="trellis.splitBy">component</option>
      </chart>
    </panel>
  </row>
  <row>
    <panel>
      <title>With Trellis</title>
      <chart>
        <search>
          <query>index=_internal sourcetype=splunkd log_level IN ("ERROR", "WARN") component IN ("AutoRegressionProcessor","EvalCommand","ExecProcessor","SearchParser","UiAuth")
| timechart count(eval(log_level="ERROR")) as ERROR count(eval(log_level="WARN")) as WARN by component</query>
          <earliest>-7d@h</earliest>
          <latest>now</latest>
          <sampleRatio>1</sampleRatio>
        </search>
        <option name="charting.axisLabelsX.majorLabelStyle.overflowMode">ellipsisNone</option>
        <option name="charting.axisLabelsX.majorLabelStyle.rotation">0</option>
        <option name="charting.axisTitleX.visibility">collapsed</option>
        <option name="charting.axisTitleY.visibility">collapsed</option>
        <option name="charting.axisTitleY2.visibility">collapsed</option>
        <option name="charting.axisX.abbreviation">none</option>
        <option name="charting.axisX.scale">linear</option>
        <option name="charting.axisY.abbreviation">none</option>
        <option name="charting.axisY.scale">linear</option>
        <option name="charting.axisY2.abbreviation">none</option>
        <option name="charting.axisY2.enabled">0</option>
        <option name="charting.axisY2.scale">inherit</option>
        <option name="charting.chart">column</option>
        <option name="charting.chart.bubbleMaximumSize">50</option>
        <option name="charting.chart.bubbleMinimumSize">10</option>
        <option name="charting.chart.bubbleSizeBy">area</option>
        <option name="charting.chart.nullValueMode">gaps</option>
        <option name="charting.chart.showDataLabels">none</option>
        <option name="charting.chart.sliceCollapsingThreshold">0.01</option>
        <option name="charting.chart.stackMode">default</option>
        <option name="charting.chart.style">shiny</option>
        <option name="charting.drilldown">none</option>
        <option name="charting.layout.splitSeries">0</option>
        <option name="charting.layout.splitSeries.allowIndependentYRanges">0</option>
        <option name="charting.legend.labelStyle.overflowMode">ellipsisMiddle</option>
        <option name="charting.legend.mode">standard</option>
        <option name="charting.legend.placement">bottom</option>
        <option name="charting.lineWidth">2</option>
        <option name="height">460</option>
        <option name="trellis.enabled">1</option>
        <option name="trellis.scales.shared">0</option>
        <option name="trellis.size">medium</option>
        <option name="trellis.splitBy">component</option>
      </chart>
    </panel>
  </row>
</dashboard>

[niketn]

@kmahamkali, were you able to try out the above solution?

[kmahamkali]

I end up in creating dashboard with 7 panels with the same query ( difference in host name).

[niketn]

@kmahamkali, so you are not on Splunk 6.6. or higher. Did you try post processing as suggested?

If you are on version prior to 6.6 you would need to have separate panels with different searches. However, you can use one of the options mentioned above. For example Post Processing will pull the data from index once and then filter results as needed by different panels.

If you tried one of the two options for version prior to 6.6., and it did not work, please share the query with us and we can assist. If you have used one of the two options, do not forget to accept the answer and up vote the comments that helped.

[kmahamkali]

yes But no luck sir.

[mayurr98]

hey @kmahamkali

Try this :

base_search JVM IN (jvm1,jvm2,jvm4,jvm5) | timechart span=10m avg(heap) as avg_heap by host

In search query, JVM is the field_name and jvm1,jvm2,jvm4,jvm5 are the values associated with that field name
Thereafter, go to visualization select column chart and use trellis layout split by host
you will get 10 charts in one query!

Let me know if this helps you!

[nikita_p]

Hi,
Can you check if below query works for you?
base search| chart avg(heap) AS heap, values(JVM) AS JVM by host | dedup host

[Kwip]

If my understating is correct below is the one you are looking for,

base search |  timechart span=10min avg(heap) by host JVM 
    OR
 base search
| eval hostjvm=host."-".JVM
| timechart span=10min avg(heap) by hostjvm

Let me know if above queries not serving your need and what is missing

[kmahamkali]

It didn't help.
I want to group the chart by host and with JVM trending data.
for example i have 10 hosts and each has 4 Jvms. I am looking for a graph 4 jvms grouping by host so that i can see 10 charts with trending data.

[Kwip]

So you are looking for a graph which contains 10 hosts for the average heap values of on 4 different JVMS. But you no need to list the JVMS in the chart. if that is the case mention your JVM names in the base search(Incase multiple JVMS are there, You just need only 4)

Something like below,

index=x sourcetype=y (jvm=1 OR jvm=2 OR jvm=3 OR jvm=4)  
| timechart span=10min avg(heap) by host

The above query will produce you 10 graphs based on the host and graph values will be the average heap of 4 jvms.

[somesoni2]

Could you provide more details on what your current output is and what's expected?