Splunk Performance Graphs Not showing up.

[jpewthers]

I've read the other posts on here, but they are still not helping me. None of my performance charts are showing up.

Let's start with CPU Load. Here is the service-perfdata raw line:

[SERVICEPERFDATA] 1350596518 captain-host CPU Load 0.408 0.438 OK CPU Load ok. 5s=24%;80;90;

I'm supposed to change 'avg(CpuLoad)' to something... but what? avg(CPU Load) does nothing. I've tried everything I can think of but no dice. What am I missing?

[lukeh · Accepted solution]

The first log entry in your example now has the correct format, whereas the second log entry is incorrect. If you were just providing the second log entry as a comparison then continue on, otherwise you have to delete the old template in your nagios configuration.

The nagios performance data is listed in the 'result' key/value pair, but as you are not using the CPU Nagios Plugin supported by Splunk for Nagios (ie. Check CPU Performance: http://exchange.nagios.org/directory/Plugins/System-Metrics/CPU-Usage-and-Load/Check-CPU-Performance... ) the graph is not displaying.

You can change the relevant search in the dashboard (eg. NagiosLinuxPerformanceGraphs.xml) to make the graph display correctly:

From:

 <param name="search">timechart span=5m avg(CpuSystem) as System,avg(CpuUser) as User,avg(CpuNice) as Nice,avg(CpuIowait) as Wait</param>

To:

 <param name="search">search name="CPU Load" | rex field=_raw "5s=(?.*)%" | eval total=100 | timechart span=5m avg(CpuLoad) as CpuLoad, max(total) as Total</param>

Then, remove the following lines from the same "CPU Usage" group:

    <param name="charting.chart.stackMode">stacked</param>

    <param name="charting.seriesColors">[0xFF0000,0x0000FF,0x00FF00,0xFF00FF]</param>

    <param name="charting.legend.masterLegend"></param>

Essentially, the new search extracts a new field called 'CpuLoad' and adds a new field called 'total' which is always 100. ie.

CpuLoad=43 -> created by the rex command which matches 43 from 5s=43%

After you update the xml in the dashboard, click on the Splunk logo at the top left in Splunk for Nagios to reload the dashboards, then click on the relevant dashboard and the graph should display correctly 🙂

All the best,

Luke 🙂

[lukeh]

You're welcome, please feel free to accept my answer 🙂

[jpewthers]

You are a gentleman and a scholar. I think I have a better understanding of not only this app but Splunk as a whole.

[jpewthers]

Many thanks for your response! It appears I should have put the line below. Here is all the data in the service-perfdata file.

---

1350598018 src_host="captain-host" perfdata="SERVICEPERFDATA" name="CPU Load" severity="OK" attempt="1" statetype="HARD" executiontime="0.309" latency="0.360" reason="OK CPU Load ok." result="5s=43%;80;90"

[SERVICEPERFDATA] 1350598018 captain-host CPU Load 0.309 0.360 OK CPU Load ok. 5s=43%;80;90

---

I tried putting 'avg(CPU Load)' in the view, since that's what I assumed the 'name' variable was, but still no dice. Any suggestions?

[lukeh]

Hi,

the reason that the performance graphs are not showing up is because you are not using the correct template for your nagios performance data.

Please read the section entitled "Nagios Configuration (REQUIRED)" at the Splunk for Nagios page:

http://splunk-base.splunk.com/apps/22374/splunk-for-nagios

Implement steps 1 through to 5 and the graphs will display 🙂

All the best,

Luke 🙂

[jpewthers]

I appreciate your response. I missed the most important line in that service-perfdata. I posted it below. ANy thoughts?