Splunk App for Unix and Linux: Help creating a dashboard that shows servers using 20% more CPU than previous week

[daniel333]

All,

I have 400+ servers with Splunk for Nix installed and collecting metrics to index=os. What I'd like to do is create a dashboard which determines which servers are showing 20% more CPU than they were last week.

That the final result is just a table of servers which have showed 20% increase or more CPU compare the previous week. I really have no idea where to start. Any ideas?

[kyaparla]

This search can dynamically look for data 7 days back same hour and minute,

|makeresults count=2 | streamstats count |eval count=count-1 | addinfo |eval timediff=(24*3600*7*count) | eval latest=info_max_time-timediff| eval earliest=info_min_time-timediff | map search="search earliest=$earliest$ latest=$latest$ index=os sourcetype=cpu | bin span=5m _time | stats avg(cpu) as cpu by _time,host | eval _time=_time+$timediff$ | eval pw_no=$count$" | eval w{pw_no}=cpu | stats values(w*) as w* by _time,host | eval pct_increase=((w0-w1)/w1)*100 | where pct_increase>20

[anthonymelita]

This maybe accomplishes what you want. Here I'm doing today versus same day last week. Using appendcols to add a stat for the previous time period then doing calculations against the two stats.

index=os sourcetype=cpu earliest=@d latest=now
| multikv fields pctIdle  
| eval Percent_CPU_Load = 100 - pctIdle  
| stats avg(Percent_CPU_Load) as avgLoad by host
| appendcols [search index=os sourcetype=cpu earliest=-8d@d latest=-7d
| multikv fields pctIdle  
| eval Percent_CPU_Load = 100 - pctIdle  
| stats avg(Percent_CPU_Load) as newAVG by host]
| eval Change_Percentage=(newAVG-avgLoad)*100/avgLoad
| where Change_Percentage >= 20
| table host avgLoad Change_Percentage newAVG

As you can see, I also made the CPU usage calculation based on 100 minus pctIdle, which is something I read to do. Someone may feel free to advise on an alternative to that calculation.