We have a set of logs from different hosts that specify a metric. I want to display a line graph over a user-selectable time period that plots the metric as a percentage difference from the 30 day average for each host.
That is to get the 30 day average for each host then plot (metric/average)*100)-100. This would mean that 80% would plot as -20, 100% as 0, 120% as 20, and so on. Additionally, we would like to highlight where the value falls outside -50 - 50. The next stage would be to alert if the value is outside the -50 to 50 more than the last, say 20 mins.
What we are looking for is to spot where the metric is larger than average for an extended period (flood condition) or lower (drought). The range could change.
I'm new to Splunk so I don't really know exactly where to start.
I can get the graph to work with averaging in the same period as plotting. It's having the average over a different time period. Hosts are selected by a pattern so would be dynamic (host matching pattern CVM_AGG). We may extend this to either a second query BUS_AGG or merge the two _AGG. Since we are using percentage against average they will scale the same.
... View more