Splunk Search
Highlighted

Sum latest entries from multiple sources & timechart as a single line

New Member

Hello,

I have multiple remote performance monitors sources, namely WMI:FOO1, WMI:FOO2 etc. up to and including WMI:FOO9. These each report on a value called BAR every 1 minute.

I am attempting to generate a timechart with a single line representing the total value of BAR across all my WMI:FOOx sources.

Additionally, any solution needs to ignore previous values of a WMI:FOOx source if that source has not reported a value for BAR within a set time period, say 2 minutes (in case the server goes down, I don't want the last value being included in a sum.)

Thanks in advance,
Justin

0 Karma
Highlighted

Re: Sum latest entries from multiple sources & timechart as a single line

Splunk Employee
Splunk Employee

First, to get results from multiple sources, use this in your search:

source=WMI:FOO*

So, something like:

source=WMI:FOO* BAR=* | timechart span=1m sum(BAR) useother=false

Could you clarify (in comments or by editing the question) what specifically you are trying to ignore? You can specify stuff like NOT BAR=0 or things like that...

0 Karma
Highlighted

Re: Sum latest entries from multiple sources & timechart as a single line

New Member

I had tried something similar, a sum with a span equal to the poll interval.

Occasionally a server would respond immediately resulting in 2 results for the same source within the 60s span. Conversely sometimes a server would take a while to respond meaning that there would be no value for the source counted within the span.

This resulted in blips of +/- a single BAR value on the graph. Tweaking the span would either decrease duplicate source counts and increase missed counts, or vice versa.

Can a sum be done on the last values for each WMI:FOOx source, rather than rely on a time based span?

0 Karma
Highlighted

Re: Sum latest entries from multiple sources & timechart as a single line

New Member

Regarding the ignoring of values - if the solution to above does involve summing of last BAR values of each WMI:FOOx source, I would want to ensure that in the event of a server being unreachable, that its last BAR value is not summed into the graphed value.

0 Karma
Highlighted

Re: Sum latest entries from multiple sources & timechart as a single line

Engager

Well this isn't perfect to your case but could be a useful substitution, if you are will to accept the modifications in behavior. I have a very similar situation with multiple QoS across multiple Interfaces across multiple Hosts. Because some interfaces utilization is drastically different than another I want to watch per host, not per interface per host. Thus far any indication of no response appears to be reporting as a gap in my chart.

My goal was to measure the interval change (new - old = delta), my only problem now is because of the forced values my first interval has an over inflated value that I need to drop. A sub search hasn't worked thus far, but this is a work in progress.

The forced values were required as a result of any of the QoS values could be null and that would result in a null sum regardless if the other 8 QoS contained values.

index=bulkstats (host=host-01 OR host=host-02) QoS8dwlinkpktdrop=* QoS8uplinkpktdrop=*
| reverse
| streamstats global=f current=f
last(QoS1dwlinkpktdrop) as pQoS1dwlinkpktdrop, last(QoS1uplinkpktdrop) as pQoS1uplinkpktdrop,
last(QoS2dwlinkpktdrop) as pQoS2dwlinkpktdrop, last(QoS2uplinkpktdrop) as pQoS2uplinkpktdrop,
last(QoS3dwlinkpktdrop) as pQoS3dwlinkpktdrop, last(QoS3uplinkpktdrop) as pQoS3uplinkpktdrop,
last(QoS4dwlinkpktdrop) as pQoS4dwlinkpktdrop, last(QoS4uplinkpktdrop) as pQoS4uplinkpktdrop,
last(QoS5dwlinkpktdrop) as pQoS5dwlinkpktdrop, last(QoS5uplinkpktdrop) as pQoS5uplinkpktdrop,
last(QoS6dwlinkpktdrop) as pQoS6dwlinkpktdrop, last(QoS6uplinkpktdrop) as pQoS6uplinkpktdrop,
last(QoS7dwlinkpktdrop) as pQoS7dwlinkpktdrop, last(QoS7uplinkpktdrop) as pQoS7uplinkpktdrop,
last(QoS8dwlinkpktdrop) as pQoS8dwlinkpktdrop, last(QoS8uplinkpktdrop) as pQoS8uplinkpktdrop,
last(QoS9dwlinkpktdrop) as pQoS9dwlinkpktdrop, last(QoS9uplinkpktdrop) as pQoS9uplinkpktdrop by group, host
| chart sum(eval((if(isnotnull(QoS1dwlinkpktdrop),QoS1dwlinkpktdrop,0) + if(isnotnull(QoS2dwlinkpktdrop),QoS2dwlinkpktdrop,0) + if(isnotnull(QoS3dwlinkpktdrop),QoS3dwlinkpktdrop,0) +
if(isnotnull(QoS4dwlinkpktdrop),QoS4dwlinkpktdrop,0) + if(isnotnull(QoS5dwlinkpktdrop),QoS5dwlinkpktdrop,0) + if(isnotnull(QoS6dwlinkpktdrop),QoS6dwlinkpktdrop,0) +
if(isnotnull(QoS7dwlinkpktdrop),QoS7dwlinkpktdrop,0) + if(isnotnull(QoS8dwlinkpktdrop),QoS8dwlinkpktdrop,0) + if(isnotnull(QoS9dwlinkpktdrop),QoS9dwlinkpktdrop,0)) -
(if(isnotnull(pQoS1dwlinkpktdrop),pQoS1dwlinkpktdrop,0) + if(isnotnull(pQoS2dwlinkpktdrop),pQoS2dwlinkpktdrop,0) + if(isnotnull(pQoS3dwlinkpktdrop),pQoS3dwlinkpktdrop,0) +
if(isnotnull(pQoS4dwlinkpktdrop),pQoS4dwlinkpktdrop,0) + if(isnotnull(pQoS5dwlinkpktdrop),pQoS5dwlinkpktdrop,0) + if(isnotnull(pQoS6dwlinkpktdrop),pQoS6dwlinkpktdrop,0) +
if(isnotnull(pQoS7dwlinkpktdrop),pQoS7dwlinkpktdrop,0) + if(isnotnull(pQoS8dwlinkpktdrop),pQoS8dwlinkpktdrop,0) + if(isnotnull(pQoS9dwlinkpktdrop),pQoS9dwlinkpktdrop,0)))) as DW-Link,
sum(eval((if(isnotnull(QoS1uplinkpktdrop),QoS1uplinkpktdrop,0) + if(isnotnull(QoS2uplinkpktdrop),QoS2uplinkpktdrop,0) + if(isnotnull(QoS3uplinkpktdrop),QoS3uplinkpktdrop,0) +
if(isnotnull(QoS4uplinkpktdrop),QoS4uplinkpktdrop,0) + if(isnotnull(QoS5uplinkpktdrop),QoS5uplinkpktdrop,0) + if(isnotnull(QoS6uplinkpktdrop),QoS6uplinkpktdrop,0) +
if(isnotnull(QoS7uplinkpktdrop),QoS7uplinkpktdrop,0) + if(isnotnull(QoS8uplinkpktdrop),QoS8uplinkpktdrop,0) + if(isnotnull(QoS9uplinkpktdrop),QoS9uplinkpktdrop,0)) -
(if(isnotnull(pQoS1uplinkpktdrop),pQoS1uplinkpktdrop,0) + if(isnotnull(pQoS2uplinkpktdrop),pQoS2uplinkpktdrop,0) + if(isnotnull(pQoS3uplinkpktdrop),pQoS3uplinkpktdrop,0) +
if(isnotnull(pQoS4uplinkpktdrop),pQoS4uplinkpktdrop,0) + if(isnotnull(pQoS5uplinkpktdrop),pQoS5uplinkpktdrop,0) + if(isnotnull(pQoS6uplinkpktdrop),pQoS6uplinkpktdrop,0) +
if(isnotnull(pQoS7uplinkpktdrop),pQoS7uplinkpktdrop,0) + if(isnotnull(pQoS8uplinkpktdrop),pQoS8uplinkpktdrop,0) + if(isnotnull(pQoS9uplinkpktdrop),pQoS9uplinkpktdrop,0)))) as UP-Link by _time, host

0 Karma