Splunk Search

Sum latest entries from multiple sources & timechart as a single line

justingawn
New Member

Hello,

I have multiple remote performance monitors sources, namely WMI:FOO1, WMI:FOO2 etc. up to and including WMI:FOO9. These each report on a value called BAR every 1 minute.

I am attempting to generate a timechart with a single line representing the total value of BAR across all my WMI:FOOx sources.

Additionally, any solution needs to ignore previous values of a WMI:FOOx source if that source has not reported a value for BAR within a set time period, say 2 minutes (in case the server goes down, I don't want the last value being included in a sum.)

Thanks in advance,
Justin

0 Karma

mannioke
Engager

Well this isn't perfect to your case but could be a useful substitution, if you are will to accept the modifications in behavior. I have a very similar situation with multiple QoS across multiple Interfaces across multiple Hosts. Because some interfaces utilization is drastically different than another I want to watch per host, not per interface per host. Thus far any indication of no response appears to be reporting as a gap in my chart.

My goal was to measure the interval change (new - old = delta), my only problem now is because of the forced values my first interval has an over inflated value that I need to drop. A sub search hasn't worked thus far, but this is a work in progress.

The forced values were required as a result of any of the QoS values could be null and that would result in a null sum regardless if the other 8 QoS contained values.

index=bulkstats (host=host-01 OR host=host-02) QoS8_dwlinkpkt_drop=* QoS8_uplinkpkt_drop=*
| reverse
| streamstats global=f current=f
last(QoS1_dwlinkpkt_drop) as p_QoS1_dwlinkpkt_drop, last(QoS1_uplinkpkt_drop) as p_QoS1_uplinkpkt_drop,
last(QoS2_dwlinkpkt_drop) as p_QoS2_dwlinkpkt_drop, last(QoS2_uplinkpkt_drop) as p_QoS2_uplinkpkt_drop,
last(QoS3_dwlinkpkt_drop) as p_QoS3_dwlinkpkt_drop, last(QoS3_uplinkpkt_drop) as p_QoS3_uplinkpkt_drop,
last(QoS4_dwlinkpkt_drop) as p_QoS4_dwlinkpkt_drop, last(QoS4_uplinkpkt_drop) as p_QoS4_uplinkpkt_drop,
last(QoS5_dwlinkpkt_drop) as p_QoS5_dwlinkpkt_drop, last(QoS5_uplinkpkt_drop) as p_QoS5_uplinkpkt_drop,
last(QoS6_dwlinkpkt_drop) as p_QoS6_dwlinkpkt_drop, last(QoS6_uplinkpkt_drop) as p_QoS6_uplinkpkt_drop,
last(QoS7_dwlinkpkt_drop) as p_QoS7_dwlinkpkt_drop, last(QoS7_uplinkpkt_drop) as p_QoS7_uplinkpkt_drop,
last(QoS8_dwlinkpkt_drop) as p_QoS8_dwlinkpkt_drop, last(QoS8_uplinkpkt_drop) as p_QoS8_uplinkpkt_drop,
last(QoS9_dwlinkpkt_drop) as p_QoS9_dwlinkpkt_drop, last(QoS9_uplinkpkt_drop) as p_QoS9_uplinkpkt_drop by group, host
| chart sum(eval((if(isnotnull(QoS1_dwlinkpkt_drop),QoS1_dwlinkpkt_drop,0) + if(isnotnull(QoS2_dwlinkpkt_drop),QoS2_dwlinkpkt_drop,0) + if(isnotnull(QoS3_dwlinkpkt_drop),QoS3_dwlinkpkt_drop,0) +
if(isnotnull(QoS4_dwlinkpkt_drop),QoS4_dwlinkpkt_drop,0) + if(isnotnull(QoS5_dwlinkpkt_drop),QoS5_dwlinkpkt_drop,0) + if(isnotnull(QoS6_dwlinkpkt_drop),QoS6_dwlinkpkt_drop,0) +
if(isnotnull(QoS7_dwlinkpkt_drop),QoS7_dwlinkpkt_drop,0) + if(isnotnull(QoS8_dwlinkpkt_drop),QoS8_dwlinkpkt_drop,0) + if(isnotnull(QoS9_dwlinkpkt_drop),QoS9_dwlinkpkt_drop,0)) -
(if(isnotnull(p_QoS1_dwlinkpkt_drop),p_QoS1_dwlinkpkt_drop,0) + if(isnotnull(p_QoS2_dwlinkpkt_drop),p_QoS2_dwlinkpkt_drop,0) + if(isnotnull(p_QoS3_dwlinkpkt_drop),p_QoS3_dwlinkpkt_drop,0) +
if(isnotnull(p_QoS4_dwlinkpkt_drop),p_QoS4_dwlinkpkt_drop,0) + if(isnotnull(p_QoS5_dwlinkpkt_drop),p_QoS5_dwlinkpkt_drop,0) + if(isnotnull(p_QoS6_dwlinkpkt_drop),p_QoS6_dwlinkpkt_drop,0) +
if(isnotnull(p_QoS7_dwlinkpkt_drop),p_QoS7_dwlinkpkt_drop,0) + if(isnotnull(p_QoS8_dwlinkpkt_drop),p_QoS8_dwlinkpkt_drop,0) + if(isnotnull(p_QoS9_dwlinkpkt_drop),p_QoS9_dwlinkpkt_drop,0)))) as DW-Link,
sum(eval((if(isnotnull(QoS1_uplinkpkt_drop),QoS1_uplinkpkt_drop,0) + if(isnotnull(QoS2_uplinkpkt_drop),QoS2_uplinkpkt_drop,0) + if(isnotnull(QoS3_uplinkpkt_drop),QoS3_uplinkpkt_drop,0) +
if(isnotnull(QoS4_uplinkpkt_drop),QoS4_uplinkpkt_drop,0) + if(isnotnull(QoS5_uplinkpkt_drop),QoS5_uplinkpkt_drop,0) + if(isnotnull(QoS6_uplinkpkt_drop),QoS6_uplinkpkt_drop,0) +
if(isnotnull(QoS7_uplinkpkt_drop),QoS7_uplinkpkt_drop,0) + if(isnotnull(QoS8_uplinkpkt_drop),QoS8_uplinkpkt_drop,0) + if(isnotnull(QoS9_uplinkpkt_drop),QoS9_uplinkpkt_drop,0)) -
(if(isnotnull(p_QoS1_uplinkpkt_drop),p_QoS1_uplinkpkt_drop,0) + if(isnotnull(p_QoS2_uplinkpkt_drop),p_QoS2_uplinkpkt_drop,0) + if(isnotnull(p_QoS3_uplinkpkt_drop),p_QoS3_uplinkpkt_drop,0) +
if(isnotnull(p_QoS4_uplinkpkt_drop),p_QoS4_uplinkpkt_drop,0) + if(isnotnull(p_QoS5_uplinkpkt_drop),p_QoS5_uplinkpkt_drop,0) + if(isnotnull(p_QoS6_uplinkpkt_drop),p_QoS6_uplinkpkt_drop,0) +
if(isnotnull(p_QoS7_uplinkpkt_drop),p_QoS7_uplinkpkt_drop,0) + if(isnotnull(p_QoS8_uplinkpkt_drop),p_QoS8_uplinkpkt_drop,0) + if(isnotnull(p_QoS9_uplinkpkt_drop),p_QoS9_uplinkpkt_drop,0)))) as UP-Link by _time, host

0 Karma

jtrucks
Splunk Employee
Splunk Employee

First, to get results from multiple sources, use this in your search:

source=WMI:FOO*

So, something like:

source=WMI:FOO* BAR=* | timechart span=1m sum(BAR) useother=false

Could you clarify (in comments or by editing the question) what specifically you are trying to ignore? You can specify stuff like NOT BAR=0 or things like that...

--
Jesse Trucks
Minister of Magic
0 Karma

justingawn
New Member

Regarding the ignoring of values - if the solution to above does involve summing of last BAR values of each WMI:FOOx source, I would want to ensure that in the event of a server being unreachable, that its last BAR value is not summed into the graphed value.

0 Karma

justingawn
New Member

I had tried something similar, a sum with a span equal to the poll interval.

Occasionally a server would respond immediately resulting in 2 results for the same source within the 60s span. Conversely sometimes a server would take a while to respond meaning that there would be no value for the source counted within the span.

This resulted in blips of +/- a single BAR value on the graph. Tweaking the span would either decrease duplicate source counts and increase missed counts, or vice versa.

Can a sum be done on the last values for each WMI:FOOx source, rather than rely on a time based span?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...