Hello Everyone,
I think that I may have a strange use case that I would love some help with.
I have a system that processes hundreds of thousands of messages.
The system categorises messages into 4 different queues: Express, High, Medium, Low.
The system has a performance monitor built into it, this perfmon will keep track of kpi's as the system runs and report the values in messages which we send to splunk.
Example message:
07/12/2018 17:12:42.815 +0200
collection="CollectionName"
object="MessageRate"
counter="High"
instance=123
Value=53.82524876723775
Now, the value that I am interested in is "Value", it is the percentage (0.0 - 100.0) of the total messages, taken up by the queue mentioned in the field "Counter" (Still with me?)
I.E.
Counter == name of the Queue,
Value == the % that the Queue represents out of the sum total of messages.
So to recap, perfmon is calculating what % of the total messages are in each queue, then sending a message PER QUEUE with the values.
What I am trying to do is to set up 4 dashboard panels that display these percentages (one per queue).
THE PROBLEM
If a Queue has no messages in it, perfmon is NOT sending a message, so the dashboard panel just says: N/A as it has no events to extract 0.0 from, because again, it is never sent by perfmon.
THE DESIRED OUTCOME
If a Queue has no messages in it the dashboard panel should just display 0 instead of N/A.
WHAT I THINK
I think that I need to some how set up a conditional search, or a nested search along the lines of:
if (searchForExpressMessages == null) . // if no messages in queue
Value == 0 . // display 0
else // there must be messages in the queue
Run query to pull Value out of the message
WHAT I TRIED
Many different ways of trying to get evals, rex's and fillnull working, but all to no avail, I can get the panels to display results, but they are not accurate at all, and are just ending up being counts of the number of different values of "Value"
Any help would be appreciated greatly.
I am heading home from work and will not look at this again for about 16 hours, just FYI.
Thank you in advance
... View more
