Alternatives to "No Results Found" (fillnull is not applicable unfortunately)

[ianbruton]

Hello Everyone,

I think that I may have a strange use case that I would love some help with.

I have a system that processes hundreds of thousands of messages.

The system categorises messages into 4 different queues: Express, High, Medium, Low.

The system has a performance monitor built into it, this perfmon will keep track of kpi's as the system runs and report the values in messages which we send to splunk.

Example message:

07/12/2018 17:12:42.815 +0200
collection="CollectionName"
object="MessageRate"
counter="High"
instance=123
Value=53.82524876723775

Now, the value that I am interested in is "Value", it is the percentage (0.0 - 100.0) of the total messages, taken up by the queue mentioned in the field "Counter" (Still with me?)

I.E.
Counter == name of the Queue,
Value == the % that the Queue represents out of the sum total of messages.

So to recap, perfmon is calculating what % of the total messages are in each queue, then sending a message PER QUEUE with the values.

What I am trying to do is to set up 4 dashboard panels that display these percentages (one per queue).

THE PROBLEM

If a Queue has no messages in it, perfmon is NOT sending a message, so the dashboard panel just says: N/A as it has no events to extract 0.0 from, because again, it is never sent by perfmon.

THE DESIRED OUTCOME

If a Queue has no messages in it the dashboard panel should just display 0 instead of N/A.

WHAT I THINK

I think that I need to some how set up a conditional search, or a nested search along the lines of:

if (searchForExpressMessages == null) .           // if no messages in queue
    Value == 0 .                                  // display 0
else                                              // there must be messages in the queue
    Run query to pull Value out of the message

WHAT I TRIED

Many different ways of trying to get evals, rex's and fillnull working, but all to no avail, I can get the panels to display results, but they are not accurate at all, and are just ending up being counts of the number of different values of "Value"

Any help would be appreciated greatly.

I am heading home from work and will not look at this again for about 16 hours, just FYI.

Thank you in advance

[DalJeanis]

Um, no. please don't think in terms of conditional searches in splunk, there's no such thing that you want to attempt. (Map is a method, but it's expensive and almost always the wrong approach.)

You didn't include your code, so we don't know whether you are doing a base search that gets all 4 and then filtering, or what. I will assume not.

You only want one record, so just append one default zero record onto the end of your results, then | head 1. If there were no results, you will get your default zero record, otherwise you will get the actual record. Done.

your search that creates either 1 or zero records for counter="Express"
| append [|makeresults | eval  counter="Express",  Percentage=0 | table counter Percentage]
| head 1

If you have a single base search that gives all four, then you do this, with the same result

your search that creates either 1 or zero records for counter="Express"
| append [|makeresults | eval  counter=mvappend("Express","High","Medium","Low"),  Percentage=0 | mvexpand counter | table counter Percentage]
| dedup counter

[somesoni2]

What is the panel search that you're using? You probably need an appendpipe-stats subsearch to achieve what you want.

e.g. if you panel search is like this

index=foo sourcetype=bar counter="Express" | stats latest(Value) as Percentage

Then try like this

index=foo sourcetype=bar counter="Express" | stats latest(Value) as Percentage | appendpipe [| stats count as Percentage | where Percentage=0 ]