topic Re: Configure an alert based on value of a dynamic field in Alerting

Configure an alert based on value of a dynamic field

vilashegde — Fri, 17 Jan 2020 14:20:05 GMT

One of our application logs prints the queue size for multiple users. Also, the same application is running on multiple hosts, the logs that we are indexing through contains queuesize info for multiple users across multiple hosts. Also, since the queuesize and user names are not standard values, I've re-named them within the query.

The below search string lists the queuesize data according to the user and host. I wished to setup an alert only when the queuesize for any user on any node goes above 1000. I've tried using "where" clause but that does not work for some reason. Here is the sample query and the sample output -

host="*event*" AND "Queue size for" | stats first(field19) as QueueSize by field17, host |  rename field17 as User, field19 as QueueSize | sort -QueueSize, User

Gives me this:

Client  host    QueueSize
A         Server1   0;
A         Server2   0;
B         Server1   0;
B         Server2   0;
C         Server1   0;
C         Server2   0;
D         Server1   0;
D         Server2   0;

I want to be able to alert when the queuesize for any user on any server goes above 1000

Re: Configure an alert based on value of a dynamic field

jpolvino — Fri, 17 Jan 2020 15:01:32 GMT

You are trying to access a field that gets thrown away with stats command:

host="event" AND "Queue size for" | stats first(field19) as QueueSize by field17, host | rename field17 as User, field19 as QueueSize | sort -QueueSize, User

After stats the only fields available are QueueSize, field17, and host. So trying to use field19 gives you nothing for that field. Try removing , field19 as QueueSize

Re: Configure an alert based on value of a dynamic field

vilashegde — Sat, 18 Jan 2020 12:50:29 GMT

But that still does not help filter with the where clause.

Re: Configure an alert based on value of a dynamic field

woodcock — Sat, 18 Jan 2020 19:54:09 GMT

What you have should work just fine. you can use either search QueueSize > 1000 or where QueueSize > 1000;

index="YouShouldAlwaysSpecifyAnIndex" AND sourcetype="AndSourcetypeToo" AND host="*event*" AND "Queue size for"
| stats first(field19) AS QueueSize BY field17, host
| rename field17 AS User, field19 AS QueueSize
| sort 0 -QueueSize, User
| where QueueSize > 1000

NEVER use sort without a number.

Re: Configure an alert based on value of a dynamic field

to4kawa — Sun, 19 Jan 2020 03:54:59 GMT

Simply:

host="*event*" "Queue size for" field19>1000

fire alert, event count >0

Re: Configure an alert based on value of a dynamic field

vilashegde — Wed, 22 Jan 2020 11:31:45 GMT

when I use the below search clause at the end, it is showing any positive value above 0.

| search QueueSize > 1000

Using | where QueueSize > 1000 at the end, is still not giving me the expected results.