Alerting

Configure an alert based on value of a dynamic field

vilashegde
New Member

One of our application logs prints the queue size for multiple users. Also, the same application is running on multiple hosts, the logs that we are indexing through contains queuesize info for multiple users across multiple hosts. Also, since the queuesize and user names are not standard values, I've re-named them within the query.

The below search string lists the queuesize data according to the user and host. I wished to setup an alert only when the queuesize for any user on any node goes above 1000. I've tried using "where" clause but that does not work for some reason. Here is the sample query and the sample output -

host="*event*" AND "Queue size for" | stats first(field19) as QueueSize by field17, host |  rename field17 as User, field19 as QueueSize | sort -QueueSize, User

Gives me this:

Client  host    QueueSize
A         Server1   0;
A         Server2   0;
B         Server1   0;
B         Server2   0;
C         Server1   0;
C         Server2   0;
D         Server1   0;
D         Server2   0;

I want to be able to alert when the queuesize for any user on any server goes above 1000

Tags (2)
0 Karma

to4kawa
Ultra Champion

Simply:

host="*event*" "Queue size for" field19>1000

fire alert, event count >0

0 Karma

woodcock
Esteemed Legend

What you have should work just fine. you can use either search QueueSize > 1000 or where QueueSize > 1000;

index="YouShouldAlwaysSpecifyAnIndex" AND sourcetype="AndSourcetypeToo" AND host="*event*" AND "Queue size for"
| stats first(field19) AS QueueSize BY field17, host
| rename field17 AS User, field19 AS QueueSize
| sort 0 -QueueSize, User
| where QueueSize > 1000

NEVER use sort without a number.

0 Karma

vilashegde
New Member

when I use the below search clause at the end, it is showing any positive value above 0.

| search QueueSize > 1000

Using | where QueueSize > 1000 at the end, is still not giving me the expected results.

0 Karma

jpolvino
Builder

You are trying to access a field that gets thrown away with stats command:

host="event" AND "Queue size for" | stats first(field19) as QueueSize by field17, host | rename field17 as User, field19 as QueueSize | sort -QueueSize, User

After stats the only fields available are QueueSize, field17, and host. So trying to use field19 gives you nothing for that field. Try removing , field19 as QueueSize

vilashegde
New Member

But that still does not help filter with the where clause.

0 Karma
Get Updates on the Splunk Community!

Enter the Splunk Community Dashboard Challenge for Your Chance to Win!

The Splunk Community Dashboard Challenge is underway! This is your chance to showcase your skills in creating ...

.conf24 | Session Scheduler is Live!!

.conf24 is happening June 11 - 14 in Las Vegas, and we are thrilled to announce that the conference catalog ...

Introducing the Splunk Community Dashboard Challenge!

Welcome to Splunk Community Dashboard Challenge! This is your chance to showcase your skills in creating ...