Alerting

Can you enrich alerts in Splunk to provide instructions to the support team?

New Member

I am looking to use Splunk as our Manager of Managers at our job but from what I have read so far it can not seem to do what I need it to do. What I want to do is to have an alert come in and be able to provide clear instructions for our operations team.

To give you an example right now Ops use IBM Tivoli Integrated Portal (TIP) . When an alert comes in it creates an alert on the screen with several pieces of information. You can double click on it and view a field called "Instructions" that has specific instructions for our operations team. The way that this is populated (and emails for that matter) is by running a script that runs a query against a database. Based on what is returned it will use the first rows instruction field and email list and insert it into TIP.

For example lets say that my database may have these rows (.* means everything):

RowNum  Server Name  Alert Group  Alert Key     Instructions
1       ServerA      FileSystem   /db_backup    Call DB
2       ServerA      FileSystem   /myapp        Call app owner
3       ServerA      FileSystem   .\*            Call OS
4       ServerA      .\*           .\*            Call app owner
5       ServerB      .\*           .\*            Call OS

Example 1
Information:

ServerA has the mountpoint /ourapp go over 80%.

Result:
It would match rows 2-4 and take the instructions from row 2 (the first row returned that matches) would be put in the alert.


Example 2
Information:

ServerA has /etc go over 80%.

Result:
It would match rows 3-4 and take the instructions from row 3 (the first row returned that matches) would be put in the alert.


Example 3
Information:

ServerA has memory go over 80%.

Result:
It would match row 4 and take the instructions from row 4 (the first and only row returned that matches) would be put in the alert.


Example 4
Information:

ServerC has memory go over 80%.

Result:
It would not match any rows and default instructions to notify the monitoring team would be put in the alert.

Tags (2)
0 Karma

Path Finder

You should get the enrichment data into a lookup file and do a query like below.

Search for alert and eval alert_type | lookup instructions_lookup alert_type output instructions