I am looking to use Splunk as our Manager of Managers at our job but from what I have read so far it can not seem to do what I need it to do. What I want to do is to have an alert come in and be able to provide clear instructions for our operations team.
To give you an example right now Ops use IBM Tivoli Integrated Portal (TIP) . When an alert comes in it creates an alert on the screen with several pieces of information. You can double click on it and view a field called "Instructions" that has specific instructions for our operations team. The way that this is populated (and emails for that matter) is by running a script that runs a query against a database. Based on what is returned it will use the first rows instruction field and email list and insert it into TIP.
For example lets say that my database may have these rows (.* means everything):
RowNum Server Name Alert Group Alert Key Instructions
1 ServerA FileSystem /db_backup Call DB
2 ServerA FileSystem /myapp Call app owner
3 ServerA FileSystem .\* Call OS
4 ServerA .\* .\* Call app owner
5 ServerB .\* .\* Call OS
Example 1
Information:
ServerA has the mountpoint /ourapp go over 80%.
Result:
It would match rows 2-4 and take the instructions from row 2 (the first row returned that matches) would be put in the alert.
Example 2
Information:
ServerA has /etc go over 80%.
Result:
It would match rows 3-4 and take the instructions from row 3 (the first row returned that matches) would be put in the alert.
Example 3
Information:
ServerA has memory go over 80%.
Result:
It would match row 4 and take the instructions from row 4 (the first and only row returned that matches) would be put in the alert.
Example 4
Information:
ServerC has memory go over 80%.
Result:
It would not match any rows and default instructions to notify the monitoring team would be put in the alert.
