topic Advanced triggered alert search in Alerting

Advanced triggered alert search

africates — Thu, 02 Jul 2020 09:12:26 GMT

Hi,

I am working on a project where we will be monitoring the windows backup logs from all our servers. The idea is to create a splunk alert whenever there are backup process that did not start, or have started but not finished, or have started but failed. If this alert is triggered, an email will be sent to admin with the list of servers that met the condition.

So far, I have sourced out the event ID's from the windows backup logs that I needed for the search;

EventCode=1 - Windows backup started
EventCode=4 - Backup Successful

This can be easily done by creating an alert that searches the eventcodes from a single server and triggers if there are no result. Now my problem is that we have at least 12 servers. Does this mean that i have to create an alert item for each server? - or is there any easier way to do this with just one alert item? or is there an app/addon that easily does this?

Thanks in advance for any suggestions.

Re: Advanced triggered alert search

renjith_nair — Thu, 02 Jul 2020 10:19:54 GMT

@africates,

You should be able to do that in one search

Find all events which have both eventCodes and count by server
Compare it against all servers and find those servers which has event count < 2

Alternatively you could replace the metadata section with a lookup file with all hostnames as well

Test and let's know if it works for your requirement.

Re: Advanced triggered alert search

africates — Thu, 02 Jul 2020 10:56:41 GMT

Hi Renjith,

Thanks for your suggestion. It may be working as required but I will need to test it fully by pausing some of the server backup.

Can you briefly explain how this search works? So, it's basically getting the list of all server from the index accumulated over time - is this right? or is this just within the period specified in the search (in my case in the Last 24HRS) ?

What would the result look like if the condition is met? - is it just the name of the host/server?

Thanks again..

Re: Advanced triggered alert search

renjith_nair — Thu, 02 Jul 2020 13:37:44 GMT

@africates ,

The search is based on the time range you select.

It works this way,

Assume you have 3 servers server1,server2,server3

Server1 has both events = successful backup

Server 2 has only 1 event = Failed back up

Server3 has no events = Back up didn't run

"your search" AND (EventCode=1 OR EventCode=4)|stats count by host

will result

host	count
server1	2
server2	1

No entry for server3

|metadata type=hosts index="your index for backup"|fields host,count]|fillnull count

host	count
server1	0
server2	0
server3	0

|stats sum(count) as events by server

host	count
server1	2
server2	1
server3	0

Hope that clarifies

Please upvote if its helpful

Re: Advanced triggered alert search

africates — Fri, 14 Aug 2020 10:52:16 GMT

Thanks renjith, the solution worked.