Advanced triggered alert search

[africates]

Hi,

I am working on a project where we will be monitoring the windows backup logs from all our servers. The idea is to create a splunk alert whenever there are backup process that did not start, or have started but not finished, or have started but failed. If this alert is triggered, an email will be sent to admin with the list of servers that met the condition.

So far, I have sourced out the event ID's from the windows backup logs that I needed for the search;

• EventCode=1 - Windows backup started
• EventCode=4 - Backup Successful

This can be easily done by creating an alert that searches the eventcodes from a single server and triggers if there are no result. Now my problem is that we have at least 12 servers. Does this mean that i have to create an alert item for each server? - or is there any easier way to do this with just one alert item? or is there an app/addon that easily does this?

Thanks in advance for any suggestions.

[renjith_nair]

@africates,

You should be able to do that in one search

"your search" AND (EventCode=1 OR EventCode=4)|stats count by host
|append [|metadata type=hosts index="your index for backup"|fields host,count]
|fillnull count
|stats sum(count) as events by server | where count <2

• Find all events which have both eventCodes and count by server
• Compare it against all servers and find those servers which has event count < 2

Alternatively you could replace the metadata section with a lookup file with all hostnames as well

Test and let's know if it works for your requirement.

[africates]

Hi Renjith,

Thanks for your suggestion. It may be working as required but I will need to test it fully by pausing some of the server backup.

Can you briefly explain how this search works? So, it's basically getting the list of all server from the index accumulated over time - is this right? or is this just within the period specified in the search (in my case in the Last 24HRS) ?

What would the result look like if the condition is met? - is it just the name of the host/server?

Thanks again..

[renjith_nair · Accepted solution]

@africates ,

The search is  based on the time range you select. 

It works this way,

Assume you have 3 servers server1,server2,server3 

Server1 has both events = successful backup

Server 2 has only 1 event = Failed back up

Server3 has no events = Back up didn't run

"your search" AND (EventCode=1 OR EventCode=4)|stats count by host

will result 

host	count
server1	2
server2	1

No entry for server3

|metadata type=hosts index="your index for backup"|fields host,count]|fillnull count

host	count
server1	0
server2	0
server3	0

|stats sum(count) as events by server

host	count
server1	2
server2	1
server3	0

Hope that clarifies

Please upvote if its helpful 

[africates]

Thanks renjith, the solution worked.