Alerting

Advanced triggered alert search

africates
Explorer

Hi,

I am working on a project where we will be monitoring the windows backup logs from all our servers. The idea is to create a splunk alert whenever there are backup process that did not start, or have started but not finished, or have started but failed. If this alert is triggered, an email will be sent to admin with the list of servers that met the condition.

So far, I have sourced out the event ID's from the windows backup logs that I needed for the search;

  • EventCode=1 - Windows backup started
  • EventCode=4 - Backup Successful

This can be easily done by creating an alert that searches the eventcodes from a single server and triggers if there are no result. Now my problem is that we have at least 12 servers. Does this mean that i have to create an alert item for each server? - or is there any easier way to do this with just one alert item? or is there an app/addon that easily does this?

Thanks in advance for any suggestions.

 

Labels (2)
0 Karma
1 Solution

renjith_nair
SplunkTrust
SplunkTrust

@africates ,

The search is  based on the time range you select. 

It works this way,

Assume you have 3 servers server1,server2,server3 

Server1 has both events = successful backup

Server 2 has only 1 event = Failed back up

Server3 has no events = Back up didn't run

 

"your search" AND (EventCode=1 OR EventCode=4)|stats count by host

 

will result 

hostcount
server12
server21

No entry for server3

 

|metadata type=hosts index="your index for backup"|fields host,count]|fillnull count

 

hostcount
server10
server20
server30

 

|stats sum(count) as events by server
hostcount
server12
server21
server30

 

Hope that clarifies

Please upvote if its helpful 

Happy Splunking!

View solution in original post

renjith_nair
SplunkTrust
SplunkTrust

@africates,

You should be able to do that in one search

"your search" AND (EventCode=1 OR EventCode=4)|stats count by host
|append [|metadata type=hosts index="your index for backup"|fields host,count]
|fillnull count
|stats sum(count) as events by server | where count <2
  • Find all events which have both eventCodes and count by server
  • Compare it against all servers and find those servers which has event count < 2

Alternatively you could replace the metadata section with a lookup file with all hostnames as well

Test and let's know if it works for your requirement.

Happy Splunking!
0 Karma

africates
Explorer

Hi Renjith,

Thanks for your suggestion. It may be working as required but I will need to test it fully by pausing some of the server backup.

Can you briefly explain how this search works? So, it's basically getting the list of all server from the index accumulated over time - is this right? or is this just within the period specified in the search (in my case in the Last 24HRS) ?

What would the result look like if the condition is met? - is it just the name of the host/server?

 

Thanks again..

Tags (1)
0 Karma

renjith_nair
SplunkTrust
SplunkTrust

@africates ,

The search is  based on the time range you select. 

It works this way,

Assume you have 3 servers server1,server2,server3 

Server1 has both events = successful backup

Server 2 has only 1 event = Failed back up

Server3 has no events = Back up didn't run

 

"your search" AND (EventCode=1 OR EventCode=4)|stats count by host

 

will result 

hostcount
server12
server21

No entry for server3

 

|metadata type=hosts index="your index for backup"|fields host,count]|fillnull count

 

hostcount
server10
server20
server30

 

|stats sum(count) as events by server
hostcount
server12
server21
server30

 

Hope that clarifies

Please upvote if its helpful 

Happy Splunking!

africates
Explorer

Thanks renjith, the solution worked.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...