I have few sourcetypes which sometimes stops sending the data because of some indexer issues. Each hosts send data to multiple sourcetypes and I want to get an alert when any of the hosts stops sending the data for a particular sourcetype. I saw some old posts related to this but they don't exactly fit to my requirement.
Also in the email alert, I need to see which host is missing data from which sourcetype??
You could run a simple negative search to alert on when it stops forwarding
Create a search to identify which sourcetype you want to monitor..
index=test host="172.30.29.188" sourcetype=access_combined
The picture below says, send an email alert when my sourcetype doesn't appear within a 10 minute window
this is a good solution, but in my case there are multiple hosts and sourcetypes. So if I create individual search queries the list will be huge, instead I would like to use the lookup option.
You can do this in the Monitoring Console (MC).. Go to
Click the Forwarders tab and enable it.. Rebuild your assets by clicking the button and time interval
Then go to Settings while in the MC then
Alert Settingsand choose
DMC Alert Missing ForwardersEnable it, then Advanced Edit, then edit it like I showed you in the image above
This will then monitor all your forwarders for the missing source/sourcetype