Alerting

How to create an email alert when a host stops sending data from a particular sourcetype?

macadminrohit
Contributor

Hi,

I have few sourcetypes which sometimes stops sending the data because of some indexer issues. Each hosts send data to multiple sourcetypes and I want to get an alert when any of the hosts stops sending the data for a particular sourcetype. I saw some old posts related to this but they don't exactly fit to my requirement.

Also in the email alert, I need to see which host is missing data from which sourcetype??

woodcock
Esteemed Legend

You need the Meta Woot! app:

https://splunkbase.splunk.com/app/2949/

mattymo
Splunk Employee
Splunk Employee

meta woot +1

- MattyMo
0 Karma

kiran331
Builder
0 Karma

skoelpin
SplunkTrust
SplunkTrust

You could run a simple negative search to alert on when it stops forwarding

Create a search to identify which sourcetype you want to monitor.. index=test host="172.30.29.188" sourcetype=access_combined

The picture below says, send an email alert when my sourcetype doesn't appear within a 10 minute window

alt text

0 Karma

macadminrohit
Contributor

this is a good solution, but in my case there are multiple hosts and sourcetypes. So if I create individual search queries the list will be huge, instead I would like to use the lookup option.

0 Karma

skoelpin
SplunkTrust
SplunkTrust

You can do this in the Monitoring Console (MC).. Go to Settings>Monitoring Console

Click the Forwarders tab and enable it.. Rebuild your assets by clicking the button and time interval

Then go to Settings while in the MC then Alert Settingsand choose DMC Alert Missing ForwardersEnable it, then Advanced Edit, then edit it like I showed you in the image above

This will then monitor all your forwarders for the missing source/sourcetype

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

SOC4Kafka - New Kafka Connector Powered by OpenTelemetry

The new SOC4Kafka connector, built on OpenTelemetry, enables the collection of Kafka messages and forwards ...

Event Series: Level up your SOC: Advancing with Splunk Enterprise Security

AI has fundamentally raised the stakes for security operations, and this three-part series is your guide to ...

Announcing Modern Navigation: A New Era of Splunk User Experience

We are excited to introduce the Modern Navigation feature in the Splunk Platform, available to both cloud and ...