Deployment Architecture

Scheduled searches and alerts on Cluster and Search Head

trodenbaugh
Explorer

I'm evaluating moving to a clustered configuration and utilizing the search head. I'm trying to determine how the search head manages scheduled searches and alerts. Specifically where is the savedsearches.conf file located and how do we allow others to create new saved searches and update those saved searches? How does the search head then manage the scheduling of the scheduled searches and alerts?

Regards,
Tom

0 Karma

gbowden_pheaa
Path Finder

Now that I have moved to a search head cluster from a search head pooling (v6.1.1 to v6.2.1), I am getting multiple sent alerts for a single alert. I was able to control this in 6.1.1 by enabling only 1 search head to send e-mail, but would this approach work in a cluster?

I am confused because I have 3 search heads in the cluster, but the cluster sends 2 of each alert, not 1 or 3 as I would expect.

Is there a way to determine which search head actually sends the alerts?

martin_muellar, would you explain why you feel the configuration in a cluster is irrelevant? It was my understanding the SH cluster captain would manage this, but I obviously have a disconnect somewhere.

Also - how should app objects created by users, specifically alerts, be managed if differing configurations are used to control this situation?

Thanks to all in advance.

0 Karma

jeremiahc4
Builder

Kind of a late add, but there's a known problem with multiple search heads sending alerts that was fixed somewhere around 6.2.4 - 6.2.6 release. It fixed the problem with our search heads, but I'm searching for a new problem where our indexers are sending alerts when they shouldn't be.

0 Karma

jnicholsenernoc
Path Finder

gbowden, you can tell which search head sent the alerts by updating the alert_actions.conf file and setting the hostname to be something uniquely identifiable. That's how you can tell.

Just a guess based on past issues like this, are all your clocks NTP'd or sync'd? Sounds like one may be a head.

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

The savedsearches.conf is stored on the cluster's search head in the same place you store it now on your current search head (which may be a combined search head & indexer instance if you have a single standalone splunk server).

Others add and edit saved searches in a cluster as they do with a standalone server.

Scheduling and alerting works the same way as well, the search head runs a search on a schedule and possibly triggers alert actions. Whether it performs a distributed search or only searches its own index is fairly irrelevant.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...