Scheduled searches and alerts on Cluster and Search Head

[trodenbaugh]

I'm evaluating moving to a clustered configuration and utilizing the search head. I'm trying to determine how the search head manages scheduled searches and alerts. Specifically where is the savedsearches.conf file located and how do we allow others to create new saved searches and update those saved searches? How does the search head then manage the scheduling of the scheduled searches and alerts?

Regards,
Tom

[gbowden_pheaa]

Now that I have moved to a search head cluster from a search head pooling (v6.1.1 to v6.2.1), I am getting multiple sent alerts for a single alert. I was able to control this in 6.1.1 by enabling only 1 search head to send e-mail, but would this approach work in a cluster?

I am confused because I have 3 search heads in the cluster, but the cluster sends 2 of each alert, not 1 or 3 as I would expect.

Is there a way to determine which search head actually sends the alerts?

martin_muellar, would you explain why you feel the configuration in a cluster is irrelevant? It was my understanding the SH cluster captain would manage this, but I obviously have a disconnect somewhere.

Also - how should app objects created by users, specifically alerts, be managed if differing configurations are used to control this situation?

Thanks to all in advance.

[jeremiahc4]

Kind of a late add, but there's a known problem with multiple search heads sending alerts that was fixed somewhere around 6.2.4 - 6.2.6 release. It fixed the problem with our search heads, but I'm searching for a new problem where our indexers are sending alerts when they shouldn't be.

[jnicholsenernoc]

gbowden, you can tell which search head sent the alerts by updating the alert_actions.conf file and setting the hostname to be something uniquely identifiable. That's how you can tell.

Just a guess based on past issues like this, are all your clocks NTP'd or sync'd? Sounds like one may be a head.

[martin_mueller]

The savedsearches.conf is stored on the cluster's search head in the same place you store it now on your current search head (which may be a combined search head & indexer instance if you have a single standalone splunk server).

Others add and edit saved searches in a cluster as they do with a standalone server.

Scheduling and alerting works the same way as well, the search head runs a search on a schedule and possibly triggers alert actions. Whether it performs a distributed search or only searches its own index is fairly irrelevant.