topic Re: How to create and trigger an alert when replication/search factors are not met on the indexer cluster master? in Alerting

How to create and trigger an alert when replication/search factors are not met on the indexer cluster master?

rcreddy06 — Sun, 22 Nov 2015 16:31:36 GMT

The status of the replication factor and search factor on indexer cluster are fluctuating and would like to set up an alert. This happens due to bucket fixups pending. Can some one help me creating an alert when the master is fixing buckets and one of the search/replication factor is not met or when the master is trying to fix the buckets.

Re: How to create and trigger an alert when replication/search factors are not met on the indexer cluster master?

Lucas_K — Tue, 29 Sep 2020 07:57:22 GMT

The clustering management console from which those three things are obtained has searches that drive them.

It can be shown by doing a "showsource" at the end of your url. ie. /en-US/app/splunk_management_console/indexer_clustering_status?showsource

You can also click the magnifying glass to break out the underlying search in the peers section to see the other searches in that dashboard.

This is the search that drives those 3 icons.

| rest splunk_server_group=dmc_group_cluster_master splunk_server_group=\"$group$\" /services/cluster/master/generation/master       | fields pending_last_reason, search_factor_met, replication_factor_met       | eval all_data_searchable_icon = if(isnull(pending_last_reason) or pending_last_reason==\"\", \"icon-check\", \"icon-alert\")       | eval all_data_searchable = if (isnull(pending_last_reason) or pending_last_reason==\"\", \"All Data is Searchable\", \"Some Data is Not Searchable\")       | eval search_factor_met_icon = if(search_factor_met == 1 or search_factor_met == \"1\", \"icon-check\", \"icon-alert\")       | eval search_factor_met = if (search_factor_met == 1 or search_factor_met == \"1\", \"Search Factor is Met\", \"Search Factor is Not Met\")       | eval replication_factor_met_icon = if(replication_factor_met == 1 or replication_factor_met == \"1\", \"icon-check\", \"icon-alert\")       | eval replication_factor_met = if (replication_factor_met == 1 or replication_factor_met == \"1\", \"Replication Factor is Met\", \"Replication Factor is Not Met\")",

Get your alert to trigger when any of those 3 factors are not equal to 1.

Re: How to create and trigger an alert when replication/search factors are not met on the indexer cluster master?

halleyglen — Thu, 26 Nov 2015 14:13:06 GMT

That shows an unknown group error & eval error on \"\", \"icon-check\", \"icon-alert\", for me.

Re: How to create and trigger an alert when replication/search factors are not met on the indexer cluster master?

rcreddy06 — Thu, 31 Dec 2015 16:26:12 GMT

It worked for me

Re: How to create and trigger an alert when replication/search factors are not met on the indexer cluster master?

sterlingteks — Wed, 31 Oct 2018 18:39:34 GMT

how did you make it work??

Re: How to create and trigger an alert when replication/search factors are not met on the indexer cluster master?

vasanthi77 — Mon, 29 Apr 2019 05:38:53 GMT

Hi Lucas or Halleyglen ,

Even i m getting the below error, how to fix this .

Error in 'eval' command: The expression is malformed. An unexpected character is reached at '\"1\", \"icon-check\", \"icon-alert\")'.

Re: How to create and trigger an alert when replication/search factors are not met on the indexer cluster master?

Lucas_K — Mon, 29 Apr 2019 06:18:30 GMT

My original search doesn't work anymore on recent versions of splunk as it doesn't like those escape characters in the evals.

You can just use the fields in the base search :

| rest /services/cluster/master/generation/master 
| fields pending_last_reason, search_factor_met, replication_factor_met

Re: How to create and trigger an alert when replication/search factors are not met on the indexer cluster master?

vasanthi77 — Mon, 29 Apr 2019 11:53:18 GMT

Hi Lucas , Thanks for the reply , this is working.

However i m getting below errors along with output.

[indexer alias] REST Processor: Failed to fetch REST endpoint uri=https://127.0.0.1:8089/services/cluster/master/generation/master?count=0 from server https://127.0.0.1:8089. Check that the URI path provided exists in the REST API.

Should i worry about this?

Re: How to create and trigger an alert when replication/search factors are not met on the indexer cluster master?

Lucas_K — Mon, 29 Apr 2019 23:06:14 GMT

needs to be performed on the cluster master. Alternatively have dist search to the cluster master from your search heads but you'll have to add splunk_server= to the query.

Re: How to create and trigger an alert when replication/search factors are not met on the indexer cluster master?

vasanthi77 — Wed, 30 Sep 2020 00:15:58 GMT

HI Lucas ,

I used below query from search head , not getting any result. Can you help here pls.

| rest /services/cluster/master/generation/master splunk_server=" Clustermaster"
| fields pending_last_reason, search_factor_met, replication_factor_met

Re: How to create and trigger an alert when replication/search factors are not met on the indexer cluster master?

Lucas_K — Tue, 30 Apr 2019 02:18:16 GMT

do you have distsearch to your cluster master? it wouldn't normally be enabled as it's not normally used as a search peer.

Re: How to create and trigger an alert when replication/search factors are not met on the indexer cluster master?

Lucas_K — Tue, 30 Apr 2019 02:19:50 GMT

also remove the white space in your splunk_server. You've quoted whitespace that will make it fail even if you have distsearch to the master configured.

Re: How to create and trigger an alert when replication/search factors are not met on the indexer cluster master?

vasanthi77 — Tue, 30 Apr 2019 06:03:45 GMT

Thanks Lucas , Its working fine 🙂