How to create and trigger an alert when replication/search factors are not met on the indexer cluster master?

[rcreddy06]

The status of the replication factor and search factor on indexer cluster are fluctuating and would like to set up an alert. This happens due to bucket fixups pending. Can some one help me creating an alert when the master is fixing buckets and one of the search/replication factor is not met or when the master is trying to fix the buckets.

[Lucas_K · Accepted solution]

The clustering management console from which those three things are obtained has searches that drive them.

It can be shown by doing a "showsource" at the end of your url. ie. /en-US/app/splunk_management_console/indexer_clustering_status?showsource

You can also click the magnifying glass to break out the underlying search in the peers section to see the other searches in that dashboard.

This is the search that drives those 3 icons.

| rest splunk_server_group=dmc_group_cluster_master splunk_server_group=\"$group$\" /services/cluster/master/generation/master       | fields pending_last_reason, search_factor_met, replication_factor_met       | eval all_data_searchable_icon = if(isnull(pending_last_reason) or pending_last_reason==\"\", \"icon-check\", \"icon-alert\")       | eval all_data_searchable = if (isnull(pending_last_reason) or pending_last_reason==\"\", \"All Data is Searchable\", \"Some Data is Not Searchable\")       | eval search_factor_met_icon = if(search_factor_met == 1 or search_factor_met == \"1\", \"icon-check\", \"icon-alert\")       | eval search_factor_met = if (search_factor_met == 1 or search_factor_met == \"1\", \"Search Factor is Met\", \"Search Factor is Not Met\")       | eval replication_factor_met_icon = if(replication_factor_met == 1 or replication_factor_met == \"1\", \"icon-check\", \"icon-alert\")       | eval replication_factor_met = if (replication_factor_met == 1 or replication_factor_met == \"1\", \"Replication Factor is Met\", \"Replication Factor is Not Met\")",

Get your alert to trigger when any of those 3 factors are not equal to 1.

[halleyglen]

That shows an unknown group error & eval error on \"\", \"icon-check\", \"icon-alert\", for me.

[vasanthi77]

Hi Lucas or Halleyglen ,

Even i m getting the below error, how to fix this .

Error in 'eval' command: The expression is malformed. An unexpected character is reached at '\"1\", \"icon-check\", \"icon-alert\")'.

[Lucas_K]

My original search doesn't work anymore on recent versions of splunk as it doesn't like those escape characters in the evals.

You can just use the fields in the base search :

| rest /services/cluster/master/generation/master 
| fields pending_last_reason, search_factor_met, replication_factor_met

[vasanthi77]

Hi Lucas , Thanks for the reply , this is working.

However i m getting below errors along with output.

[indexer alias] REST Processor: Failed to fetch REST endpoint uri=https://127.0.0.1:8089/services/cluster/master/generation/master?count=0 from server https://127.0.0.1:8089. Check that the URI path provided exists in the REST API.

Should i worry about this?

[Lucas_K]

needs to be performed on the cluster master. Alternatively have dist search to the cluster master from your search heads but you'll have to add splunk_server= to the query.

[vasanthi77]

HI Lucas ,

I used below query from search head , not getting any result. Can you help here pls.

| rest /services/cluster/master/generation/master splunk_server=" Clustermaster"
| fields pending_last_reason, search_factor_met, replication_factor_met

[Lucas_K]

do you have distsearch to your cluster master? it wouldn't normally be enabled as it's not normally used as a search peer.

[Lucas_K]

also remove the white space in your splunk_server. You've quoted whitespace that will make it fail even if you have distsearch to the master configured.

[vasanthi77]

Thanks Lucas , Its working fine 🙂

[rcreddy06]

It worked for me

[sterlingteks]

how did you make it work??