Alerting

How to create and trigger an alert when replication/search factors are not met on the indexer cluster master?

Path Finder

The status of the replication factor and search factor on indexer cluster are fluctuating and would like to set up an alert. This happens due to bucket fixups pending. Can some one help me creating an alert when the master is fixing buckets and one of the search/replication factor is not met or when the master is trying to fix the buckets.

alt text

1 Solution

Motivator

The clustering management console from which those three things are obtained has searches that drive them.

It can be shown by doing a "showsource" at the end of your url. ie. /en-US/app/splunk_management_console/indexer_clustering_status?showsource

You can also click the magnifying glass to break out the underlying search in the peers section to see the other searches in that dashboard.

This is the search that drives those 3 icons.

| rest splunk_server_group=dmc_group_cluster_master splunk_server_group=\"$group$\" /services/cluster/master/generation/master       | fields pending_last_reason, search_factor_met, replication_factor_met       | eval all_data_searchable_icon = if(isnull(pending_last_reason) or pending_last_reason==\"\", \"icon-check\", \"icon-alert\")       | eval all_data_searchable = if (isnull(pending_last_reason) or pending_last_reason==\"\", \"All Data is Searchable\", \"Some Data is Not Searchable\")       | eval search_factor_met_icon = if(search_factor_met == 1 or search_factor_met == \"1\", \"icon-check\", \"icon-alert\")       | eval search_factor_met = if (search_factor_met == 1 or search_factor_met == \"1\", \"Search Factor is Met\", \"Search Factor is Not Met\")       | eval replication_factor_met_icon = if(replication_factor_met == 1 or replication_factor_met == \"1\", \"icon-check\", \"icon-alert\")       | eval replication_factor_met = if (replication_factor_met == 1 or replication_factor_met == \"1\", \"Replication Factor is Met\", \"Replication Factor is Not Met\")",

Get your alert to trigger when any of those 3 factors are not equal to 1.

View solution in original post

Motivator

The clustering management console from which those three things are obtained has searches that drive them.

It can be shown by doing a "showsource" at the end of your url. ie. /en-US/app/splunk_management_console/indexer_clustering_status?showsource

You can also click the magnifying glass to break out the underlying search in the peers section to see the other searches in that dashboard.

This is the search that drives those 3 icons.

| rest splunk_server_group=dmc_group_cluster_master splunk_server_group=\"$group$\" /services/cluster/master/generation/master       | fields pending_last_reason, search_factor_met, replication_factor_met       | eval all_data_searchable_icon = if(isnull(pending_last_reason) or pending_last_reason==\"\", \"icon-check\", \"icon-alert\")       | eval all_data_searchable = if (isnull(pending_last_reason) or pending_last_reason==\"\", \"All Data is Searchable\", \"Some Data is Not Searchable\")       | eval search_factor_met_icon = if(search_factor_met == 1 or search_factor_met == \"1\", \"icon-check\", \"icon-alert\")       | eval search_factor_met = if (search_factor_met == 1 or search_factor_met == \"1\", \"Search Factor is Met\", \"Search Factor is Not Met\")       | eval replication_factor_met_icon = if(replication_factor_met == 1 or replication_factor_met == \"1\", \"icon-check\", \"icon-alert\")       | eval replication_factor_met = if (replication_factor_met == 1 or replication_factor_met == \"1\", \"Replication Factor is Met\", \"Replication Factor is Not Met\")",

Get your alert to trigger when any of those 3 factors are not equal to 1.

View solution in original post

Explorer

That shows an unknown group error & eval error on \"\", \"icon-check\", \"icon-alert\", for me.

Explorer

Hi Lucas or Halleyglen ,

Even i m getting the below error, how to fix this .

Error in 'eval' command: The expression is malformed. An unexpected character is reached at '\"1\", \"icon-check\", \"icon-alert\")'.

0 Karma

Motivator

My original search doesn't work anymore on recent versions of splunk as it doesn't like those escape characters in the evals.

You can just use the fields in the base search :

| rest /services/cluster/master/generation/master 
| fields pending_last_reason, search_factor_met, replication_factor_met
0 Karma

Explorer

Hi Lucas , Thanks for the reply , this is working.

However i m getting below errors along with output.

[indexer alias] REST Processor: Failed to fetch REST endpoint uri=https://127.0.0.1:8089/services/cluster/master/generation/master?count=0 from server https://127.0.0.1:8089. Check that the URI path provided exists in the REST API.

Should i worry about this?

0 Karma

Motivator

needs to be performed on the cluster master. Alternatively have dist search to the cluster master from your search heads but you'll have to add splunk_server= to the query.

0 Karma

Explorer

HI Lucas ,

I used below query from search head , not getting any result. Can you help here pls.

| rest /services/cluster/master/generation/master splunk_server=" Clustermaster"
| fields pending_last_reason, search_factor_met, replication_factor_met

0 Karma

Motivator

do you have distsearch to your cluster master? it wouldn't normally be enabled as it's not normally used as a search peer.

0 Karma

Motivator

also remove the white space in your splunk_server. You've quoted whitespace that will make it fail even if you have distsearch to the master configured.

0 Karma

Explorer

Thanks Lucas , Its working fine 🙂

0 Karma

Path Finder

It worked for me

0 Karma

New Member

how did you make it work??

0 Karma
Don’t Miss Global Splunk
User Groups Week!

Free LIVE events worldwide 2/8-2/12
Connect, learn, and collect rad prizes
and swag!