Alerting

How to create and trigger an alert when replication/search factors are not met on the indexer cluster master?

rcreddy06
Path Finder

The status of the replication factor and search factor on indexer cluster are fluctuating and would like to set up an alert. This happens due to bucket fixups pending. Can some one help me creating an alert when the master is fixing buckets and one of the search/replication factor is not met or when the master is trying to fix the buckets.

alt text

1 Solution

Lucas_K
Motivator

The clustering management console from which those three things are obtained has searches that drive them.

It can be shown by doing a "showsource" at the end of your url. ie. /en-US/app/splunk_management_console/indexer_clustering_status?showsource

You can also click the magnifying glass to break out the underlying search in the peers section to see the other searches in that dashboard.

This is the search that drives those 3 icons.

| rest splunk_server_group=dmc_group_cluster_master splunk_server_group=\"$group$\" /services/cluster/master/generation/master       | fields pending_last_reason, search_factor_met, replication_factor_met       | eval all_data_searchable_icon = if(isnull(pending_last_reason) or pending_last_reason==\"\", \"icon-check\", \"icon-alert\")       | eval all_data_searchable = if (isnull(pending_last_reason) or pending_last_reason==\"\", \"All Data is Searchable\", \"Some Data is Not Searchable\")       | eval search_factor_met_icon = if(search_factor_met == 1 or search_factor_met == \"1\", \"icon-check\", \"icon-alert\")       | eval search_factor_met = if (search_factor_met == 1 or search_factor_met == \"1\", \"Search Factor is Met\", \"Search Factor is Not Met\")       | eval replication_factor_met_icon = if(replication_factor_met == 1 or replication_factor_met == \"1\", \"icon-check\", \"icon-alert\")       | eval replication_factor_met = if (replication_factor_met == 1 or replication_factor_met == \"1\", \"Replication Factor is Met\", \"Replication Factor is Not Met\")",

Get your alert to trigger when any of those 3 factors are not equal to 1.

View solution in original post

Lucas_K
Motivator

The clustering management console from which those three things are obtained has searches that drive them.

It can be shown by doing a "showsource" at the end of your url. ie. /en-US/app/splunk_management_console/indexer_clustering_status?showsource

You can also click the magnifying glass to break out the underlying search in the peers section to see the other searches in that dashboard.

This is the search that drives those 3 icons.

| rest splunk_server_group=dmc_group_cluster_master splunk_server_group=\"$group$\" /services/cluster/master/generation/master       | fields pending_last_reason, search_factor_met, replication_factor_met       | eval all_data_searchable_icon = if(isnull(pending_last_reason) or pending_last_reason==\"\", \"icon-check\", \"icon-alert\")       | eval all_data_searchable = if (isnull(pending_last_reason) or pending_last_reason==\"\", \"All Data is Searchable\", \"Some Data is Not Searchable\")       | eval search_factor_met_icon = if(search_factor_met == 1 or search_factor_met == \"1\", \"icon-check\", \"icon-alert\")       | eval search_factor_met = if (search_factor_met == 1 or search_factor_met == \"1\", \"Search Factor is Met\", \"Search Factor is Not Met\")       | eval replication_factor_met_icon = if(replication_factor_met == 1 or replication_factor_met == \"1\", \"icon-check\", \"icon-alert\")       | eval replication_factor_met = if (replication_factor_met == 1 or replication_factor_met == \"1\", \"Replication Factor is Met\", \"Replication Factor is Not Met\")",

Get your alert to trigger when any of those 3 factors are not equal to 1.

halleyglen
Explorer

That shows an unknown group error & eval error on \"\", \"icon-check\", \"icon-alert\", for me.

vasanthi77
Explorer

Hi Lucas or Halleyglen ,

Even i m getting the below error, how to fix this .

Error in 'eval' command: The expression is malformed. An unexpected character is reached at '\"1\", \"icon-check\", \"icon-alert\")'.

0 Karma

Lucas_K
Motivator

My original search doesn't work anymore on recent versions of splunk as it doesn't like those escape characters in the evals.

You can just use the fields in the base search :

| rest /services/cluster/master/generation/master 
| fields pending_last_reason, search_factor_met, replication_factor_met
0 Karma

vasanthi77
Explorer

Hi Lucas , Thanks for the reply , this is working.

However i m getting below errors along with output.

[indexer alias] REST Processor: Failed to fetch REST endpoint uri=https://127.0.0.1:8089/services/cluster/master/generation/master?count=0 from server https://127.0.0.1:8089. Check that the URI path provided exists in the REST API.

Should i worry about this?

0 Karma

Lucas_K
Motivator

needs to be performed on the cluster master. Alternatively have dist search to the cluster master from your search heads but you'll have to add splunk_server= to the query.

0 Karma

vasanthi77
Explorer

HI Lucas ,

I used below query from search head , not getting any result. Can you help here pls.

| rest /services/cluster/master/generation/master splunk_server=" Clustermaster"
| fields pending_last_reason, search_factor_met, replication_factor_met

0 Karma

Lucas_K
Motivator

do you have distsearch to your cluster master? it wouldn't normally be enabled as it's not normally used as a search peer.

0 Karma

Lucas_K
Motivator

also remove the white space in your splunk_server. You've quoted whitespace that will make it fail even if you have distsearch to the master configured.

0 Karma

vasanthi77
Explorer

Thanks Lucas , Its working fine 🙂

0 Karma

rcreddy06
Path Finder

It worked for me

0 Karma

sterlingteks
New Member

how did you make it work??

0 Karma
Get Updates on the Splunk Community!

New in Observability - Improvements to Custom Metrics SLOs, Log Observer Connect & ...

The latest enhancements to the Splunk observability portfolio deliver improved SLO management accuracy, better ...

Improve Data Pipelines Using Splunk Data Management

  Register Now   This Tech Talk will explore the pipeline management offerings Edge Processor and Ingest ...

3-2-1 Go! How Fast Can You Debug Microservices with Observability Cloud?

Register Join this Tech Talk to learn how unique features like Service Centric Views, Tag Spotlight, and ...