How to configure an alert when the Indexer cluster restart takes more than 1 hour to complete.

[Arpit_S]

We want to create an alert in our instance that triggers only when the indexers takes more than 60 minutes to complete rolling restart process.

We currently have an alert that checks the indexer status in last 15 min and sends the list of indexers that were down in that time period. This causes ambiguity as whenever a rolling restart is done we start getting the alert.

Query:
index=_internal component=CMPeer to=StreamingError host=xxx | stats dc(peer_name) as Count, values(peer_name) as Indexer

We want this to trigger only when an indexers have been in that state since atleast an hour.

Thanks in advance,
Arpit

[nickhills · Accepted solution]

Sorry, I initially misread the question, try this:

...|eventstats earliest(_time) as et|eval moreThanHour=if((et<(now()-3600)), "yes","no") |search moreThanHour="yes"|...

So using your query:

index=_internal component=CMPeer to=StreamingError host=xxx |eventstats earliest(_time) as et|eval moreThanHour=if((et<(now()-3600)), "yes","no") |search moreThanHour="yes"| stats dc(peer_name) as Count, values(peer_name) as Indexer

You will need to make sure your search windows is more than 60 minutes, otherwise the condition will never be met.