Archive
Highlighted

How to configure an alert when the Indexer cluster restart takes more than 1 hour to complete.

Path Finder

We want to create an alert in our instance that triggers only when the indexers takes more than 60 minutes to complete rolling restart process.

We currently have an alert that checks the indexer status in last 15 min and sends the list of indexers that were down in that time period. This causes ambiguity as whenever a rolling restart is done we start getting the alert.

Query:
index=internal component=CMPeer to=StreamingError host=xxx | stats dc(peername) as Count, values(peer_name) as Indexer

We want this to trigger only when an indexers have been in that state since atleast an hour.

Thanks in advance,
Arpit

Tags (1)
0 Karma
Highlighted

Re: How to configure an alert when the Indexer cluster restart takes more than 1 hour to complete.

Ultra Champion

Sorry, I initially misread the question, try this:

...|eventstats earliest(_time) as et|eval moreThanHour=if((et<(now()-3600)), "yes","no") |search moreThanHour="yes"|...

So using your query:

index=_internal component=CMPeer to=StreamingError host=xxx |eventstats earliest(_time) as et|eval moreThanHour=if((et<(now()-3600)), "yes","no") |search moreThanHour="yes"| stats dc(peer_name) as Count, values(peer_name) as Indexer

You will need to make sure your search windows is more than 60 minutes, otherwise the condition will never be met.

View solution in original post