Alerting

Why am I getting a false DMC Alert that search peer not responding?

ReachDataScient
Explorer

The DMC Alert - search peer not responding has false positives. Anyone addressed this issue with a better modified search.

0 Karma

ReachDataScient
Explorer

Have you made this change and what would you suggest to set the statusTimeout in seconds. Are there any negative effects due to increasing the statusTimeout.

0 Karma

p_gurav
Champion

Can you try increasing the statusTimeout in distsearch.conf on the DMC will give the searchPeers more slack as the DMC tries to get each Peers info, which in turn will result in less peers showing up as "Down" in /services/search/distributed/peers/.

statusTimeout = <int, in seconds>
 * Set connection timeout when gathering a search peer's basic info (/services/server/info).
 * Note: Read/write timeouts are automatically set to twice this value.
 * Defaults to 10.

anilyelmar
Explorer

You can do this from Setting >>Distributed search >>Distributed search>>Timeout settings and changing the Status timeout (in seconds) from default value 10 to something larger considering your environment.

0 Karma
Get Updates on the Splunk Community!

Routing Data to Different Splunk Indexes in the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...

Getting Started with AIOps: Event Correlation Basics and Alert Storm Detection in ...

Getting Started with AIOps:Event Correlation Basics and Alert Storm Detection in Splunk IT Service ...

Register to Attend BSides SPL 2022 - It's all Happening October 18!

Join like-minded individuals for technical sessions on everything Splunk!  This is a community-led and run ...