Deployment Architecture

What would trigger Indexer Cluster Master restart?

tkw03
Communicator

So my indexer cluster master seems to be reloading all of the buckets. I didn't initiate a restart of the master or a rolling restart of the cluster. Is there anything else that would trigger either automatically?

Thanks for the input

0 Karma
1 Solution

mdillon_splunk
Splunk Employee
Splunk Employee

Is it possible you had a network interruption from your Cluster Master ?
Has this situation only happened the once ?
Check for disk I/O errors, or NIC errors on your Cluster Master around the time the peers were being added back.

Note: Cluster heartbeats have improved with more recent Splunk versions:

https://conf.splunk.com/files/2017/slides/scaling-indexer-clustering-5-million-unique-buckets-and-be...

View solution in original post

mdillon_splunk
Splunk Employee
Splunk Employee

Is it possible you had a network interruption from your Cluster Master ?
Has this situation only happened the once ?
Check for disk I/O errors, or NIC errors on your Cluster Master around the time the peers were being added back.

Note: Cluster heartbeats have improved with more recent Splunk versions:

https://conf.splunk.com/files/2017/slides/scaling-indexer-clustering-5-million-unique-buckets-and-be...

dkeck
Influencer

HI,

what do you mean by "reloading buckets". What is the master doing that it shouldn´t?

0 Karma

tkw03
Communicator

What I mean is it looks like the master got restarted as it went into making peers check back in and re-meeting rep factor and search factor

0 Karma

dkeck
Influencer

Did you confirm that it wasn´t restarted in the first place?

index=_internal source=splunkd.log "(build"

0 Karma

tkw03
Communicator

No restart, no results

So why would it resync with all peers?

0 Karma
Get Updates on the Splunk Community!

Splunk Forwarders and Forced Time Based Load Balancing

Splunk customers use universal forwarders to collect and send data to Splunk. A universal forwarder can send ...

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...