Hi everyone,
i've currently deployed the following instances in my Splunk infrastructure using Splunk 8.1.0:
- 1 Search Head
- 1 Cluster Master
- 2 Indexers in cluster
- 2 Heavy Forwarders
Everything seems to work fine except for Cluster Master. Since i added the 2 Indexers to the cluster, the following messages are repeated in splunkd.log on Cluster Master system:
10-28-2020 16:59:54.703 +0100 WARN Fixup - GenCommitFixup::finish error in scheduler sendQueued= 10-28-2020 16:59:54.704 +0100 WARN CMMaster - Unable to send scheduled jobs, err="" 10-28-2020 16:59:55.202 +0100 DEBUG CMMaster - event=serviceHeartbeats size=1 10-28-2020 16:59:55.202 +0100 DEBUG CMMaster - event=setPeerStatus Skipping since peer=AAAAA peer_name=splunk-indexer-2 is already in status=Up reason=heartbeat received. 10-28-2020 16:59:55.202 +0100 DEBUG CMMaster - event=serviceRecreateIndexJobs No indexes to be recreated 10-28-2020 16:59:55.203 +0100 DEBUG CMMaster - event=serviceRecreateBucketJobs No buckets to be recreated 10-28-2020 16:59:55.203 +0100 WARN Fixup - GenCommitFixup::finish error in scheduler sendQueued= 10-28-2020 16:59:55.203 +0100 WARN CMMaster - Unable to send scheduled jobs, err="" 10-28-2020 16:59:55.703 +0100 DEBUG CMMaster - event=serviceHeartbeats size=1 10-28-2020 16:59:55.704 +0100 DEBUG CMMaster - event=setPeerStatus Skipping since peer=BBBBB peer_name=splunk-indexer-1 is already in status=Up reason=heartbeat received. 10-28-2020 16:59:55.704 +0100 DEBUG CMMaster - event=serviceRecreateIndexJobs No indexes to be recreated 10-28-2020 16:59:55.704 +0100 DEBUG CMMaster - event=serviceRecreateBucketJobs No buckets to be recreated
Can you help me with this issue?
Other useful information that may help:
NTP and Splunk License are not yet installed.
No errors are shown in splunkd.log on the remaining components.
No errors are shown on Cluster Master web GUI.
Indexer discovery is enable.
Internal log are forwarded to Indexers.
btool check utility didn't find any error.
Each Indexer has indexes.conf with repFactor = auto under [main]
Monitor console is enabled on Cluster Master
Cluster Master server.log
[general] serverName = splunk-clustermaster-1 pass4SymmKey = pass [sslConfig] sslPassword = pass enableSplunkdSSL = true serverCert = /opt/splunk/etc/auth/server.pem sslRootCAPath = /opt/splunk/etc/auth/cacert.pem [lmpool:auto_generated_pool_download-trial] description = auto_generated_pool_download-trial quota = MAX slaves = * stack_id = download-trial [lmpool:auto_generated_pool_forwarder] description = auto_generated_pool_forwarder quota = MAX slaves = * stack_id = forwarder [lmpool:auto_generated_pool_free] description = auto_generated_pool_free quota = MAX slaves = * stack_id = free [clustering] cluster_label = cluster mode = master pass4SymmKey = pass replication_factor = 2 [indexer_discovery] pass4SymmKey = pass
Indexers server.conf
[general] serverName = splunk-indexer-1 pass4SymmKey = pass parallelIngestionPipelines = 2 pipelineSetSelectionPolicy = weighted_random [sslConfig] enableSplunkdSSL = true sslPassword = pass sslRootCAPath = /opt/splunk/etc/auth/cacert.pem serverCert = /opt/splunk/etc/auth/server.pem [lmpool:auto_generated_pool_download-trial] description = auto_generated_pool_download-trial quota = MAX slaves = * stack_id = download-trial [lmpool:auto_generated_pool_forwarder] description = auto_generated_pool_forwarder quota = MAX slaves = * stack_id = forwarder [lmpool:auto_generated_pool_free] description = auto_generated_pool_free quota = MAX slaves = * stack_id = free [replication_port-ssl://8080] serverCert = /opt/splunk/etc/auth/server.pem sslPassword = password [clustering] master_uri = https://clustermaster:8089 mode = slave pass4SymmKey = pass
Thank you!
... View more