Monitoring Splunk

search head cluster error??

AzmathShaik
Path Finder

Hello

i am deploy new search head cluster, the deployment went successful but when i check the splunkd.log i fount one strange warning
04-08-2017 03:46:21.294 +0000 WARN ConfReplicationBookmark - serializeToJson: tried to serialize an empty bookmark
04-08-2017 03:46:21.294 +0000 WARN SHCHouseKeepingThread - writeSHCStateToStateStore: failed to serialize the bookmark

i am curious to know about this. what does it exactly means and is this a harmful warning?
do i face issues with this warning?

can some help me ?

Thanks in advance

0 Karma
1 Solution

AzmathShaik
Path Finder

some how i figured out the problem. the issue was with the kvstore port (8191). firewall rule was blocking it. the captain was unable to connect to other nodes and write to kvstore

View solution in original post

AzmathShaik
Path Finder

some how i figured out the problem. the issue was with the kvstore port (8191). firewall rule was blocking it. the captain was unable to connect to other nodes and write to kvstore

Masa
Splunk Employee
Splunk Employee

It indicates this Splunk instance could not understand SHC node status. I suspect you cannot see SHC status (splunk show shcsluter-status) in this node when the message was generated. I also suspect your conf replication is failing.

In that case, you might need to run destructive resync command to resolve the issue. You should make a backup of etc directory before running the command.

You said a new search head cluster. Are those Search head instances new ? If so, do you have three or more search heads? Please make sure you meet requirement to deploy a new SHC.

Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Get Agentic with Splunk Lantern: Connect to Cisco Cloud Control, Transform ...

Splunk Lantern is Splunk’s customer success center that provides practical guidance from Splunk experts on key ...

July Community Events: Master ITSI 5.0 & Automate Splunk

Struggling with alert fatigue or feeling like you're spending more time on infrastructure maintenance than ...

New Release of Federated Search: Bringing Splunk Analytics to More of Your Data

Organizations today are generating more data than ever and storing it across cloud object stores, data lakes, ...