topic Re: search head cluster error?? in Monitoring Splunk

search head cluster error??

AzmathShaik — Sat, 08 Apr 2017 03:56:18 GMT

Hello

i am deploy new search head cluster, the deployment went successful but when i check the splunkd.log i fount one strange warning
04-08-2017 03:46:21.294 +0000 WARN ConfReplicationBookmark - serializeToJson: tried to serialize an empty bookmark
04-08-2017 03:46:21.294 +0000 WARN SHCHouseKeepingThread - writeSHCStateToStateStore: failed to serialize the bookmark

i am curious to know about this. what does it exactly means and is this a harmful warning?
do i face issues with this warning?

can some help me ?

Thanks in advance

Re: search head cluster error??

Masa — Sat, 15 Apr 2017 01:14:29 GMT

It indicates this Splunk instance could not understand SHC node status. I suspect you cannot see SHC status (splunk show shcsluter-status) in this node when the message was generated. I also suspect your conf replication is failing.

In that case, you might need to run destructive resync command to resolve the issue. You should make a backup of etc directory before running the command.

You said a new search head cluster. Are those Search head instances new ? If so, do you have three or more search heads? Please make sure you meet requirement to deploy a new SHC.

Re: search head cluster error??

AzmathShaik — Mon, 17 Apr 2017 14:06:17 GMT

some how i figured out the problem. the issue was with the kvstore port (8191). firewall rule was blocking it. the captain was unable to connect to other nodes and write to kvstore