We have new SH node which we are trying to add to the Search head cluster,  updated the configs in shcluster config and other configs.  After adding this node in the cluster , now we have two nodes as pert of  the SH cluster. We can see both the nodes up and running part of the cluster,   when we check it with "splunk show shcluster-status". But, when we check the kvstore status with " splunk show kvstore-status" old nodes shows as captain , but the newly built node is not joining this cluster and giving the below error in the logs. Error in Splunkd log on the search head which has issue.. 12-04-2024 16:36:45.402 +0000 ERROR KVStoreBulletinBoardManager [534432 KVStoreConfigurationThread] - Local KV Store has replication issues. See introspection data and mongod.log for details. Cluster has not been configured on this member. KVStore cluster has not been configured We have configured all the cluster related info on the newly built search head server(server.conf), dont see any configs missing. We also see below error on the SH ui page messages tab.. Failed to synchronize configuration with KVStore cluster. Quorum check failed because not enough voting nodes responded; required 2 but only the following 1 voting nodes responded: search-head01:8191; the following nodes did not respond affirmatively: search-head01:8191 failed with Error connecting to search-head01:8191 (172.**.***.**:8191) :: caused by :: compression disabled. Anyone else faced this error before...need some support here... ... View more

I have a strange issue, when i search for specific event in Splunk and I am looking for specific fields( ex field1, field2) i can not see them in selected fields and interested fields. But, when i run the same query and table those fields , i can see them.  index=1234 sourcetype=4567 --> can not see those fields when this search is triggered. But when i add a table command with those field names in the search i can see the fields and the values for them.   index=1234 sourcetype=4567 | table field1, field2 --> this query i can see the fields.   Did any one face this issue ? We are on latest Splunk version 9.0.X..     ... View more

Hello Splunk team... I am facing this issue while we run any searches on my splunk setup., can you help me on how we can fix this.. 02-29-2024 06:58:53.370 ERROR DispatchThread [4125 phase_1] - code=10 error="" 02-29-2024 06:58:53.370 ERROR ResultsCollationProcessor [4125 phase_1] - SearchMessage orig_component=ResultsCollationProcessor sid=1709189933.399443_**** message_key=DISPATCHCOMM:PEER_PIPE_EXCEPTION__%s message=Search results might be incomplete: the search process on the peer:  ended prematurely. Check the peer log, such as $SPLUNK_HOME/var/log/splunk/splunkd.log and as well as the search.log for the particular search.   Thank you.. ... View more