Splunk Search
Highlighted

Search peer and search process errors

Path Finder

hello,

Our physical servers had to restart and as such the splunk servers dropped.

we are now having issues on our cluster master and our indexers.

our deployment looks like this,

DCAXXXG013 CM and LM
DCAXXXG014 IDX
DCAXXXG015 IDX
DCAXXXG016 IDX
DCAXXXG017 SH

DCPXXXG013 DS
DCPXXXG014 IDX
DCPXXXG015 IDX
DCPXXXG016 IDX
DCPXXXG017 SH

The indexers on site A and Site P are both clustered. just wondering if anyone can shed some light on where to go and how to progress from here if possible.

Search peer DCAOVSG016 has the following message: Failed to register with cluster master reason: failed method=POST path=/services/cluster/master/peers/?outputmode=json master=dcaovsg013:8089 rv=0 gotConnectionError=0 gotUnexpectedStatusCode=1 actualresponsecode=500 expectedresponsecode=2xx statusline="Internal Server Error" socketerror="No error" remoteerror=Cannot add peer=172.26.10.49 mgmtport=8089 (reason: non-zero pending job count=1, guid=ADA4AE8A-B93F-48E2-88CC-F47CDDCB9AE4). [ event=addPeer status=retrying AddPeerRequest: { id= activebundleid=EDA5C78B2096F563800873D7CBD2A6DF addtype=ReAdd-As-Is basegenerationid=2073 batchserialno=1 batchsize=3 forwarderdatarcvport=9197 forwarderdatausessl=0 lastcompletegenerationid=2077 latestbundleid=EDA5C78B2096F563800873D7CBD2A6DF mgmtport=8089 name=ADA4AE8A-B93F-48E2-88CC-F47CDDCB9AE4 registerforwarderaddress= registerreplicationaddress= registersearchaddress= replicationport=9100 replicationusessl=0 replications= servername=DCAOVSG016 site=site1 splunkversion=7.2.0 splunkdbuild_number=8c86330ac18 status=Up } ].

Indexer Clustering: The search process with sid=rtscheduleradminQkNOX1RBX1dpbmRvd3MtU2VydmVycw_RMD5d0958093cdddf4f3at15512701201818 on peer=DCAOVSG014 may have returned partial results due to a reading error while waiting for the peer. This can occur if the peer unexpectedly closes or resets the connection during a planned restart. Try running the search again. Learn more.
2/27/2019, 12:22:34 PM

Search peer DCAOVSG014 has the following message: Failed to register with cluster master reason: failed method=POST path=/services/cluster/master/peers/?outputmode=json master=dcaovsg013:8089 rv=0 gotConnectionError=0 gotUnexpectedStatusCode=1 actualresponsecode=500 expectedresponsecode=2xx statusline="Internal Server Error" socketerror="No error" remoteerror=Cannot add peer=172.26.10.47 mgmtport=8089 (reason: non-zero pending job count=2, guid=3724715E-6BAC-46F9-AFE7-06917EF3FD3C). [ event=addPeer status=retrying AddPeerRequest: { id= activebundleid=EDA5C78B2096F563800873D7CBD2A6DF addtype=ReAdd-As-Is basegenerationid=2086 batchserialno=1 batchsize=2 forwarderdatarcvport=9197 forwarderdatausessl=0 lastcompletegenerationid=2093 latestbundleid=EDA5C78B2096F563800873D7CBD2A6DF mgmtport=8089 name=3724715E-6BAC-46F9-AFE7-06917EF3FD3C registerforwarderaddress= registerreplicationaddress= registersearchaddress= replicationport=9100 replicationusessl=0 replications= servername=DCAOVSG014 site=site1 splunkversion=7.2.0 splunkdbuild_number=8c86330ac18 status=Up } ].

Any help is greatly appreciated.

Cheers

0 Karma
Highlighted

Re: Search peer and search process errors

You'll want to check the logs on dcaovsg013 because it's returning 500 errors ( actual_response_code=500 ) because of reason: non-zero pending job - there's probably some outstanding issue or load on that machine.

0 Karma