Splunk Enterprise Security

Unwanted self-deleted Peer

New Member

Lately i've been having many problems with my peers disponibility. Many times it stops working and cause me issues. I figured that if i could make a script up my problems would end. I use the splunk atom feed to this, follow the script:


ignore invalid SSL Certs - Do Not Change
try {
add-type @"
using System.Net;
using System.Security.Cryptography.X509Certificates;
public class TrustAllCertsPolicy : ICertificatePolicy {
public bool CheckValidationResult(
ServicePoint srvPoint, X509Certificate certificate,
WebRequest request, int certificateProblem) {
return true;
catch { }
[System.Net.ServicePointManager]::CertificatePolicy = New-Object TrustAllCertsPolicy

function Set-PeerStats{
Invoke-WebRequest -Method POST -Headers @{"Authorization"="Basic xxxxxxxxxbase64xxxxxxxxxx="} -Uri "hxxps://$Peer%3A8089/disable" >$null
Start-Sleep 5;
Invoke-WebRequest -Method POST -Headers @{"Authorization"="Basic xxxxxxxxxxxxxbase64xxxxxxxxxxxx="} -Uri "hxxps://$Peer%3A8089/enable" >$null

If my peer goes down several times, like 20 times in a row, the peer automatically desappears as if it was deleted, but genereates no logs from it, i also can ensure that isn't quarentined.

Does anyone have any clue about what could be happening?

0 Karma


Could you tell us what you're actually trying to do? So, what was the problem causing you to switch over to a script doing some REST calls?

Nothing in the splunkd.logs?


0 Karma

New Member

I'm facing an everyday problem with my customers peers. Every single day the peer goes down about 20 times. We use this ditribute search peer to get data from our costumers splunk. Every time that the peer goes down, we disable and enable the peer and it begins to work again.

I looked up the logs to see what i could find, but nothing really usefull.


Audit log if we delete the peer manually:

index=_audit* user="vandrade" action=edit_dist_peer operation=remove


12:50:29.462 PM
Audit:[timestamp=07-22-2019 12:50:29.462, user=vandrade, action=edit_dist_peer, info=granted object="" operation=remove][n/a]
source = audittrail sourcetype = audittrail user = vandrade


Search Audit log for the unwanted auto-deleted peer:

index=_audit* user="service_prtg" action=edit_dist_peer


(They're all the same)

Audit:[timestamp=07-22-2019 17:54:27.679, user=service_prtg, action=edit_dist_peer, info=granted object="" operation=list][n/a]
action = edit_dist_peer source = audittrail sourcetype = audittrail user = service_prtg


If i use the operation ="remove" at the second query i'll get no result, altought, if the script run several times, the peer get deleted and generate no logs of it.

0 Karma


We need actual data of the peer that had problems.

There are so many questions about this problem I'm not sure we can help you here. You need to find out why the indexer is going down. Is it overloaded, crashing due to running out of memory or anythinf else happening?

Look for any errors around the time in splunkd.log.


0 Karma
Get Updates on the Splunk Community!

Using Machine Learning for Hunting Security Threats

WATCH NOW Seeing the exponential hike in global cyber threat spectrum, organizations are now striving more for ...

Observability Newsletter Highlights | March 2023

 March 2023 | Check out the latest and greatestSplunk APM's New Tag Filter ExperienceSplunk APM has updated ...

Security Newsletter Updates | March 2023

 March 2023 | Check out the latest and greatestUnify Your Security Operations with Splunk Mission Control The ...