Monitoring Splunk

After Deployment Server upgrade Manager Node is in sick state as Search Peer

lukasmecir
Path Finder

Hello,

I have distributed environment with IDX cluster and DS. DS is used for deploy config to IDX cluster Manager Node and from it to IDX cluster nodes then. It is working fine.

I upgraded DS from 8.1.6 to 8.1.10.1 (yes, because SVD-2022-0608...). Manager Node is on 8.1.6. After upgrade I noticed this log messages on MN:

10.88.28.93 - - [13/Jul/2022:15:56:33.540 +0200] "GET /services/server/info HTTP/1.1" 401 130 "-" "Splunk/8.1.10.1 (Linux 3.10.0-1160.62.1.el7.x86_64; arch=x86_64)" - 0ms

 10.88.28.93 is IP address of DS

I checked Search peers config on DS and there was MN in "sick" state. I edited its config by re-enter Remote username and Remote password and then MN changed status to Healthy and everything is working fine.

My question is: what happened during upgrade of DS? My idea is that new pair of private+public keys was generated on DS on first run after upgrade (and then I had to distribute new public key to MN by re-entering Remote username and password of course), but am I right? And if I am right, why this happened? I made many Splunk upgrades before and I experienced this never before...

Any info/hint/clue will be highly appreciated. Thank you.

Best regards

Lukas Mecir

Labels (1)
0 Karma
Get Updates on the Splunk Community!

Updated Team Landing Page in Splunk Observability

We’re making some changes to the team landing page in Splunk Observability, based on your feedback. The ...

New! Splunk Observability Search Enhancements for Splunk APM Services/Traces and ...

Regardless of where you are in Splunk Observability, you can search for relevant APM targets including service ...

Webinar Recap | Revolutionizing IT Operations: The Transformative Power of AI and ML ...

The Transformative Power of AI and ML in Enhancing Observability   In the realm of IT operations, the ...