topic Re: Where are login credentials to peers stored? And why bin/scripts/ is deleted after upgrading? in Splunk Enterprise

Where are login credentials to peers stored? And why bin/scripts/ is deleted after upgrading?

verbal_666 — Fri, 05 Dec 2025 14:33:41 GMT

Hi.

QUESTION #1: search peer login credentials

In previous versions, i'm talking about v7 and/or v8, in my memory, login credential to search peers were stored into filesystem, inside /var/... in some files named search_peer#1.something, search_peer#2.something etc..

Now i upgraded to 9.4 version, and can't find them anymore. Where are credentials stored? In KVSTORE?

I ask this question since today i need to untar a 9.4 version over a same 9.4 version to take some files a user deleted for error🤦‍♂️and after doing it, it was a searchpeer/indexer, on the SearchHead and Deployer (Management Console) the peer was DOWN, i needed to reinsert the login credential again as first time to make it UP & Healthy again.

Is it for security or is it a bug? I remember i need to do it also months ago when we upgraded 9.3 to 9.4!!! After upgrading, all peer credentials was gone. I thought it was a temporary bug, but now it seems to be the normal behaviour: reinstalling an instance, over the same, removes all previous registered credentials for searchpeers?

QUESTION #2: Splunk upgrading removes all $SPLUNK_HOME/bin/scripts/

And, also, after upgrading, all out custom scripts in $SPLUNK_HOME/bin/scripts/ was totally deleted, SPLUNK UPGRADER remove all of them, i found only the "readme.txt", we had something like 50 custom scripts, all gone 🙄 obviously we have a backup!!! But is it normal???

Thanks.

Re: Where are login credentials to peers stored? And why bin/scripts/ is deleted after upgrading?

livehybrid — Fri, 05 Dec 2025 13:08:44 GMT

Hi @verbal_666

Regarding Question 2 - It is expected behaviour that scripts not in the manifest file are deleted during an upgrade.

This is a security enhancement in 9.1.5, 9.2.3, 9.3.5 or 9.4.0 and above.

Scripts should really be placed within a custom app in $SPLUNK_HOME/etc/apps/<app_name>/bin/

Check out https://splunk.my.site.com/customer/s/article/Shell-scripts-are-missing-after-upgrading-the-HF-to-v9-2-3 for more info on this.

🌟 Did this answer help you? If so, please consider:

Adding karma to show it was useful
Marking it as the solution if it resolved your issue
Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

Re: Where are login credentials to peers stored? And why bin/scripts/ is deleted after upgrading?

verbal_666 — Fri, 05 Dec 2025 13:41:59 GMT

Oh, really thanks. I was worried about this behavious. Luckily we have daily backups. Next we create a custom app to deploy them to 👍👍👍i really miss this security feature 👍👍👍

Re: Where are login credentials to peers stored? And why bin/scripts/ is deleted after upgrading?

verbal_666 — Fri, 05 Dec 2025 14:30:41 GMT

Gosh!
I think i did a mistake for Question#1.
The credentials/keys are stored in every searchpeer path dir "$SPLUNK_HOME/etc/auth/distServerKeys/search_peer#x/", inside where you can find all the kets... i did confusion with var 🤧

And the files the user deleted, since i need a full reinstall, were at fact inside $SPLUNK_HOME/etc/ (the user removed for an error ☹️ 🙁 all the etc in $SPLUNK_HOME).

So, there's no mistery at all. He deleted all previous recorded keys the peer stored when called from SearchHead. So it really needed to recreate them, they were gone 😖

Sorry, problem solved!!! My fault... 😑