Where are login credentials to peers stored? And why bin/scripts/ is deleted after upgrading?

[verbal_666]

Hi.

QUESTION #1: search peer login credentials

In previous versions, i'm talking about v7 and/or v8, in my memory, login credential to search peers were stored into filesystem, inside /var/... in some files named search_peer#1.something, search_peer#2.something etc..

Now i upgraded to 9.4 version, and can't find them anymore. Where are credentials stored? In KVSTORE?

I ask this question since today i need to untar a 9.4 version over a same 9.4 version to take some files a user deleted for error🤦‍♂️and after doing it, it was a searchpeer/indexer, on the SearchHead and Deployer (Management Console) the peer was DOWN, i needed to reinsert the login credential again as first time to make it UP & Healthy again.

Is it for security or is it a bug? I remember i need to do it also months ago when we upgraded 9.3 to 9.4!!! After upgrading, all peer credentials was gone. I thought it was a temporary bug, but now it seems to be the normal behaviour: reinstalling an instance, over the same, removes all previous registered credentials for searchpeers?

QUESTION #2: Splunk upgrading removes all $SPLUNK_HOME/bin/scripts/

And, also, after upgrading, all out custom scripts in $SPLUNK_HOME/bin/scripts/ was totally deleted, SPLUNK UPGRADER remove all of them, i found only the "readme.txt", we had something like 50 custom scripts, all gone 🙄 obviously we have a backup!!! But is it normal???

Thanks.

[verbal_666 · Accepted solution]

Gosh!
I think i did a mistake for Question#1.
The credentials/keys are stored in every searchpeer path dir "$SPLUNK_HOME/etc/auth/distServerKeys/search_peer#x/", inside where you can find all the kets... i did confusion with var 🤧

And the files the user deleted, since i need a full reinstall, were at fact inside $SPLUNK_HOME/etc/ (the user removed for an error ☹️ 🙁 all the etc in $SPLUNK_HOME).

So, there's no mistery at all. He deleted all previous recorded keys the peer stored when called from SearchHead. So it really needed to recreate them, they were gone 😖

Sorry, problem solved!!! My fault... 😑

[livehybrid]

Hi @verbal_666 

Regarding Question 2 - It is expected behaviour that scripts not in the manifest file are deleted during an upgrade.

This is a security enhancement in 9.1.5, 9.2.3, 9.3.5 or 9.4.0 and above. 

Scripts should really be placed within a custom app in $SPLUNK_HOME/etc/apps/<app_name>/bin/

Check out https://splunk.my.site.com/customer/s/article/Shell-scripts-are-missing-after-upgrading-the-HF-to-v9... for more info on this.

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[verbal_666]

Oh, really thanks. I was worried about this behavious. Luckily we have daily backups. Next we create a custom app to deploy them to 👍👍👍i really miss this security feature 👍👍👍