Accidently deleted the shcluster/apps directories on my Search Head Cluster Deployer

[mgiddens]

Accidently deleted the entire shcluster/apps directories on my Search Head Cluster Deployer. Is there a way to get the directory back? I thought about pull over the directory from the search head cluster member since it has the same directory with nothing in it? Anyone know of a solution?

Mike

[mgiddens · Accepted solution]

Thanks to everyone for the assist on this! All have been very helpful and some good nuggets of info to take away. The answer was to simply recreate directories then assign the "splunk" as the ower/group for those directories. I relatively new to Linux and Splunk so this was good experience having to fix this.

Thanks again to everyone!

Regards

Mike

[codebuilder]

Assuming you have not pushed out a new version since deleting the directory on the deployer, just copy the contents from the app directory back to the deployer.

From the search head:
scp -RP $SPLUNK_HOME/etc/apps/ deployer_name:/$SPLUNK_HOME/etc/shcluster/apps/

After copying over, you probably want to check the contents of the local directory. It can/will contain local changes made by users on the search heads. If you deploy again with that directory intact, it will merge those into the default directory of you app.

[ddrillic]

Doing that is great, but we might have issues with dynamic lookups for example. They shouldn't be held back at the deployer, because then these static versions will overwrite the user modified ones.

Part of my issue back then is described at Can an admin delete any lookup, owned by anybody?

My understanding there was -

Apparently, the ones from the deployer are immutable by design, as under the app directory, we only have the lookups directory and unlike other knowledge objects there are not under the local and default directories, which make them behave differently.

[mgiddens]

Ok,, so when I scp this back over to the deployer from the search head with the command you put, that will re-create the "shcluster/apps" folder in the deployer? Right now I only have the path up to /opt/splunk/etc on the deployer; "shcluster/apps" was deleted completely.

[pruthvikrishnap]

Hi,
I am sorry to hear this, few things here, for changes to reflect from deployer you will have to push the changes using a command, check if you already have pushed these changes, else you should see the apps on search head and can copy them back from search head nodes to deployer. Login to node and see if there are any apps.
If you already have pushed these changes then all the content in search head nodes will also vanish, the only option i see is to check if there are any backups created for this deployer. Generally we create a backup in any production environment.

command to push changes from deployer should be something like this "splunk apply shcluster-bundle"

[mgiddens]

Awesome, thank you for the response! I actually only have up to the /etc directory (opt/splunk/etc),, after I "cd" here I do an "ll" and see no "shcluster" directory there. I am trying to figure out the best way to get that folder "shcluster" back in the "etc' directory,, then get "apps" and "users" folder underneath that. Any ideas?

[codebuilder]

Just recreate the directories:

mkdir -p /opt/splunk/etc/shcluster/apps
mkdir -p /opt/splunk/etc/shcluster/users

Then SCP your app/users from the search head to the deployer:

scp -RP $SPLUNK_HOME/etc/apps/app_name deployer_name:/$SPLUNK_HOME/etc/shcluster/apps/
scp -RP $SPLUNK_HOME/etc/apps/users deployer_name:/$SPLUNK_HOME/etc/shcluster/users/

[mgiddens]

thanks for the input, seems like when I re-create them and try to push from this location, it gives me an error. States cannot push from directory with "0" apps /opt/splunk/etc/shcluster

I have 2 apps in the re-created directories

Any thoughts why or how to correct?

[codebuilder]

Try this:

cp -RP $SPLUNK_HOME/etc/apps/* deployer_name:/$SPLUNK_HOME/etc/shcluster/apps/

[codebuilder]

Oh, so you were able to get them uploaded back to the deployer, I think I misread.
If you SCP'd them up as root you'll need to change ownerships back to splunk on those directories on the deployer.

chown -RP splunk:splunk /opt/splunk/etc/shcluster/apps
chown -RP splunk:splunk /opt/splunk/etc/shcluster/users

Splunk won't recognize any files or directories not owned by the splunk user.
Once you make the changes it should allow you to deploy again.
Don't forget to examine/remove any local files that you might have transferred up to the deployer.

[mgiddens]

you were right on the money with the ownership of those directories. The lightbulb clicked last night after I got home; since I created with root, it assigned root as owner/group. Change this to splunk this morning right as I got in and bam it pushed the bundle with not issues. Thanks for all the help!

Regards

Mike

[codebuilder]

Always happy to help, but seems odd that you accepted your own answer after all that.

[mgiddens]

Yeah I wasn't sure which I should post as the final answer since collectively there many that offered assistance. So I figured I would just summarize the ultimate fix in a post and mark as answered; thought it might make it easier for anyone who experiences this in the future to find what exactly fixed my issue of all the inputs received. Can you accept multiple post as the answer? I didn't try.