Deploy App to SearchHeads so they are Available in role creation

[LegalPrime]

I have 3 search heads, 1 search head deployer.

I need to create "a new app"  for which I will create a role that will be able to query few indices that are assigned to it in resources tab.

I need to create roles on SearchHead, because I have SAML auth setup there.

When I create this new app on one search head (via UI -> Manage Apps -> Create App), I do not see it available on other search heads, or in the list of resources when I try to create the role.

When I create this new app on the Search Head Deployer, I cannot see it on SHs when creating role.

I cannot make `splunk shbundle-deploy` (or whatever the command actually is), when I try, it fails, because splunk tries to create something in `/dev/null/.splunk` (it runs under system account which does not have shell access; so I guess that's related to that and it cannot change).

Where do I create the app and how do I push it to search heads, so I can see it there?

Thanks for your time.

[scelikok]

Hi @LegalPrime,

In fact deployer just creates a tar bundle using all files under shcluster/apps folder. These apps should go to $SPLUNK_HOME/etc/apps directory on search heads.

Did you restart the Deployer after copying the new app to shcluster/apps folder?

[richgalloway]

---

I cannot make `splunk shbundle-deploy` (or whatever the command actually is), when I try, it fails, because splunk tries to create something in `/dev/null/.splunk` (it runs under system account which does not have shell access; so I guess that's related to that and it cannot change).

This is the key part of this question.  The proper way to deploy apps from a deployer to the search heads is by using the splunk apply shcluster-bundle command so it's important to get that working.

Let me ask a few questions to clarify the picture of your environment. 

1. Are the 3 search heads in a cluster? 
2. Is the deployer NOT one of the search heads? 
3. Is the new app is in the $SPLUNK_HOME/etc/deployment-apps directory on the deployer? 
4. Are you executing the command on the deployer? 
5. What user is Splunk running as?
6. You say the user cannot be given shell access, but can it be given a home directory?  If so, that may solve the problem.

[LegalPrime]

Hey, thanks for reply.

1. Yes, the search heads are in the same cluster

2. Deployer is a standalone EC2 instance, it is not on one of the SH instances

3. It is not. When I create the app via UI, it only appears in `/opt/splunk/etc/apps` as a new directory.

4. I was executing it on the deployer.

5. It runs under `splunkusr` which is a user that was created to run it. I can sudo as this user (but it doesnt change anything in retards to /dev/null/.splunk creation attempt.

6. I tried that now with the help of `mkhomedir_helper`, but it does not do anything. Based on my documentation read through, I would also need to force logout on the user that is actually running the splunk deployer (to put the home folder into effect). Can I provide temporary folder for storage through parameters/environment before running the splunk executable somehow?

[richgalloway]

Deployed apps MUST be in $SPLUNK_HOME/etc/shcluster/apps.  They will not deploy from other locations.

The home directory of user splunkusr must be set on each search head.  You'll likely need to restart Splunk on each SH for the change to be picked up by splunkd. 

[scelikok]

@richgalloway,

I wanted to warn you about the app path on the deployer, it is $SPLUNK_HOME/etc/shcluster/apps.

[richgalloway]

Thanks, @scelikok !  I conflated the deployer and the deployment server.  I'll fix my answer to avoid confusion.

[LegalPrime]

Hello, so I figured how to run splunk binary with home directory that can be written into by the user I run the command as. TL;DR, I am leveraging sudo's option to provide env variables on input and change them to the directory I created under splunk's service account.

But it is not behaving as it should (based on what you say and what documentation say).

I create app via UI (is created in /opt/splunk/etc/apps), I copy it from there to /opt/splunk/etc/shcluster/apps and run the

splunk apply shcluster-bundle 

 (+ target and answer-yes params). This did not help and I cannot see the apps on search heads. Not in the UI and not in their app directories (or shcluster/apps directories).

What now? How can I debug this further and figure where the issue lies?

My assumption is that Apps are not considered to be a configuration bundle and that's why they don't get pushed to search heads.

Tagging both of @richgalloway, @scelikok.

Additional info: I have checked that all search heads are in the same cluster and that captain is elected (it's a dynamic captain, if that matters).

[LegalPrime]

Quick followup question here: What happens if I create the app manually on each of the search heads via UI? Will there be issues in the future?

I realized that newly created role is automatically replicated between SearchHeads. I created one on SH where I already created app in UI, and then created the app on the other search head as well. The replicated role immediatelly picked up the Default app setting, once the app was created there.

Maybe this is a one-time solution for me before I figure out how to rebuild the infrastructure better?

[richgalloway]

You can install apps manually on each SH without harm.  The DS automates that and helps ensure all SHs are the same.

[isoutamo]

When you are logging in to deployer you probably logging as your own account. Please try then next: sudo -u splunk bash. This should give you a shell access even that splunk-user haven't home and login rights. Then you should do /opt/splunk/bin/splunk apply shcluster-bundle ....

r. Ismo

[LegalPrime]

Hello, I have already tried this before. It says the same thing about the /dev/null/.splunk not being a directory.