Best Practice For Managing Alerts/Reports/Dashboards/etc. on a Clustered Search

[dstuder]

To manage our knowledge objects on our search heads I created a blank app on the deployer under /opt/splunk/etc/shcluster/apps/ and deployed it to the search heads. That part worked well. I can now create knowledge objects in that app and they sync across the search heads. That works well too. However, I am realizing that the app on the deployer is empty and I'm wondering if that could come back to bite me some day somehow. Should I instead be creating the knowledge objects in an app on the deployer in /opt/splunk/etc/apps/, and then copy the changes to /opt/splunk/etc/shcluster/apps/, and finally deploy it to the search heads? That way I can still use the UI to create the knowledge obejects but won't have to worry about anything or is my worry overrated? I'm curious what is the best way to manage this that others have found. Should I just not worry about it and let the apps be different between the deployer and the search heads or should I have some process for configuring it on the deployer, copying it to the deployment apps, and then pushing out? Thanks.

[KulvinderSingh]

I will suggest to develop  the app in dev environment and then move it to /opt/splunk/etc/shcluster/apps/ once its all done/complete. do not install it deployer. Normally cluster captain will be responsible for replicating changes to apps and config on cluster. It will replicate the changes if its made on one of the nodes to other nodes but this way you will lose control on changes made to app and just in case of  an issue you will not have the actual config on the nodes. so best is to make changes to deployer and push them to cluster nodes. hope that helps.

[dstuder]

How would you account for end users wanting to create saved reports/alerts/etc that are maybe not so developer minded, but can craft basic searches?

[isoutamo]

Hi

as usually that depends. Your current way is working as well if you are creating those first in deployer and then push those to SHC. Also one way is create those first in development sh and then using e.g. git and CI/CD to deploy those there. 
It’s hard to say which one is best methods in your environment. Do you need strict change management or is the flexibility must in your case.

r. Ismo

[dstuder]

I do have users that will want to created saved searches/alerts/etc. ... so maybe I do need to just be more flexible and not so rigid with the development process.

[KulvinderSingh]

@dstuder--

For this Knowledge objects have to be shared globally and not with in the app.  As you mentioned users are not splunk admins. There are couple of approaches here:

1. You can let users play with searches and dashboards and then if some searches or dashboards. Use the search below to check for the new searches and dashboards and update deployer monthly or weekly

"| rest /servicesNS/admin/-/saved/searches | table author title eai:acl.app | eval Type="SavedSearch/Report"
| append
[| rest /servicesNS/admin/-/data/ui/views | table author title eai:acl.app | eval Type="Dashboard"]
| rename author as Owner title as Name eai:.acl.app as AppName"

                     OR

Let them create their own dashboards and searches in search app. Which is default for all users and what will happen is when they create a dashboard it will have permissions as private and they will be the only ones that can see that dashbaord or search. So unless the dashboard is needed for a team or all the users let them create the dashboards in "search and reporting". Still you will have to make Knowledge objects like macros etc. to be with global permissions. 

hope that helps.