What's the best practice for removing configurations for defining existing indexes on a new search head?

[soniquella]

Good morning.

I hope someone can advise as to the best practice solution for the below issue:

I had previously been using a single system with dual role of indexer and search head. Recently, I created a new SH and copied all apps across to the new SH without issue.
I still have roles on the indexer which I am assuming can now be removed (aside from local admin access)?

There are some indexes on the indexer which do not seem to be defined on the new search head. i.e. the indexes can be searched when specified but if I am settings the default index for a new role then the indexes do not seem to be available, only internal and summary and a new index which has been created on the search head.

What is the best practice to follow here as far as removing any configurations relating to the SH ability on the now sole indexer and also for defining the existing indexes on the new search head?

Any help or advice would be greatly appreciated.

Kind regards,

Rob

[bcyates · Accepted solution]

What version of Splunk are you on? There's a known issue with version 7.0 that sounds like it may apply to you.

SPL-145546 - in 7.x in Roles admin Indexes are for local search head only

Workaround:

Step 1) Create a local directory in the search app on the SH with the correct permissions for splunkd to access i.e.

$SPLUNK_HOME/etc/apps/search/local/data/ui/manager

Step 2) Copy an old "authentication_roles.xml" file from "$SPLUNK_HOME/etc/apps/search/default/data/ui/manager" in any 6.x version or simply download a new 6.x version of Splunk and extract the file there, then place it into the folder created in step 1.

Step 3) Refresh the SH configuration with debug refresh via the web browser:

http://:8000/en-US/debug/refresh

Step 4) Create a new role on the SH and you should see all your indexes configured on the index cluster.

[micahkemp]

One common solution is to define your indexes in an app, and track that app in some form of version control.

This allows you to deploy and update indexes.conf quickly and easily to both your indexers (or cluster master) and your search heads.

[krusty]

Could you manage the indexes.conf by the deployment server also for the cluster master? Do you have an example for me how to configure this?

[bcyates]

You can manage your cluster master with your deployment server. You have to set your deploymentclient.conf on your CM to send apps to your master apps directory instead of just etc/apps. It would look something like this:

[deployment-client]
serverRepositoryLocationPolicy = rejectAlways
repositoryLocation = $SPLUNK_HOME/etc/master-apps

And on your deployment server in your serverclass.conf, you would have to set something like this for your cluster master server class:

[serverClass:]
stateOnClient = noop

Then you'd be able to create an app with your indexes.conf, push it to your cluster master master apps with your deployment server, which would then push to the slave apps on your indexers.

[krusty]

@bcyates, thank you very much! You saved my day. Your suggested solution works fine.

Just for information, I checked the latest version of splunk (7.0.2) there the authentication_roles.xml is the same as mine with version 7.0.0. So the issue isn't fixed by Splunk yet.

[soniquella]

Thank you for your advice. I have taken this onboard.

I was looking for further definitive steps on how best to progress with this situation as it stands if possible i.e.

Can I now just remove all roles from the indexer?
How do I define the missing indexes on the search head?

Thank you.