How to do the equivalent of a debug/refresh on cluster master master-apps?

[thisissplunk]

I'm updating /master-apps/_cluster/local/indexes.conf and then pushing the bundle. I check the cluster's search head and my new index is there and ready to go. HOWEVER, the following API endpoint has yet to see my new index:

/services/cluster/master/indexes

I solved a similar issue for my /system/local/inputs.conf changes on my forwarder by using this to force a reloadvia the api:

/services/data/inputs/monitor/_reload

There seems to be no such api endpoint for the /services/cluster/master though. Even doing the following on the master CLI doesn't update the index list:

.../splunk reload index

What gives? How do I reload the indexes.conf after I make a change so that I can confirm my index was created? Not even a splunk restart on the master is picking up the master-apps changes.

[dxu_splunk · Accepted solution]

You need data in the newly created index, and then the Cluster Master will create an entry at cluster/master/indexes.
(The cluster master just sits there and listens for all the buckets the indexers has, it actually has no idea of what indexes its supposed to have - it never peeks into etc/master-apps/)

[MuS]

That's what I tried to explain, but much better wording here 😉 Thanks !

[thisissplunk]

Thanks, this was it. Sounds like I might want to check the search head or an indexer for the indexes then, as I need to know if an index exists whether it has data or not. Thoughts?

[dxu_splunk]

i'd check the indexers themselves for list of indexes. some indexes might be not replicated, and wont show up on the master/indexes endpoint.

| rest services/data/indexes | fields title repFactor

searching this will give u a list of indexes and whether they're replicated

[thisissplunk]

Ok thanks, I guess that will have to do. Was trying to avoid touching additional servers. It's already slightly annoying there is no API way to get changes into /master-apps/_cluster config files without also touching the master's filesystem.

Edit: Actually, if I go this route (the only route) for updating the master configs, then I can just view the local config on the master and assume that is what is on the indexers: https://answers.splunk.com/answers/387133/how-to-create-index-using-rest-api-in-a-clustered.html

Not ideal, but /shrug.

[MuS]

Do I get you right here, you created an index on the master in master-apps that will be pushed out to the idx cluster peers only? In this case the idx will never show up anywhere on the cluster master, because you did not create it there.

Hope to understand the question correct ¯\_(ツ)_/¯

cheers, MuS

[thisissplunk]

That is correct, but why do I see all of my other indexes in the master-apps/_cluster/local/indexes.conf when using that API endpoint? I feel like there is some kind of delay, or the new indexes aren't showing up because there isn't data in them yet. Testing that now.

[MuS]

Well, on the cluster master you will never have any data in the indexes shown using this API endpoint /services/cluster/master/indexes. Maybe the cluster-bundle apply or verify commands update the endpoint?

[thisissplunk]

Once I sent data to the index, this API endpoint on the master worked. Not sure what this endpoint is pulling from, but its not just a indexes.conf file like the inputs endpoint does.