How to create index using REST API in a clustered environment?

[dhavamanis]

Can you please tell us, How to create index using REST API in a clustered environment?

[Flesz]

4 years later....

Have there been any improvements with that? 

[MuS · Accepted solution]

Hi dhavamanis,

according to the docs http://docs.splunk.com/Documentation/Splunk/6.3.3/RESTREF/RESTcluster#cluster.2Fmaster.2Findexes a POST is not supported, therefore you will not be able to create an index in the cluster directly.
But, you can create it on the Cluster Master itself using POST on this REST endpoint http://docs.splunk.com/Documentation/Splunk/6.3.3/RESTREF/RESTconf#configs.2Fconf-.7Bfile.7D using indexes.conf. This will create your indexes.conf on the Cluster Master in $SPLUNK_HOME/etc/system/local/ which can be copied over to $SPLUNK_HOME/etc/master-apps/_cluster/local on the Cluster Master using a cron job. After the copy the cluster bundle can be applied using this undocumented REST end point:

 curl -k -u admin:changme https://localhost:8089/services/cluster/master/control/default/apply

This is a work around but it will get you there in the end.

Hope this helps ...

cheers, MuS

[thisissplunk]

Could you symlink system/local to master-apps/_cluster/local instead of copying from one to the other?

[MuS]

The question would be here should you instead of could you 😉 While technically possible this will for sure not be support you will be on your own when things start to get crazy and you need help from support ....

cheers, MuS

[thisissplunk]

Well until Splunk gets off their lazy butts and make the API cover all basic use cases, we'll need a way to not require OS level access for users running automation scripts and this sounded like a solution. The API is pretty weak.

[MuS]

So, what prevents you from using your automation script to change indexes.conf in $SPLUNK_HOME/etc/master_apps/<whatevername>/local/indexes.conf and call the un-documented REST endpoint /services/cluster/master/control/default/apply to apply the cluster bundle ?

here is an example to call the endpoint:

curl -k -u admin:changeme https://localhost:8089/services/cluster/master/control/control/apply  -X POST

cheers, MuS

[thisissplunk]

Well, the API does not write to master-apps folder at all. You can see that in this answer: https://answers.splunk.com/answers/387133/how-to-create-index-using-rest-api-in-a-clustered.html or try using the /servicesNS/nobody/ endpoint. I'm already using the undocumented bundle push endpoint like you mentioned though.

We have to use the API because of US vs UK data sensitivity, and well, best practice. We cannot allow certain employees to have OS access at all, hence trying to figure out how to prevent the need for OS accounts doing anything on the master or forwarder, hence symlinks.

[MuS]

Look, in my almost 8 years of providing Splunk PS I came along a lot of things that are not supported by Splunk, but doable 😉
For example: one customer had its cluster master setup as deployment client to work around this problem; they created the indexes.conf on the deployment server using REST calls, the indexes.conf was symlinked into a deployment app, which was deployed to the cluster master into $SPLUNK_HOME/etc/master-apps/ ... it worked. Is it supported, probably not.

But you can always file an enhancement request with Splunk if you reckon this is a needed feature, use the Support Portal here https://www.splunk.com/en_us/support-and-services.html

[thisissplunk]

Yes, that is effectively what we are trying to do here, just on the same box. Not sure we want to spin up another box for it. Don't get me wrong, I love working with Splunk, but I am very disappointed at how such an integral part of Splunk administration is not possible with the API.

Thanks for the help so far and I will look into filing a request.

[thisissplunk]

Is this still the only way to create an index in a cluster with the api? Very disappointed.