Deployment Architecture

How to force shcluster member to send config from local dir (e.g. savedsearches.conf) to captain from cli or via rest

gots
Path Finder

I have reason to change *.conf in local directory of app on the searchhead in shcluster, and i need to tell to this searchhead to send changes to captain for spreading via whole cluster.
How i can do it via cli or REST?

Thank you

0 Karma

gjanders
SplunkTrust
SplunkTrust

May I suggest you change this concept completely?

Instead of working against the clustering and updating configuration files on the individual search heads, work with the clustering by updating saved searches via the REST API
You could an updated search via REST and that will work with the clustering...

https://localhost:8000/en-US/debug/refresh?entity=saved/searches

Might also work, I'm unsure if that is officially supported or not, note that you would have to update all search heads at the same time and then trigger something like that, and I'm not sure it would work as expected...but it's something you could try

0 Karma

skalliger
SplunkTrust
SplunkTrust

That's just against the concept of a master or - in this case - the captain. The captain always has the working version. If you'd update a config file on one search head and thus destroy your app on the whole cluster, this would be a big problem.

The concept is good, because even if you mess up your configuration on a node, you can simply resync/push out the last working configuration files from your captain and everything is fine. I would not recommend using any version control system on Search Heads. If you do development, do it on a standalone Splunk instance and deploy the app to the cluster via the captain.

0 Karma

koshyk
Super Champion

welcome to the pain of Splunk SH clustering. We never change anything via directly into the SH member, but always makes changes into deployer and pushes it out. Unfortunately it goes into "default" of the members

0 Karma

gots
Path Finder

Thank you.
Splunk shcluster is like wedding - no one says why it's bad, but everyone talk why it is great... and you have no way back.

adonio
SplunkTrust
SplunkTrust

how do you make that change? if you save a search or report or anything of that sort, change should replicate across cluster members regardless to where you made the change

0 Karma

gots
Path Finder

i working directly with savedsearches.conf with vi-editor.

0 Karma

adonio
SplunkTrust
SplunkTrust

and when you save the file, the saved file does not replicate to other members?

0 Karma

gots
Path Finder

Yes, because splunkd nothing know about changed savedsearches.conf.
In standalone searchhead i can go to https://splunk/debug/refresh and splunk will reload configs, but in shcluster i need not only reload configs, but also send to captain information about changed configs so old method is not suitable.

0 Karma

adonio
SplunkTrust
SplunkTrust

what is the use case for editing savedsearches.conf using VI directly on a search head?
if its part of an app, why not modify on deployer and push to members?
on the other hand, maybe use case is worthwhile to bring to splunk attention in a form of ER.

0 Karma

gots
Path Finder

🙂 it is long, sad and strange story with some bugs in splunk which was not fixed in current version, but have fixed in some build of previous...
Reload from cli can make my life more comfortable and the hair silky.

0 Karma

adonio
SplunkTrust
SplunkTrust

sounds like General Hospital, but if you are already so deep in #@$%, why not just scp to the other search heads and get it over with?

0 Karma

gots
Path Finder

I already doing it via github for versions control.

0 Karma
.conf21 CFS Extended through 5/20!

Don't miss your chance
to share your Splunk
wisdom in-person or
virtually at .conf21!

Call for Speakers has
been extended through
Thursday, 5/20!