some help about Splunk cluster add-on dev

[lremember]

When I developed an add on based on Splunk's cluster， encountering some problems:

1、I created an index named test from the indexes cluster of Splunk，Also through "./splunk edit cluster-config -mode searchhead -master_uri <Indexer Cluster Master URI>"    command linked search head cluster nodes with index clusters.   I want to write data to the index of this test through the Splunk API and obtain the written data from other search header nodes, but I found that it is not working. Is this related to my previous creation of indexes in the search header node. If it's relevant, how can I remove the index from the search header cluster？

2、Is kvsore data synchronized in the search head cluster. What should I do if I want to clean up the environment and delete a KVStore in the search header cluster？

3、What is the data communication mechanism between search head clusters, and do I want to achieve some data synchronization on add on between multiple search heads? Is there any good method?

BR!

[isoutamo]

Hi

I'm not sure what you are trying, but some comments which maybe helps you?

1. If you have created indexes on SHC side, it's not affect on indexer cluster. Those are totally different entities and all indexes to indexer cluster must created via MN (manager node).

What you are meaning with this " I want to write data to the index of this test through the Splunk API and obtain the written data from other search header nodes,"?

Usually data is written to indexes by normal ingesting process.

I suppose that you have set SHC side outputs.conf send all logs to indexer cluster and as you have added your MN as search target on your SHC side this should work. Of course it depends on how you have configured your SHC is this needed on every SHC node or is it enough to do it only in one node (if I recall right)? There are instructions on docs for this.

2. Kvstore data is synchronised all time on SHC. If not then you must fix your SHC cluster configuration. You can see this from internal logs, MC or CLI. 

If you are using your own kvstore collections, you could clean those. But if you are cleaning SHC's own collections then you mesh up the SHC itself and you must create it again. Look more from docs.

3. What you are meaning with this? How SHC manage its internal communication, synchronisation etc have described on docs. Just read more from here https://docs.splunk.com/Documentation/Splunk/9.1.2/DistSearch/SHCarchitecture

r. Ismo

[lremember]

Dear isoutamo：

Thanks for you reply！

I created indexes through MN (manager node) and then linked the search header nodes with the index of the peer cluster by executing the command "./split edit cluster configuration mode searchhead master_uri<Index Cluster Master URI>" on each search header set node. In fact, this approach is feasible.

However, my self-developed add on also has automatic index creation configured. I found this behavior in GPT, and if I write data to this index through the API, it is actually written to the same named index of the peer cluster. This is also the result I want, because I want to achieve synchronization of add on data between search head clusters in this way.

The current situation is that I have three search head nodes, and two of them have achieved this effect. The other node still writes data to the index created by its own node, not the index in the peer cluster

[isoutamo]

It's (almost) impossible to say what is your real issue based on these information what you have told.

Normally you shouldn't create automatic indexes by your Add Ons etc. Just create separate SA (etc.) which is separate from your main Add On or App and install that part only on your indexers / MN. Probably it's best to remove that indexes definition on your SHC side and check if this helps. Also if your Add On create automatic that index on SH side, remove that part too.

To be honest, I don't trust Chat GPT's instructions how and what to do with Splunk. Most of those instructions, which I have look are "scrap" and sends you to wrong direction / habits!

[lremember]

Thank you for your help. You are right. With your guidance, I have successfully solved my problem!

[lremember]

Dear splunkers:

@PickleRick  @richgalloway @isoutamo 

can you give me some suggestions to help me solve my problem?

Thank you!