Why do so few apps support search head clustering?

[dbizzle]

When all the apps that the business finds and wants to use, do not work with search head clustering? I'm not talking about user contributed apps either, I'm talking about apps that Splunk writes and supports like Splunk Mint App, Splunk for Windows Infrastructure, Splunk app for Unix and Linux, Splunk app for CEF.. and I haven't looked but I can only assume the PCI, Security and Advanced threat apps probably wont work with search head clustering either. I've watched release after release of Splunk come out and each one of those apps get "updates" to support those new releases, yet none of them make any progress in supporting search head clustering?

What gives?

[dbizzle]

Na, I'm not taking your comments as being defensive, in fact I welcome them. I want to know just as much as the others that peruse this forum. What I meant by bringing them together was more along the lines of still being able to configure things via the GUI even if those aspects remain on the master node.

When I have the opportunity to do more testing I can certainly post specific examples and hopefully get the community to weigh in. Usually problems I encounter I'll try to work with support to resolve, I am paying for it after all.. but sometimes it make sense to present the question to the collective conscious.

[lguinn2]

tl;dr - It appears that you have had a variety of problems, some more clearly related to apps/clustering than others.

Indexers getting duplicated events may be a problem caused by an app, but it doesn't have anything to do with search head clustering - unless perhaps you are talking about summary indexing. In the case of summary indexing, this may be an indication of another problem listed below.

You should always install an app on a staging server and configure it there, then use the deployer to push the already-configured app to the search head cluster. (I note that you have tried this, but just wanted to mention it for completeness.)

If knowledge objects are not replicating properly, that is a bug or misconfiguration - unless you are editing the knowledge objects by editing or copying the .conf files directly. Splunk can't detect and replicate that. If you make any direct edits to .conf files, you must use the deployer to update all the search head cluster members. Also, if search artifacts aren't replicating properly, or if scheduled reports are running more than once: that is a bug or misconfiguration. Neither of these problems should be related to an app. (Exception: the app setup may be making the kind of "direct edits" that cause problems, so set up as explained above.)

I laughed when you said "servers must grow on trees at Splunk." Actually, when you look at the apps in Splunkbase, it is clear that most of them were originally developed and tested on standalone Splunk servers - not in a distributed environment and not in a clustered environment. That is true even for most of the Splunk apps. But it is changing.

It appears that you are doing most of the things I've mentioned (staging, etc.) already. And not being a member of either Splunk Engineering or Splunk Support, and without the details of each problem, I really can't venture why this didn't work for you. All I can say is that search head clustering is a very new feature. If Splunk had waited to release it until all the apps were vetted against the new feature... let's just say that it would still be a while before Splunk 6.3 could have been released.

Since you have been around Splunk for a long time, you probably have noticed that new features often become available for manual configuration in Splunk before the feature can be managed via the Splunk GUI. Indexer clustering and search head clustering are two recent examples. I hope and expect that we will continue to see this pattern. I expect to see more areas - such as users and roles - included in SH cluster management and in the Splunk GUI - and hopefully soon! (I specifically didn't use any inside knowledge to find out about plans in this area.)

Finally, indexer clustering and search head clustering are two independent features. You can have one without the other. So I don't expect that we will co-mingle them, as the indexer tier and the search head tier have different needs and behaviors. But I do believe that the features will continue to be loved, and that more apps will be designed and tested in distributed environments. As an example, look at the tremendous improvements in indexer clustering since its introduction in Splunk 5.0.

I hope I am not coming across as defensive, as I agree with what you want. I just wanted to sort through some things for other readers.
If you have the time and energy to re-visit these issues and post specific problems on Answers, the community might be able to help now. As a group, I think we know a lot more about clustering than we did 6 months ago.

[lguinn2]

The Splunk® Enterprise Security app does support both search head clustering and indexer clustering. See the Deployment Planning page in the manual. Splunk® IT Service Intelligence also supports search head clustering and mentions it explicitly in the installation manual.

Many other apps, such as the Splunk® App for Unix and Linux, should run equally well on a search head cluster and a stand-alone search head. If there is nothing in the documentation that excludes search head clustering, then I would expect the app to work.

What problems have you experienced in running these apps in a search head cluster?

Update: it appears that I am wrong about some of the Splunk apps, and specifically about the Splunk® App for Unix and Linux. See comments below!

[dbizzle]

Thats good to know that apps that require additional licensing do in-fact work with search head clustering, but I would expect paid apps to work. However, in regards to the Splunk App for Unix and Linux, it states in its release notes that it does not support search head clustering. Splunk Mint App goes as far to say is 'kinda' supports search head clustering but doesn't really elaborate on what that means exactly. And I can attest that the CEF app, the Windows Infrastructure app (also mentioned in its release notes) and the Checkpoint LEA app all do not work properly with search head clustering in my experience.

The issues I've encountered can vary from app to app but range from configurations, search artifacts, and customized outputs not replicating properly to indexers getting duplicated firewall events, different cluster members executing the same schedule reports and apps that have setup wizards never seem to share the fact they've been configured.

I opened a case and worked with Splunk support for a few weeks trying to figure out why the CEF app would just stop working and was eventually told it just wasn't supported in a search head cluster. I've gone as far as configuring these apps on independent search heads, then pushing that configuration out the cluster to only be greeted with scheduling errors and splunkd crashes for scheduled reports related to the apps, or just the same kind of problems I was experiencing before.

Don't get me wrong, I love Splunk. I've been with you guys since the beginning (well, close to it at least), but sometimes it seems like servers must grow on trees there at the Splunk HQ. In an environment where high availability and data redundancy is a must, search head clustering helps fill the gap between individually managed search head configurations with a large user base spread across an multi-datacenter environment. Just enabling search head clustering alone you have to relinquish the ability to even use the interface to perform simple tasks such as adding new users and roles.

While I appreciate the new features that are introduced with each release, I think its high time you guys started to give some serious love to the clustering aspects and finally bring indexer and search head clustering together without having users sacrifice the simple UI based management tasks or forfeit some truly awesome apps in the name of resiliency.

[ntankersley_spl]

Dbizzle,

You're right on target, the issues around SHC support for many of our apps has to do with data management, configuration and customization replication across all the nodes in your cluster. In some instances apps will work out-of-the-box with SHC but as you've seen, that is rarely the case.

I am happy to report that Unix/Linux, Windows Infrastructure and Exchange will all support SHC on their next release which is coming very soon. I will update this ticket as soon as we officially release the updates.

[dbizzle]

this is very good news.

[lguinn2]

Good question and discussion, BTW.

Would love to see more people join in.

[Richfez]

dbizzle, I edited the title of your question to hopefully get it more attention - a lot of people just scan titles and don't click through unless it piques their interest.

[Michael]

Excellent question!

Of course, I would be happy if they would simply indicate where in my clustered environment a particular app should be installed... on the cluster manager? Indexers? Search heads? Heavy forwarders? I know, it "depends on whether or not there's transforms, or, whatever..." You'd think that would be easy to indicate in on page-one of any docs where it goes...