Thats good to know that apps that require additional licensing do in-fact work with search head clustering, but I would expect paid apps to work. However, in regards to the Splunk App for Unix and Linux, it states in its release notes that it does not support search head clustering. Splunk Mint App goes as far to say is 'kinda' supports search head clustering but doesn't really elaborate on what that means exactly. And I can attest that the CEF app, the Windows Infrastructure app (also mentioned in its release notes) and the Checkpoint LEA app all do not work properly with search head clustering in my experience.
The issues I've encountered can vary from app to app but range from configurations, search artifacts, and customized outputs not replicating properly to indexers getting duplicated firewall events, different cluster members executing the same schedule reports and apps that have setup wizards never seem to share the fact they've been configured.
I opened a case and worked with Splunk support for a few weeks trying to figure out why the CEF app would just stop working and was eventually told it just wasn't supported in a search head cluster. I've gone as far as configuring these apps on independent search heads, then pushing that configuration out the cluster to only be greeted with scheduling errors and splunkd crashes for scheduled reports related to the apps, or just the same kind of problems I was experiencing before.
Don't get me wrong, I love Splunk. I've been with you guys since the beginning (well, close to it at least), but sometimes it seems like servers must grow on trees there at the Splunk HQ. In an environment where high availability and data redundancy is a must, search head clustering helps fill the gap between individually managed search head configurations with a large user base spread across an multi-datacenter environment. Just enabling search head clustering alone you have to relinquish the ability to even use the interface to perform simple tasks such as adding new users and roles.
While I appreciate the new features that are introduced with each release, I think its high time you guys started to give some serious love to the clustering aspects and finally bring indexer and search head clustering together without having users sacrifice the simple UI based management tasks or forfeit some truly awesome apps in the name of resiliency.
... View more