Deployment Architecture

How to configure Splunk in a SUSE Linux Cluster?


I have an environment that is small enough for a simple single server setup of Splunk, but the data itself and access to Splunk is very important, so I have configured a 2-node High Availability SUSE Linux cluster (SLES 11 amd64) with a clustered DRBD storage back-end, file system and virtual ip.

I have installed Splunk into the DRBD storage area so that it can fail betweeen my 2 node cluster. This gives me everything I need except for clustering the Splunk services.

Does anyone have by chance, an example cluster cib.xml file, or the cib entries that would be applicable for Splunk? I'm assuming it would use a generic-service resource agent as I could not find any cluster resource agents specific for Splunk.

Just trying to save myself lots of work doing this myself. If no one has this info, and I'm successful, I'll be more than happy posting back how it's done.


Splunk Employee
Splunk Employee

I know the commercial side of DRBD built an example configuration to try to do some co-selling with us. I think we had too many projects on our end and didn't pursue so far, but if you have a commercial relationship with that enterprise, I bet they could dig it up.

I haven't used drbd since 2001 so I'm out of date.

Things to be aware of:

  • Splunk can shut down slowly if it's being fed by light forwarders with large files. If you want to avoid missplit events etc, it's best to let this finish.
  • Splunk however should always have a searchable index regardless of how it shuts down.
  • Splunk doesn't ensure indexes are locked against other splunk indexes, because that's not really a supported model (you can finagle it in some cases). Multiple splunks writing to the same hot buckets will be v. v. bad, while multiple splunks rolling the same buckets will be bad. Be sure your favourite cluster manager is capable of avoiding this case very well.


Thanks jrodman. I've written my own heartbeat cluster resource agent which seems to work OK. I have extended the timeouts for start/stop (from default recommended cluster timings) and it now starts, stops and is monitored correctly. The points you've raised are very valid and I'll now be sure to test it thoroughly with those in mind.
So at this stage I have a working clustered setup with a DRBD, file system, virtual ip, syslog-ng (separate instance - I know Splunk supports syslog udp out of the box but I need it for other reasons), and Splunk which successfully starts, stops and fails over.

0 Karma


scarteratwork ,

Can you share with us the heartbeat cluster resource ?

I´m trying to update a project with current corosync/pacemaker/drbd but got stucked at the
pcs heartbeat daemon.

Tried to get an apache daemon and modify it to splunk start|stop|status but it´s still failing ;(

0 Karma



Have you been able to get it to work ? I am trying to achieve the same too 😉

Thanks !

0 Karma
Get Updates on the Splunk Community!

2024 Splunk Career Impact Survey | Earn a $20 gift card for participating!

Hear ye, hear ye! The time has come again for Splunk's annual Career Impact Survey!  We need your help by ...

Optimize Cloud Monitoring

  TECH TALKS Optimize Cloud Monitoring Tuesday, August 13, 2024  |  11:00AM–12:00PM PST   Register to ...

What's New in Splunk Cloud Platform 9.2.2403?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.2.2403! Analysts can ...