Deployment Architecture

How to configure Splunk in a SUSE Linux Cluster?

scarteratwork
Explorer

I have an environment that is small enough for a simple single server setup of Splunk, but the data itself and access to Splunk is very important, so I have configured a 2-node High Availability SUSE Linux cluster (SLES 11 amd64) with a clustered DRBD storage back-end, file system and virtual ip.

I have installed Splunk into the DRBD storage area so that it can fail betweeen my 2 node cluster. This gives me everything I need except for clustering the Splunk services.

Does anyone have by chance, an example cluster cib.xml file, or the cib entries that would be applicable for Splunk? I'm assuming it would use a generic-service resource agent as I could not find any cluster resource agents specific for Splunk.

Just trying to save myself lots of work doing this myself. If no one has this info, and I'm successful, I'll be more than happy posting back how it's done.

Thanks,
Steve

jrodman
Splunk Employee
Splunk Employee

I know the commercial side of DRBD built an example configuration to try to do some co-selling with us. I think we had too many projects on our end and didn't pursue so far, but if you have a commercial relationship with that enterprise, I bet they could dig it up.

I haven't used drbd since 2001 so I'm out of date.

Things to be aware of:

  • Splunk can shut down slowly if it's being fed by light forwarders with large files. If you want to avoid missplit events etc, it's best to let this finish.
  • Splunk however should always have a searchable index regardless of how it shuts down.
  • Splunk doesn't ensure indexes are locked against other splunk indexes, because that's not really a supported model (you can finagle it in some cases). Multiple splunks writing to the same hot buckets will be v. v. bad, while multiple splunks rolling the same buckets will be bad. Be sure your favourite cluster manager is capable of avoiding this case very well.

scarteratwork
Explorer

Thanks jrodman. I've written my own heartbeat cluster resource agent which seems to work OK. I have extended the timeouts for start/stop (from default recommended cluster timings) and it now starts, stops and is monitored correctly. The points you've raised are very valid and I'll now be sure to test it thoroughly with those in mind.
So at this stage I have a working clustered setup with a DRBD, file system, virtual ip, syslog-ng (separate instance - I know Splunk supports syslog udp out of the box but I need it for other reasons), and Splunk which successfully starts, stops and fails over.

0 Karma

theunf
Communicator

scarteratwork ,

Can you share with us the heartbeat cluster resource ?

I´m trying to update a project with current corosync/pacemaker/drbd but got stucked at the
pcs heartbeat daemon.

Tried to get an apache daemon and modify it to splunk start|stop|status but it´s still failing ;(

0 Karma

guilmxm
Influencer

Hi,

Have you been able to get it to work ? I am trying to achieve the same too 😉

Thanks !

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

(view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...