Deployment Architecture

How to configure Splunk in a SUSE Linux Cluster?

scarteratwork
Explorer

I have an environment that is small enough for a simple single server setup of Splunk, but the data itself and access to Splunk is very important, so I have configured a 2-node High Availability SUSE Linux cluster (SLES 11 amd64) with a clustered DRBD storage back-end, file system and virtual ip.

I have installed Splunk into the DRBD storage area so that it can fail betweeen my 2 node cluster. This gives me everything I need except for clustering the Splunk services.

Does anyone have by chance, an example cluster cib.xml file, or the cib entries that would be applicable for Splunk? I'm assuming it would use a generic-service resource agent as I could not find any cluster resource agents specific for Splunk.

Just trying to save myself lots of work doing this myself. If no one has this info, and I'm successful, I'll be more than happy posting back how it's done.

Thanks,
Steve

jrodman
Splunk Employee
Splunk Employee

I know the commercial side of DRBD built an example configuration to try to do some co-selling with us. I think we had too many projects on our end and didn't pursue so far, but if you have a commercial relationship with that enterprise, I bet they could dig it up.

I haven't used drbd since 2001 so I'm out of date.

Things to be aware of:

  • Splunk can shut down slowly if it's being fed by light forwarders with large files. If you want to avoid missplit events etc, it's best to let this finish.
  • Splunk however should always have a searchable index regardless of how it shuts down.
  • Splunk doesn't ensure indexes are locked against other splunk indexes, because that's not really a supported model (you can finagle it in some cases). Multiple splunks writing to the same hot buckets will be v. v. bad, while multiple splunks rolling the same buckets will be bad. Be sure your favourite cluster manager is capable of avoiding this case very well.

scarteratwork
Explorer

Thanks jrodman. I've written my own heartbeat cluster resource agent which seems to work OK. I have extended the timeouts for start/stop (from default recommended cluster timings) and it now starts, stops and is monitored correctly. The points you've raised are very valid and I'll now be sure to test it thoroughly with those in mind.
So at this stage I have a working clustered setup with a DRBD, file system, virtual ip, syslog-ng (separate instance - I know Splunk supports syslog udp out of the box but I need it for other reasons), and Splunk which successfully starts, stops and fails over.

0 Karma

theunf
Communicator

scarteratwork ,

Can you share with us the heartbeat cluster resource ?

I´m trying to update a project with current corosync/pacemaker/drbd but got stucked at the
pcs heartbeat daemon.

Tried to get an apache daemon and modify it to splunk start|stop|status but it´s still failing ;(

0 Karma

guilmxm
Influencer

Hi,

Have you been able to get it to work ? I am trying to achieve the same too 😉

Thanks !

0 Karma
Get Updates on the Splunk Community!

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...

Edge Processor Scaling, Energy & Manufacturing Use Cases, and More New Articles on ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Get More Out of Your Security Practice With a SIEM

Get More Out of Your Security Practice With a SIEMWednesday, July 31, 2024  |  11AM PT / 2PM ETREGISTER ...