integrating splunk with upstart (ubuntu/debian)

[crazygir]

is there a recommended way to integrate splunk with upstart, or should this simply be ignored for splunk's built-in init/rc scripts?

thanks!

[gkanapathy]

So, it turns out you can launch splunkd via "splunk start" in a non-exiting mode using the --nodaemon parameter:

./splunk start splunkd --nodaemon

This might make it a lot easier. It's unfortunately not actually in the official docs or the help, but it works.

The PID you care about really will just be the splunk start wrapper, so it kind of doesn't entirely do it to just look for that PID. Worse, if the splunkd process ends or fails, the wrapper doesn't exit, which is kind of fail. I guess with this, you can chose one problem or another.

[Lowell]

I messed around for several hours trying to make this work, but couldn't get what I was looking for. There are a couple of issues with this. One, upstart wants to directly launch the daemon processes, which doesn't play well with the "splunk start" command which really a wrapper that kicks off several background processes (splunkd and splunkweb).

So the options basically come down to (1) launching the all the various splunk daemons independently with separate upstart configurations, or (2) using upstart as a lame wrapper and simply sticking "splunks start" and "splunk stop" as pre-start and post-stop scripts at which point upstart wouldn't even know the currently status of the process. In older upstart versions you could manually specify a PID file, but that's gone away quite some time ago.

The problem with #1 is splunkd. Launching splunk web (which is really just cherrypy, which is really python) wouldn't be to bad, but launching splunkd manually would probably take some additional understanding of what the "splunk start" process does before actually launching the "splunkd" process, and chances are that could change a bit between versions of splunk, so even if you straced the whole thing and converted it into a nice little pre-start script in a upstart job, it could change next time you upgrade splunk, and some critical startup tasks could be missed. Which could be a big deal.

I stuck with the default (or mostly default) init.d script myself, but if anyone comes up with something better, I'd like to know about it too.

[gkanapathy]

So, it turns out you can launch splunkd via "splunk start" in a non-exiting mode using the --nodaemon parameter:

./splunk start splunkd --nodaemon

This might make it a lot easier. It's unfortunately not actually in the official docs or the help, but it works. Sort of.

[gkanapathy]

You could launch Splunk using upstart. Splunk will generate a default rc script and the corresponding links, but that is just a convenience and it doesn't depend on them at all. You can basically do whatever you like. If you want to generate the script, copy and use it for reference, then disable/delete it, that's fine. (Or ask someone who has a copy to send you one, whatever.)

All you have to do (and you will see this in the script) is simply run "$SPLUNK_HOME/bin/splunk start" as the correct user, and "$SPLUNK_HOME/bin/splunk stop" to stop, using whatever means you like.

[crazygir]

Great!

anyone have a working copy to share with the world (there is surprisingly little about this topic available)?

[gkanapathy]

Note that the command "$SPLUNK_HOME/bin/splunk start" starts up Splunk and then exits, so you'll have to take that into account in any scripting.