Scripting Splunk through automated systems

[hajducko]

Running into a strange issue here.

We're attempting to run through scripts through our config management system - Opsware (HP BSA) to configure splunk agents from the command line on our Windows systems.

The script, when placed manually on the system and run there - works fine.

However, when running the script through the tool - certain commands fail to work or produce any output at all.

For instance, the following don't work:

splunk add forward-server server.example.com:9997
splunk help add monitor
splunk list forward-server
splunk help

The following do work:

splunk restart
splunk stop
splunk start

And it's not just an issue of capturing the output from the commands - the commands simply do not run. For instance, when adding a forward-server, no outputs.conf is being created or written to.

The commands all return with -1073741819, which translates to 0xc0000005 - STATUS_ACCESS_VIOLATION.

We've tried several different ways of authenticating too - environment variables, doing a 'login' command before, using '-auth user:pass'. Nothing seems to help.

The scripts also seem to work fine on our Linux boxes - just not Windows. I realize it's an issue of our config management tool interfacing with splunk, but I'm curious as to why some commands would work and others wouldn't. I'm assuming something with how the config management tool's executing environment is setup, but I'm up for ideas on how to solve this.

[hajducko]

( Sorry for replying in another answer - comment field wasn't big enough )

We do provide the -auth user:password. Also tried setting SPLUNK_USER and SPLUNK_PASS and using the login command with -auth user:password before attempting the command - all of them fail.

The script runs with administrative privileges and we've tested it running with other users - all still fail.

We don't get an error either - we don't receive any output whatsoever.

Ironically enough, we had a few systems work today - the only difference was the ones that work are running Splunk 3.3.1. None of the systems running Splunk 4.x work. So something changed here.

I should also note that all of these systems are simply running the lightweight forwarder. We haven't tried upgrading to the 4.2 universal forwarder yet.

Tried changing the default password for the admin account - same results.

Opsware runs a local agent and executes the script by downloading it and executing the script locally, so every command we're issuing is coming from the localhost.

Is there some debug logging we could enable to see any command issued to splunk via the CLI?

[hajducko]

Also just tried the following:

splunk list forward-server -uri https://127.0.0.1:8089 -auth admin:test123

It doesn't even show up in the splunkd_access log as even having attempted to login. Like I said, it's as if the command isn't even being run - no idea why this works on Splunk 3.x but not 4.x

[gkanapathy]

Answer below. But first...

I agree with dwaddle. If I have the ability to push out Splunk config files, I always do so in preference to running commands. Config files have the advantage that you can completely control the final state of what the Splunk configuration does, i.e., they are "idempotent", unlike running commands. They have the advantage that the order in which they get deployed doesn't matter, and are trivial to recover from if there is some kind of failure, which isn't necessarily the case with running commands. Note also that if you're concerned about merging configurations, the right thing to do is to take advantage of Splunk's config file system, which allows an infinite number of (say) outputs.conf files that are all read and merged together by Splunk at run time: http://www.splunk.com/base/Documentation/latest/Admin/Aboutconfigurationfiles

What's going on here is that the ones that don't work (well, except for splunk help, though depending in the version you're running, there are bugs that do require it) require Splunk authentication and connect to a running Splunkd over the localhost (127.0.0.1) interface. If you are running as a non-interactive user, you'll have to provide -auth user:password, or you can use splunk login make sure the user has a persistent home directory where an authentication file can be saved.

Furthermore, you should make sure that the user that runs the commands actually has the right access to all the necessary Splunk config and executable files. splunk start, etc., don't necessarily need that much on Windows, because they call the OS net start calls, which have the service run as the service account. Finally, you have to make sure that the configuration you are running is either not using the default admin account with default password, or that it's connecting only over 127.0.0.1, and not getting routed over a different interface to get back to the splunkd process.

[hajducko]

Agreed - in fact, I do that for adding the win event logs to the inputs.conf.

I was a little nervous about doing it in terms of the outputs.conf, as we often have other items in there.

We'll eventually be making it so that all the config files are simply deployed and managed through Opsware - this was more of a one-time run that we were doing and this issue became an itch I needed to scratch to figure out why this was happening.

[hajducko]

I don't consider it answered. Even if we're not trying to configure splunk, getting output from the command terminal won't work.

For instance, I can't create scripts that will get me a list of the forward-servers via the 'list forward-servers -auth user:pass' because of this issue.

[dwaddle]

The answers site won't consider this question 'answered' until the answer is accepted by you. Please click the outline of the green checkbox next to the answer "accept" it. Thanks.

[dwaddle]

You could always have your Opsware stuff just write the appropriate config files for Splunk. They are just text files, afterall. It should be a simple Powershell script.