What is the best way to have Splunk run a shell command on demand on a Splunk server?

[teedilo]

What is the best way to have Splunk run a shell command on a Splunk server on demand? I found a way to do this via scripted input. A bit cumbersome for my needs, but it works. Are there any better ways to do this? Further explanation is warranted to understand why I'm asking...

Our Splunk servers were configured to run Splunk as root (on Linux). I don't know why they were set up this way, but my guess is that this was needed as an easy way to provide Splunk with access to certain files that it normally might not have access to. That works OK, but the problem comes in whenever we need to run commands on Splunk data folders (e.g., du, rm, etc.). Those of us who administer Splunk do not have root access since we are not part of the group that administers the servers themselves. However, we are given access to run specific sudo commands that typically meet our needs.

For example, to run the du command for the volume where our Splunk indexes are located, I need to:

1) Shut down Splunk.

2) Change the ownership of all files to our own user account (since many of them will have been set to root).

3) Run the du command.

4) Restart Splunk.

We currently have sudo commands for all of these except for du. I could certainly request to have du added to the list of sudo commands, but maintaining the sudo commands has been a headache as well, so I sought an alternative that might eliminate the need for many types of sudo commands.

I found that by creating a script that provides scripted input, I was able to run whatever commands I wanted by Splunk itself, since Splunk was running as root. This is a little kludgy, since it's a use of scripted input that was never intended. For example, you need to provide a schedule for the script to run, even though I just want to run it on demand.

Can anyone think of a better alternative for what I'm trying to do? The method I found is certainly workable, but I figured I would solicit other ideas.

I see that it's possible to call a Python or Perl script via the search inteface and commands.conf file. I don't know Python or Perl, however. I guess I could try to figure out how to create a Python or Perl wrapper to regular bash script. I see a runshellscript.py in the bin folder for Search, but apparently it's only designed to run in an alert context.

Thanks.

[yuanliu]

@martin_mueller's is really the best way. One possible workaround, if your main use is du and commands that do not change file system, is to change those directories to a group that includes you, then give group read to them, e.g., cd /opt/splunk; chgrp -R YOURGROUP .; chmod -R g+r ..

The caveat is that Splunk must not produce new temporary directories. This is also why it may not work well with rm. You can run rm with files with no problem. But if you delete a directory and the directory is recreated, it will belong to root again. Commands like du doesn't care if a file is not readable.

Which leads to a second caveat. The simplistic recursive commands above changes permission for existing files under those directories. If this is a concern for SysAdmin, you can ask them to change only directories. It's a very simple change.

[teedilo]

Thanks for the suggestions, yuanliu. I have no doubt that martin_mueller's suggestion is really the best way. I just don't know that I'll have the bandwidth to try that out, as I've been spending way too much time on Splunk admin as it is.

[martin_mueller · Accepted solution]

The best alternative (seriously!) is to stop running Splunk as root.

Stop Splunk, chown all files including all buckets to a non-root user, run splunk enable boot-start -user thatuser, start Splunk, and off you go.

[teedilo]

Thanks for your reply, Martin. I asked the person who first installed Splunk here about this and he couldn't recall why Splunk was configured to run as root. It might not have been a good reason, but there must have been a reason. If we do what you suggest then we'd no doubt have to then tackle whatever problems arise by not running as root. In any case, I'm accepting your answer as the correct one. Thanks.