Windows Deployment Server to *Nix deployment client permission issues.

[bmacias84]

Hello All,

Problem: Using a Splunk Deployment Server on Windows with *Nix Deployment Clients. When using a Win Deployment Server *Nix Deployment Clients are unable excute any .py or .sh. I assume that this is due to *Nix system do not understand Win file permissions.

I thought I could get around this by using umask 0002 (effective permission 0755) for the splunk user on *Nix by editing the .bashrc in splunk's $HOME dir. This did not appear to fix the issue.

Temp solution: chmod -R 775 *.sh for my deployed app; however, this is great until I modify the deployment app from the deployment server and reload. Then I face the problem all over again.

Currently testing some solutions, but does any one know how to overcome this besides running a *Nix Deployment Server or unpack and repacking on *Nix box before adding to deployment Server?

Update:

The splunk daemon runs under the splunk user context. All file system objects are owned by splunk user and splunk group.

The deployment client applies the following permissions:

folders - 700

file - 644

From bash if you define the interpreter /bin/sh $SPLUNK_HOME/etc/apps/Splunk_TA_nix/bin/cpu.sh I no longer receive a permission denied.

From the inputs.conf scripts stanza [script:///bin/sh bin/cpu.sh] or [script:///bin/sh ./bin/ps.sh] I receive an errors:

ERROR FrameworkUtils - Incorrect path to script: /bin/sh. Script must be located inside $SPLUNK_HOME/bin/scripts.

ERROR ExecProcessor - Ignoring: "/bin/sh ./bin/cpu.sh"

Thanks in advance!

Previous post from 2010:application-scripts-not-executable-when-deployed-via-deployment-server

[lukejadamec]

We have the same problem, and since there are no good solutions we created a patch.

1) Create a script called setattributes.sh that changes the permissions on the scripts (or App directories) and place it in splunkforwarder/etc/apps/SplunkUniversalForwarder/bin. Note: we chose this app because for us it is Not a deployed app, so the permissions are never reset.

#!/bin/bash
chmod 755 -R $SPLUNK_HOME/etc/apps/DeployedAppfolder1
dos2unix $SPLUNK_HOME/etc/apps/Appfolder1/bin/*.sh
chmod 755 -R $SPLUNK_HOME/etc/apps/DeployedAppfolder2
dos2unix $SPLUNK_HOME/etc/apps/Appfolder2/bin/*.sh

2) Create an inputs.conf file in splunkforwarder/etc/apps/SplunkUniversalForwarder/local with interval=-1, so it runs the script once on startup (whether it is needed or not).
Note: we chose this app because for us it is Not a deployed app, so the permissions are never reset.

[script://$SPLUNK_HOME/etc/apps/SplunkUniversalForwarder/bin/setattributes.sh]
disabled=false
interval=-1

3) We treat all servers as remote (whether or not we have admin access), so we created a script called configure-setattributes.sh that configures the system, and we deploy all three files from the deployment server to a 'scripts' app.

#!/bin/bash
dos2unix /opt/splunkforwarder/etc/apps/scripts/setattributes.sh
mkdir /opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/bin cp /opt/splunkforwarder/etc/apps/scripts/setattributes.sh  /opt/splunkforwarder/etc/apps/Splunk/UniversalForwarder/bin/
mkdir /opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/local cp /opt/splunkforwarder/etc/apps/scripts/inputs.conf  /opt/splunkforwarder/etc/apps/Splunk/UniversalForwarder/local/

4) We provide instructions to the remote administrators, or run them ourselves if we have access:
cd /opt/splunkforwarder/etc/apps/scripts
chmod 775 ./configure-setattributes.sh
./configure-setattributes.sh

Notes:
The script needs to be run in an App that is Not deployed, this way the permissions will not change due to a deployment.
Everytime splunkd starts the setattributes script will run. If it is needed then great - the deployed apps will work like normal. If it is not needed then the permissions are refreshed.
Do not move or edit the files from the deployed scripts folder on the forwarder because this will trigger a redeployment.
If you add a new app then the setattributes script will need to be updated, and all of the above steps repeated.

[bmacias84]

I considered the same, but opted not after a debate with security, server, and info_sec teams.

[bmacias84 · Accepted solution]

I have a work around, but I am not entirely happy about it. For this work around to work you have to create a symbolic link to bash and use absolute paths in your inputs.conf, which can be little of a pain. Thanks to @jrodman for the idea of using symbolic link.

On your *nix box:

#will need to log in with root or root like permission depending on security.

ln –s /bin/bash /opt/splunk/bin/scripts/sh  #creates symbolic link. If you create symbolic link #in bin splunk with ignore the stanza at startup.

ls –ali /opt/splunk/scripts #verify link-link is owned by root other user.  Need to be owned by #user running splunkd

chown –h splunk:splunk /opt/splunk/bin/scripts/sh #changes symbolic link ownership to splunk #user and splunk group. Be sure to use the –H switch or you will change bash ownership and not #the link.

ls –ali /opt/splunk/bin/scripts #verify permissions were applied correctly

ls –ali /bin #verify bash permission and ownership was not changed

Edit your inputs.conf stanza from the Windows deployment server:

[script://$SPLUNK_HOME/bin/scripts/sh $SPLUNK_HOME/etc/apps/Splunk_TA_nix/bin/cpu.sh]

Or

[script://$SPLUNK_HOME/bin/scripts/sh $SPLUNK_HOME/etc/apps/<TA/App_Name>/bin/<scripts>]

Output from splunkd.log:

12-28-2012 11:09:35.187 -0800 INFO  ExecProcessor - Ran script: /opt/splunk/bin/scripts/sh /opt/splunk/etc/apps/Splunk_TA_nix/bin/cpu.sh, took 1108.7 milliseconds to run, 236 bytes read

This will need to be done for every interpretor you wish to use in scripted inputs.

I would appreciate any comments or other ideas.

Cheers,

[tlay]

For the record, in the year 2019 this resolved this Windows Server corner issue for us.
I tried with the above verbatim, but it only worked when I removed the delimiting slash in the inputs.conf stanza in the nix_TA. This may be due to improvements with the way Windows handles *nix inputs (mentioned in the release notes at some point over the last few years).
[script://$SPLUNK_HOME/bin/scripts/sh $SPLUNK_HOME/etc/apps/Splunk_TA_nix/bin/cpu.sh]
[script://$SPLUNK_HOME/bin/scripts/sh $SPLUNK_HOME/etc/apps//bin/]
We also set the permissions on the scripts locally on the system. If we update, then we will have to go back through and verify the permissions are proper per client unless we choose to modify our approach. 😕

[donald_xero]

On RHEL/CentOS/Debian that's chown -h (not -H) that you want.

[bmacias84]

I would also consider hacking the rpm package to add the symbolic link for sh as part of the install. If you plan on doing this consider using rpmrebuild.

[jrodman]

We don't currently have a solution to encode permission bits from windows files, as Windows itself doesn't really have. Also inferring executable bits seems really dirty -- either going to be a headache or introduce security hazards or both.

I guess you could say it was a design error to have our ground truth be filesystem files that we replicate across systems.

We also refuse to let people trivially script executing arbitrary programs outside splunk/app dirs as a basic security measure -- the idea is that an admin has to very explicitly choose programs that splunk should be willing to run (ie, put them in a splunk dir, or at least symlink them there), and not just accidentally have them be run. If we provide less than that assurance in any area, it's a bug IMO.

How to handle this cleanly for old and new patterns is a subject of some debate. For now I would recommend having a deployment server running on a unix system.

[spsponger2]

It's still extremely frustrating that this lack of functionality continues to exist after so many years. Switching to Linux deployment servers is very painful in our specific use case. If the functionality just isn't going to be added can we get a recommended work around from Splunk?

[bmacias84]

@donald_xero, I look forward to having another another workaround. Please post once you have something. Cheers

[donald_xero]

@bmacias84: you say "Adding a parma to script stanza for chmod would not work because splunk account would not have access to event change file permission only root would."

But that's not true. The splunk account does have permission to write to $SPLUNK_HOME/etc/apps, how else would the deployment client be able to deploy apps to that folder? If splunkd can create files and folders in that location, nothing is stopping it chmodding some of the files 0755 too.

Maybe the script stanza isn't the best place to put the config option, but it is possible for splunk to change those permissions.

[bmacias84]

@donald_xero, Not sure if I understand your question. The problem is not with deployment of the files, but execution of any .sh, .pl, .py, etc file. When deploying from windows deployment server to a linux server Execution rights are removed.

[donald_xero]

@bmacias84: The splunk process has to be able to write to /etc/apps for the deployment client to be able to deploy apps there. If it can create the files, it can set their permissions.

[jrodman]

It is possible to deploy files as executable from unix to unix. The intermediary representation is just a tarfile, which encodes executable bits just fine.

[bmacias84]

@donald_xero,

What bin folder are you referring to. Apps are deploy under $SPLUNK_HOME/etc/apps. Have files deployed as executable not currently possible within splunk. Adding a parma to script stanza for chmod would not work because splunk account would not have access to event change file permission only root would. So if you didn't run splunk as root you couldn't change file perms.

[donald_xero]

Would it be possible to automatically make e.g. files in an app's bin folder executable when deployed on a Unix/Linux host? Or maybe add a parameter to the [script://...] section like chmod=0755, to be applied to the script that is referenced?

[bmacias84]

@jrodman, I'll probably add a comment to the deployment server documentation regarding this. Thanks for the idea regarding using the a symbolic link as it resolves my immediate issue. I have a feeling the workaround is going to have some caveats. I do have an open support case.

[jrodman]

It runs just fine, it's just there's a gap in terms of cross-platform metadata encoding. If you needed specific ACLs on the windows files from unix you would also be unable to achieve that.

It's a fair point that this isn't documented. I think it took a looong time to surface after the implementation and documentation were created, and is probably in a known issues list but that's probably not high visibility enough.

[spsponger2]

What can we do to get more visibility on this or bump up the priority on the known issues list?

[bmacias84]

Reread my comment. It comes across as harsh and it was not my intent. I am not dropping my Win implementation, but not have a properly running Deployment Server gives Linux implementation a serious advantage.

[bmacias84]

I've always been told by Splunk employees that Splunk run just as well on *Nix and Win systems. If I want to run a All-in-one solution on a Windows platform I couldn't if I had any *Nix servers in my environment. I've read a vast majority of the documentation and nowhere is this documented (as far as I can tell), which makes it hard to account for in a deployment. For this to be a bug as far back as two years plus just show that Windows implementations are still second class citizens in the Splunk world. Making a case to dump Splunk Win in favor of a *Nix implementation.