Script failing to install the forwarder remotely

[macadminrohit]

Hi Experts, i have a script which is supposed to install forwarder on the remote machines . The script does quite good job of installing it but its showing errors at two places. Below is the script and the errors it is throwing.

######## UF_install.sh Script ##############
#!/bin/sh
#### forwarderlist.txt contains the IP address of the forwarder to SSH into
HOSTS_FILE="/opt/splunk/bin/scripts/forwarderlist.txt"
### Download the latest version of the installer from splunk site
WGET_CMD="wget -O splunkforwarder-7.2.5.1-962d9a8e1586-Linux-x86_64.tgz 'https://www.splunk.com/bin/splunk/DownloadActivityServlet?architecture=x86_64&platform=linux&version=7.2.5.1&product=universalforwarder&filename=splunkforwarder-7.2.5.1-962d9a8e1586-Linux-x86_64.tgz&wget=true'"
INSTALL_FILE="splunkforwarder-7.2.5.1-962d9a8e1586-Linux-x86_64.tgz"
DEPLOY_SERVER="10.0.1.39"
PASSWORD="ExpertInsight"
### installation steps
REMOTE_SCRIPT="
cd /opt
yum install wget
yum install vim
sudo $WGET_CMD
sudo tar -xzf $INSTALL_FILE
sudo useradd -m -r splunk
sudo chown -R splunk:splunk /opt/splunkforwarder
### /opt/splunkforwarder/bin/splunk enable boot-start -user splunk
sudo -u splunk /opt/splunkforwarder/bin/splunk start --accept-license --answer-yes --auto-ports --no-prompt --seed-passwd $PASSWORD
sudo -u splunk /opt/splunkforwarder/bin/splunk enable boot-start
sudo -u splunk /opt/splunkforwarder/bin/splunk set deploy-poll $DEPLOY_SERVER --accept-license --answer-yes --auto-ports --no-prompt  -auth admin:changeme
sudo -u splunk /opt/splunkforwarder/bin/splunk edit user admin -password $PASSWORD -auth admin:changeme
### SCP (copy) the files from Search head into the folder where the user has access to
sudo scp -r /opt/splunk/bin/scripts/deploymentclient.conf ec2-user@$HOSTS_FILE:/home/ec2-user/deploymentclient.conf
# Change permissions to splunk user
sudo chown -R splunk:splunk /home/ec2-user/deploymentclient.conf
# Then copy the file to appropriate directory
sudo cp -r /home/ec2-user/deploymentclient.conf /opt/splunkforwarder/etc/system/local/
# once the file in /etc/system/local restart to take effect
sudo -u splunk /opt/splunkforwarder/bin/splunk restart
"
### Continue the same for other UF hosts
echo "In 5 seconds, will run the following script on each remote host:"
echo
echo "===================="
echo "$REMOTE_SCRIPT"
echo "===================="
echo
sleep 5
echo "Reading host logins from $HOSTS_FILE"
echo
echo "Starting."
for DST in `cat "$HOSTS_FILE"`; do
if [ -z "$DST" ]; then
continue;
fi
echo "---------------------------"
echo "Installing to $DST"
sudo ssh -t "$DST" "$REMOTE_SCRIPT"
done
echo "---------------------------"

Can't open file "/etc/systemd/system/SplunkForwarder.service" for writing
: Permission denied
Login failed
Login failed
cp: cannot stat ‘/opt/splunk/bin/scripts/deploymentclient.conf’: No such file or directory
chown: cannot access ‘/home/ec2-user/deploymentclient.conf’: No such file or directory
cp: cannot stat ‘/home/ec2-user/deploymentclient.conf’: No such file or directory

The deploymentclient file is kept at its location defined in the script on the deployment server but still shows the error and also not sure what this error is about Can't open file "/etc/systemd/system/SplunkForwarder.service" for writing

[DavidHourani]

Hi @macadminrohit,

This error : Can't open file "/etc/systemd/system/SplunkForwarder.service" for writing is due to sudo -u splunk /opt/splunkforwarder/bin/splunk enable boot-start as you try to add Splunk as a service. Authentication failures are due to some of the splunk commands you used that require you to be authenticated first. As for the cp: cannot stat ‘...' error that's indicating that the path you're copying from doesn't exist or could be a possible typo.

I think best way to troubleshoot your script is to run step by step on one of your destination servers and see which command works and which one does. Then you can compile it into this one script.

PS: It's best to use a tarball that you modify and push instead of downloading directly to all your forwarding hosts. Also put your deploymentclient.conf in an app and not in system/local that way you'll be able to overwrite it via deployment server if needed : sudo cp -r /home/ec2-user/deploymentclient.conf /opt/splunkforwarder/etc/system/local/

Cheers,
David

[koshyk]

I feel, the script is too much depedent on wget and no much error capture and lot of errors like the file is copied to "/opt/splunkbin/scripts/deploymentclient.conf"

The simple way is to
1. Download the tgz of Splunk in your staging server which has ssh connectivity to your clients
2. Ensure all settings are created as apps within your staging Server (or git repository). For example, you may need MY_uf_app (which contains deploymentclient.conf settings, TLS settings etc.)
3. So ultimately the installation is, just SCP the splunk installable tgz & your settings app to client. Untar and copy your apps to $SPLUNK_HOME/etc/apps/ directory
4. Create "splunk" user and group in client
5. If you need you can then do boot-enable and restart as "splunk" user