Hi all! Recently there has been a need to implement a centralized Splunk setup on Linux machines. I managed to do this without using ansible, starting from the script by lguinn2 : https://community.splunk.com/t5/Getting-Data-In/Simple-installation-script-for-Universal-Forwarder/m-p/21517. Today I want to share with everyone, of course, if you have any comments or improvements, please write! I got two slightly different scripts, 1 for CentOS and SuSe, 2 for Ubuntu and Debian. Why, because different installation packages were used. Yes, yes, it was possible to do something differently, I just had to make it as quickly as possible, and the level of knowledge of Linux is also hoarse)  Well, this post is for people like me)) Note. The script has been tested on an Ubuntu distribution, so it is recommended that you work on an Ubuntu machine. Successful completion of work requires SSH access to target devices, the ability to execute ssh and sshpass commands. All target machines must have an identical account with the same password, and this account must be as superuser (with the ability to run commands via sudo). MyForwarders and MyForwarders_U are simple text files for storing IP addresses of target machines: I think for the rest everything is clear from the description, even if there are questions during the time, you will understand everything! 1. #!/bin/bash # Credentials of the user who will connect to the target host and run Splunk. read -p "Enter SSH user name: " username echo -n "Enter SSH user password: " stty -echo read password stty echo echo INSTALLED=False # Logging file for Splunk status STATUS_LOG="/home/zhanali/splunk_status.txt" # File with machine's IPs HOSTS_FILE="/home/zhanali/MyForwarders" # Installation file location INSTALL_FILE="/home/zhanali/splunkforwarder-9.1.0.1-77f73c9edb85.x86_64.rpm" PREPARE_COMMANDS=" echo $password | sudo -S -k mkdir /opt/splunkforwarder 2>/dev/null echo $password | sudo -S -k chown -R splunk:splunk /opt/splunkforwarder 2>/dev/null " INSTALL_COMMANDS=" echo $password | sudo -S -k chmod 644 /opt/splunkforwarder/splunkforwarder-9.1.0.1-77f73c9edb85.x86_64.rpm 2>/dev/null echo $password | sudo -S -k rpm -i /opt/splunkforwarder/splunkforwarder-9.1.0.1-77f73c9edb85.x86_64.rpm 2>/dev/null echo $password | sudo -S -k /opt/splunkforwarder/bin/splunk start --accept-license --answer-yes --no-prompt --seed-passwd '!@#qweasdZXC' 2>/dev/null echo 'Please wait 10 second....' sleep 10 echo $password | sudo -S -k /opt/splunkforwarder/bin/splunk stop 2>/dev/null echo $password | sudo -S -k /opt/splunkforwarder/bin/splunk disable boot-start 2>/dev/null echo $password | sudo -S -k chown -R splunk:splunk /opt/splunkforwarder 2>/dev/null echo $password | sudo -S -k /opt/splunkforwarder/bin/splunk enable boot-start -user $username 2>/dev/null echo $password | sudo -S -k /opt/splunkforwarder/bin/splunk start 2>/dev/null echo $password | sudo -S -k mkdir /home/$username/.splunk 2>/dev/null echo $password | sudo -S -k chmod 777 -R /home/$username/.splunk 2>/dev/null echo $password | sudo -S -k sudo -u $username /opt/splunkforwarder/bin/splunk add forward-server 172.16.30.104:9997 -auth 'admin':'!@#qweasdZXC' 2>/dev/null echo $password | sudo -S -k sudo -u $username /opt/splunkforwarder/bin/splunk set deploy-poll 172.16.30.104:8089 -auth 'admin':'!@#qweasdZXC' 2>/dev/null " echo "In 5 seconds, will run the following script on each remote host:" echo sleep 5 echo "Reading host IPs from $HOSTS_FILE" echo echo "Starting." for DST in `cat "$HOSTS_FILE"`; do if [ -z "$DST" ]; then continue; fi echo "---------------------------------" | tee -a $STATUS_LOG echo "Starting work with $DST" | tee -a $STATUS_LOG sshpass -p $password ssh -q $username@$DST [[ -f /opt/splunkforwarder/bin/splunk ]] && INSTALLED=True || INSTALLED=False if [ "$INSTALLED" = "True" ]; then echo "Splunk UF is already installed" | tee -a $STATUS_LOG version=$(sshpass -p $password ssh $username@$DST "echo $password | sudo -S -k /opt/splunkforwarder/bin/splunk version | grep 'Splunk Universal Forwarder'" 2>/dev/null) echo "Splunk UF version: $version" | tee -a $STATUS_LOG status=$(sshpass -p $password ssh $username@$DST "echo $password | sudo -S -k /opt/splunkforwarder/bin/splunk status | grep 'splunkd is '" 2>/dev/null) echo "Splunk UF status: $status" | tee -a $STATUS_LOG dep=$(sshpass -p $password ssh $username@$DST "echo $password | sudo -S -k cat /opt/splunkforwarder/etc/system/local/deploymentclient.conf | grep '172.16.30.104:8089'" 2>/dev/null) fwd=$(sshpass -p $password ssh $username@$DST "echo $password | sudo -S -k cat /opt/splunkforwarder/etc/system/local/outputs.conf | grep '172.16.30.104:9997'" 2>/dev/null) if [ -z "$dep" ]; then echo "Deployment server is not configured" | tee -a $STATUS_LOG else echo "Deployment server is configured" | tee -a $STATUS_LOG fi if [ -z "$fwd" ]; then echo "Forward server is not configured" | tee -a $STATUS_LOG else echo "Forward server is configured" | tee -a $STATUS_LOG fi INSTALLED=False else echo "Splunk UF is not installed to host $DST" | tee -a $STATUS_LOG echo "Installing..." | tee -a $STATUS_LOG sshpass -p $password ssh $username@$DST "$PREPARE_COMMANDS" sshpass -p $password scp $INSTALL_FILE $username@$DST:/opt/splunkforwarder sshpass -p $password ssh $username@$DST "$INSTALL_COMMANDS" echo "Installation is done" | tee -a $STATUS_LOG echo "Checking..." | tee -a $STATUS_LOG status=$(sshpass -p $password ssh $username@$DST "echo $password | sudo -S -k /opt/splunkforwarder/bin/splunk status | grep 'splunkd is '" 2>/dev/null) echo "Splunk UF status: $status" | tee -a $STATUS_LOG dep=$(sshpass -p $password ssh $username@$DST "echo $password | sudo -S -k cat /opt/splunkforwarder/etc/system/local/deploymentclient.conf | grep '172.16.30.104:8089'" 2>/dev/null) fwd=$(sshpass -p $password ssh $username@$DST "echo $password | sudo -S -k cat /opt/splunkforwarder/etc/system/local/outputs.conf | grep '172.16.30.104:9997'" 2>/dev/null) if [ -z "$dep" ]; then echo "Deployment server is not configured" | tee -a $STATUS_LOG else echo "Deployment server is configured" | tee -a $STATUS_LOG fi if [ -z "$fwd" ]; then echo "Forward server is not configured" | tee -a $STATUS_LOG else echo "Forward server is configured" | tee -a $STATUS_LOG fi fi echo "---------------------------------" | tee -a $STATUS_LOG done echo "Done" And 2. #!/bin/bash # Credentials of the user who will connect to the target host and run Splunk. read -p "Enter SSH user name: " username echo -n "Enter SSH user password: " stty -echo read password stty echo echo INSTALLED=False # Logging file for Splunk status STATUS_LOG="/home/zhanali/splunk_status.txt" # File with machine's IPs HOSTS_FILE="/home/zhanali/MyForwarders_U" # Installation file location INSTALL_FILE="/home/zhanali/splunkforwarder-9.1.0.1-77f73c9edb85-linux-2.6-amd64.deb" PREPARE_COMMANDS=" echo $password | sudo -S -k mkdir /opt/splunkforwarder 2>/dev/null echo $password | sudo -S -k chown -R splunk:splunk /opt/splunkforwarder 2>/dev/null " INSTALL_COMMANDS=" echo $password | sudo -S -k dpkg -i /opt/splunkforwarder/splunkforwarder-9.1.0.1-77f73c9edb85-linux-2.6-amd64.deb 2>/dev/null echo $password | sudo -S -k /opt/splunkforwarder/bin/splunk start --accept-license --answer-yes --no-prompt --seed-passwd '!@#qweasdZXC' 2>/dev/null echo 'Please wait 10 second....' sleep 10 echo $password | sudo -S -k chown -R splunk:splunk /opt/splunkforwarder 2>/dev/null echo $password | sudo -S -k /opt/splunkforwarder/bin/splunk stop 2>/dev/null echo $password | sudo -S -k /opt/splunkforwarder/bin/splunk enable boot-start -user $username 2>/dev/null echo $password | sudo -S -k /opt/splunkforwarder/bin/splunk start 2>/dev/null echo 'Please wait 5 second....' sleep 5 echo $password | sudo -S -k sudo -u $username /opt/splunkforwarder/bin/splunk add forward-server 172.16.30.104:9997 -auth 'admin':'!@#qweasdZXC' 2>/dev/null echo $password | sudo -S -k sudo -u $username /opt/splunkforwarder/bin/splunk set deploy-poll 172.16.30.104:8089 -auth 'admin':'!@#qweasdZXC' 2>/dev/null " echo "In 5 seconds, will run the following script on each remote host:" echo sleep 5 echo "Reading host IPs from $HOSTS_FILE" echo echo "Starting." for DST in `cat "$HOSTS_FILE"`; do if [ -z "$DST" ]; then continue; fi echo "---------------------------------" | tee -a $STATUS_LOG echo "Starting work with $DST" | tee -a $STATUS_LOG sshpass -p $password ssh -q $username@$DST [[ -f /opt/splunkforwarder/bin/splunk ]] && INSTALLED=True || INSTALLED=False if [ "$INSTALLED" = "True" ]; then echo "Splunk UF is already installed" | tee -a $STATUS_LOG version=$(sshpass -p $password ssh $username@$DST "echo $password | sudo -S -k /opt/splunkforwarder/bin/splunk version | grep 'Splunk Universal Forwarder'" 2>/dev/null) echo "Splunk UF version: $version" | tee -a $STATUS_LOG status=$(sshpass -p $password ssh $username@$DST "echo $password | sudo -S -k /opt/splunkforwarder/bin/splunk status | grep 'splunkd is '" 2>/dev/null) echo "Splunk UF status: $status" | tee -a $STATUS_LOG dep=$(sshpass -p $password ssh $username@$DST "echo $password | sudo -S -k cat /opt/splunkforwarder/etc/system/local/deploymentclient.conf | grep '172.16.30.104:8089'" 2>/dev/null) fwd=$(sshpass -p $password ssh $username@$DST "echo $password | sudo -S -k cat /opt/splunkforwarder/etc/system/local/outputs.conf | grep '172.16.30.104:9997'" 2>/dev/null) if [ -z "$dep" ]; then echo "Deployment server is not configured" | tee -a $STATUS_LOG else echo "Deployment server is configured" | tee -a $STATUS_LOG fi if [ -z "$fwd" ]; then echo "Forward server is not configured" | tee -a $STATUS_LOG else echo "Forward server is configured" | tee -a $STATUS_LOG fi INSTALLED=False else echo "Splunk UF is not installed to host $DST" | tee -a $STATUS_LOG echo "Installing..." | tee -a $STATUS_LOG sshpass -p $password ssh $username@$DST "$PREPARE_COMMANDS" sshpass -p $password scp $INSTALL_FILE $username@$DST:/opt/splunkforwarder sshpass -p $password ssh $username@$DST "$INSTALL_COMMANDS" echo "Installation is done" | tee -a $STATUS_LOG echo "Checking..." | tee -a $STATUS_LOG status=$(sshpass -p $password ssh $username@$DST "echo $password | sudo -S -k /opt/splunkforwarder/bin/splunk status | grep 'splunkd is '" 2>/dev/null) echo "Splunk UF status: $status" | tee -a $STATUS_LOG dep=$(sshpass -p $password ssh $username@$DST "echo $password | sudo -S -k cat /opt/splunkforwarder/etc/system/local/deploymentclient.conf | grep '172.16.30.104:8089'" 2>/dev/null) fwd=$(sshpass -p $password ssh $username@$DST "echo $password | sudo -S -k cat /opt/splunkforwarder/etc/system/local/outputs.conf | grep '172.16.30.104:9997'" 2>/dev/null) if [ -z "$dep" ]; then echo "Deployment server is not configured" | tee -a $STATUS_LOG else echo "Deployment server is configured" | tee -a $STATUS_LOG fi if [ -z "$fwd" ]; then echo "Forward server is not configured" | tee -a $STATUS_LOG else echo "Forward server is configured" | tee -a $STATUS_LOG fi fi echo "---------------------------------" | tee -a $STATUS_LOG done echo "Done"   ... View more

Hi all! My question is, why splunk can't run a scripted input on Windows even though it has full access to that script? Entry of inputs.conf:     [script://.\\bin\deploy.bat] disabled = 0 interval = 300 index=windows [script://.\\bin\update.bat] disabled = 0 interval = 60 index=windows     These stanzas are responsible for installing and updating Sysmon on Windows servers and hosts. I deploy this configuration by deployment server to UFs. Splunk UFs ran by domain account. This account isn't admin. Here, there is full access: Here, as this user is running: But in the logs I see that access is denied: Please help me figure out why this is happening and how to solve it!   ... View more