Hello, I am trying to collect bash_history logs in real-time from multiple Linux hosts using Splunk. I have deployed the following script to append executed commands to /var/log/bash_history.log: #!/bin/bash LOG_FILE="/var/log/bash_history.log" PROMPT_COMMAND_STR='export PROMPT_COMMAND='\''RECORD_CMD=$(history 1 | sed "s/^[ ]*[0-9]*[ ]*//"); echo "$(date "+%Y-%m-%d %H:%M:%S") $(whoami) $RECORD_CMD" >> /var/log/bash_history.log'\''' # 1. Create log file if it doesn't exist and set permissions if [ ! -f "$LOG_FILE" ]; then touch "$LOG_FILE" echo "[INFO] Log file created: $LOG_FILE" fi chmod 666 "$LOG_FILE" chown root:users "$LOG_FILE" echo "[INFO] Log file permissions set" # 2. Add PROMPT_COMMAND to /etc/bash.bashrc if ! grep -q "PROMPT_COMMAND" /etc/bash.bashrc; then echo "$PROMPT_COMMAND_STR" >> /etc/bash.bashrc echo "[INFO] PROMPT_COMMAND added to /etc/bash.bashrc" fi # 3. Force loading of ~/.bashrc through /etc/profile if ! grep -q "source ~/.bashrc" /etc/profile; then echo 'if [ -f ~/.bashrc ]; then source ~/.bashrc; fi' >> /etc/profile echo "[INFO] ~/.bashrc now loads via /etc/profile" fi # 4. Add PROMPT_COMMAND to all users' ~/.bashrc and ~/.profile for user in $(ls /home); do for FILE in "/home/$user/.bashrc" "/home/$user/.profile"; do if [ -f "$FILE" ] && ! grep -q "PROMPT_COMMAND" "$FILE"; then echo "$PROMPT_COMMAND_STR" >> "$FILE" echo "[INFO] PROMPT_COMMAND added to $FILE (user: $user)" fi done done # 5. Add PROMPT_COMMAND for root user for FILE in "/root/.bashrc" "/root/.profile"; do if [ -f "$FILE" ] && ! grep -q "PROMPT_COMMAND" "$FILE"; then echo "$PROMPT_COMMAND_STR" >> "$FILE" echo "[INFO] PROMPT_COMMAND added to $FILE (root)" fi done # 6. Ensure ~/.bashrc is sourced in ~/.profile for all users for user in $(ls /home); do PROFILE_FILE="/home/$user/.profile" if [ -f "$PROFILE_FILE" ] && ! grep -q ". ~/.bashrc" "$PROFILE_FILE"; then echo ". ~/.bashrc" >> "$PROFILE_FILE" echo "[INFO] ~/.bashrc now sources from ~/.profile (user: $user)" fi done # 7. Ensure all users use Bash shell while IFS=: read -r username _ _ _ _ home shell; do if [[ "$home" == /home/* || "$home" == "/root" ]]; then if [[ "$shell" != "/bin/bash" ]]; then echo "[WARNING] User $username has shell $shell, changing to Bash..." usermod --shell /bin/bash "$username" fi fi done < /etc/passwd # 8. Apply changes exec bash echo "[INFO] Configuration applied" The script runs correctly, and /var/log/bash_history.log is created on all hosts. However, Splunk is not collecting logs from all hosts. Some hosts send data properly, while others do not. What I have checked: Permissions on /var/log/bash_history.log → The file is writable by all users (chmod 666 and chown root:users). Presence of PROMPT_COMMAND in user sessions → When running echo $PROMPT_COMMAND, it appears correctly for most users. SU behavior → If users switch with su - username, it works. However, if they switch with su username, sometimes the logs are missing. Splunk Inputs Configuration: [monitor:///var/log/bash_history.log] disabled = false index = os sourcetype = bash_history This is properly deployed to all hosts. Questions: Could there be permission issues with writing to /var/log/bash_history.log under certain circumstances? Would another directory (e.g., /tmp/) be better? How can I ensure that all user sessions (including su username) log commands consistently? Could there be an issue with Splunk Universal Forwarder not properly monitoring /var/log/bash_history.log on some hosts? Any insights or best practices would be greatly appreciated! Thanks.
... View more