Splunk Not Collecting bash_history Logs from All Hosts in Real-Time

[salikovsky]

Hello,

I am trying to collect bash_history logs in real-time from multiple Linux hosts using Splunk. I have deployed the following script to append executed commands to /var/log/bash_history.log:

#!/bin/bash

LOG_FILE="/var/log/bash_history.log"

PROMPT_COMMAND_STR='export PROMPT_COMMAND='\''RECORD_CMD=$(history 1 | sed "s/^[ ]*[0-9]*[ ]*//"); echo "$(date "+%Y-%m-%d %H:%M:%S") $(whoami) $RECORD_CMD" >> /var/log/bash_history.log'\'''

# 1. Create log file if it doesn't exist and set permissions

if [ ! -f "$LOG_FILE" ]; then

    touch "$LOG_FILE"

    echo "[INFO] Log file created: $LOG_FILE"

fi

chmod 666 "$LOG_FILE"

chown root:users "$LOG_FILE"

echo "[INFO] Log file permissions set"

# 2. Add PROMPT_COMMAND to /etc/bash.bashrc

if ! grep -q "PROMPT_COMMAND" /etc/bash.bashrc; then

    echo "$PROMPT_COMMAND_STR" >> /etc/bash.bashrc

    echo "[INFO] PROMPT_COMMAND added to /etc/bash.bashrc"

fi

# 3. Force loading of ~/.bashrc through /etc/profile

if ! grep -q "source ~/.bashrc" /etc/profile; then

    echo 'if [ -f ~/.bashrc ]; then source ~/.bashrc; fi' >> /etc/profile

    echo "[INFO] ~/.bashrc now loads via /etc/profile"

fi

# 4. Add PROMPT_COMMAND to all users' ~/.bashrc and ~/.profile

for user in $(ls /home); do

    for FILE in "/home/$user/.bashrc" "/home/$user/.profile"; do

        if [ -f "$FILE" ] && ! grep -q "PROMPT_COMMAND" "$FILE"; then

            echo "$PROMPT_COMMAND_STR" >> "$FILE"

            echo "[INFO] PROMPT_COMMAND added to $FILE (user: $user)"

        fi

    done

done

# 5. Add PROMPT_COMMAND for root user

for FILE in "/root/.bashrc" "/root/.profile"; do

    if [ -f "$FILE" ] && ! grep -q "PROMPT_COMMAND" "$FILE"; then

        echo "$PROMPT_COMMAND_STR" >> "$FILE"

        echo "[INFO] PROMPT_COMMAND added to $FILE (root)"

    fi

done

# 6. Ensure ~/.bashrc is sourced in ~/.profile for all users

for user in $(ls /home); do

    PROFILE_FILE="/home/$user/.profile"

    if [ -f "$PROFILE_FILE" ] && ! grep -q ". ~/.bashrc" "$PROFILE_FILE"; then

        echo ". ~/.bashrc" >> "$PROFILE_FILE"

        echo "[INFO] ~/.bashrc now sources from ~/.profile (user: $user)"

    fi

done

# 7. Ensure all users use Bash shell

while IFS=: read -r username _ _ _ _ home shell; do

    if [[ "$home" == /home/* || "$home" == "/root" ]]; then

        if [[ "$shell" != "/bin/bash" ]]; then

            echo "[WARNING] User $username has shell $shell, changing to Bash..."

            usermod --shell /bin/bash "$username"

        fi

    fi

done < /etc/passwd

# 8. Apply changes

exec bash

echo "[INFO] Configuration applied"

The script runs correctly, and /var/log/bash_history.log is created on all hosts. However, Splunk is not collecting logs from all hosts. Some hosts send data properly, while others do not.

What I have checked:

1. Permissions on /var/log/bash_history.log → The file is writable by all users (chmod 666 and chown root:users).
2. Presence of PROMPT_COMMAND in user sessions → When running echo $PROMPT_COMMAND, it appears correctly for most users.
3. SU behavior → If users switch with su - username, it works. However, if they switch with su username, sometimes the logs are missing.
4. Splunk Inputs Configuration:
   
   

[monitor:///var/log/bash_history.log]

disabled = false

index = os

sourcetype = bash_history

This is properly deployed to all hosts.

Questions:

1. Could there be permission issues with writing to /var/log/bash_history.log under certain circumstances? Would another directory (e.g., /tmp/) be better?
2. How can I ensure that all user sessions (including su username) log commands consistently?
3. Could there be an issue with Splunk Universal Forwarder not properly monitoring /var/log/bash_history.log on some hosts?
    

Any insights or best practices would be greatly appreciated!

Thanks.

[PickleRick]

I see what you're trying to do but be aware that whole process is flawed here.

1. You're relying on a file with 666 permissions. This way anyone can manipulate this file's contents in any way they see fit. They can remove contents, counting on race condition so that UF won't pick it up, they can inject any contents. And you're not able to tell whether it's legitimate or not.

2. The $PROMPT_COMMAND is only run at prompt time. Which means that when there is no prompt, the command is not run. And there is a lot of situations like that.

3. You're relying on bash being spawned as a shell and being the only shell for a user. That is not true. It's trivial to spawn any other shell or any other process in a non-tracked way.

4. You're also relying that startup scripts are run for a bash session. That also doesn't have to be true. (see your "su" case).

So if it's your way of providing accountability... that's not gonna work very well. For that you're gonna need other tools. For example very limited sudo configuration. (with sudo you have logging included). Or a whole user session monitoring tool but that's completely out of scope here.

If it's just so that you have some form of tracking what people are doing for future reference and to avoid situaitions like "how did we do that???", that might be a way to do so. In fact I'm doing a similar thing on my computers but I use logger in my $PROMPT_COMMAND so that it gets pushed to system-wide syslog. Yes, it also has some of the aforementioned issues but the log is a bit less easy to manipulate after it's been written to.

As a side remark - you have several mistakes in your script. For example, your grep -q will find _any_ PROMPT_COMMAND and even if it's commented out or being just part of an echoed string.

Possible issues with the file monitoring (

[salikovsky]

@PickleRick Thank you so much, I understand my mistakes. What methods would you recommend for collecting user-entered commands in real-time?

[PickleRick]

Unfortunately, monitoring users is not an easy task. There are huge challenges with it on every system I know - you either get too little data and it's easy to circumvent the auditing proces or you're getting way too much data and it's hard to make anything of it.

That's why there's a relatively big market for third-party solution - either agent-based or working as man-in-the-middle external component.

If you're ok with the limitations, you could use the PROMPT_COMMAND method (but again - I wouldn't use world-writeable files), add to it some auditd logging and you have... something.

Oh, and be very careful with using the term "realtime".

[livehybrid]

Hi @salikovsky 

Regarding SU Behaviour - when the expected logs are missing, are you able to confirm that they are present in the /var/log/bash_history.log file and only missing from Splunk.

Also, run the following search and check that each of the UFs is correctly configured to monitor the files:

index=_internal TailingProcessor "/var/log/bash_history.log"

You should see something like

02-21-2025 13:47:29.836 +0000 INFO  TailingProcessor [1354 MainTailingThread] - Parsing configuration stanza: monitor:///var/log/bash_history.log.

02-21-2025 13:47:29.836 +0000 INFO  TailingProcessor [1354 MainTailingThread] - Adding watch on path: /var/log/bash_history.log.

Please let me know how you get on and consider adding karma to this or any other answer if it has helped.
Regards

Will

[salikovsky]

@livehybrid  I saw successful result of creating the file /var/log/bash_history.log, but events from this file came from less than 10% of the hosts. I did not see any errors related to permissions or inability to view the file.

[kiran_panchavat]

@salikovsky 

• Check UF Logs: tail -f /opt/splunkforwarder/var/log/splunk/splunkd.log | grep -i bash_history
• File truncation, rotations, or buffering delays could be preventing logs from reaching Splunk.
• Run on affected hosts: /opt/splunkforwarder/bin/splunk list monitor

Ensure /var/log/bash_history.log is listed.