Splunking bash_history

[lsolberg]

Hi

I am trying to add the bash_history file but have runned into some problems.

• The bash_history is timestamped (but not my very oldest entries)
• The timestamp is placed above the command, in the format #12345678 (unix-time)
• Old entries gets pushed out (after eg. 2000 entries).

Example entries:

cd /root/backup
ls -la
nano /root/.bashrc
exit
#1348736649
ls
#1348736654
cd /etc
#1348737978
ps aux

How can I make splunk index this nicely? And is there a way I can include both /root/.bash_history, and /home/.../.bash_history in the same stanza? Or even better, populate the files included from /etc/passwd (or a script).

[fulldanad]

Hello,

I used the following modification in /etc/profile

PROMPT_COMMAND='logger -i -p local5.info -t bash "$USER $(tty): $(history 1)"'
or
trap 'logger -i -p local5.info -t bash "$USER $(tty): $(fc -ln -1)"' DEBUG

And rsyslog.d/shell.conf to store it on a common file for all users :

local5.* -/var/log/shell.log

My only issue is that I can't get sh or ksh commands with this.

any suggestions ?

Rgds

[edonze]

What I did:
Adding the following line:

PROMPT_COMMAND="history -a;$PROMPT_COMMAND";

to /etc/bashrc This causes the .bash_history file to be written to immediately after a command finishes, which makes each command its own entry with its own time stamp.

Entries like the ones below in splunk inputs.conf files will cause the .bash_history files to be indexed in splunk, thus keeping a record of commands run on linux servers.

[monitor:///root/.bash_history]
index = os
sourcetype = cmdhistory
source = root
disabled = 0
followTail = 1
ignoreOlderThan = 1d

For other accounts I want to monitory I add another stanza:

[monitor:///home/oracle/.bash_history]
index = os
sourcetype = cmdhistory
source = oracle
disabled = 0
followTail = 1
ignoreOlderThan = 1d

That works for me because I only have a small number of accounts that I want to monitor, but /etc/passwd is delimited, you could process it with a shell script that would add stanzas for each user.

[echalex · Accepted solution]

lsolberg,
If you don't mind adding the entire file as an event, you can use the file change monitor with fullEvent=true. This will work around the problem with the oldest entries being pushed out.

[fschange:/home/*/.bash_history]
index=bash
pollPeriod=600
fullEvent=true
sourcetype=bash_history

Another option would be to set HISTFILESIZE to be very very big, so the entries won't be pushed out. However, you eventually you'll hit that limit and your users would have to clear the history and start a new one, maybe saving the old history to a different location. If you don't like this option, you'll probably need to create a scripted input.

As for the timestamps, the following settings in props.conf should help you, if you use monitor rather than fschange:

[bash_history]
BREAK_ONLY_BEFORE = #(?=\d+)
MAX_TIMESTAMP_LOOKAHEAD = 11
SHOULD_LINEMERGE = true
TIME_FORMAT = %s
TIME_PREFIX = #

HTH

[echalex]

No worries. 🙂

I did some further playing around with your idea and found that these settings in .bashrc are nice:

HISTFILESIZE=1000000
HISTTIMEFORMAT='[%c] '
export HISTFILESIZE HISTTIMEFORMAT
export PROMPT_COMMAND='history -a'

The PROMPT_COMMAND setting will append to your history after each command, so it will end up in the index quicker. And makes it just a little bit harder for someone to cover up the tracks. Not impossible, mind you.

[lsolberg]

Thanks, I was able to create my own regex, but yours was more elegant.

Added this to props.conf

[bash_history]
EXTRACT-command = #\d+\n(?P<command>.*)$

[echalex]

Hi,
Using the field-extraction feature, you should be able to do what you want with a regex something like this:

#\d+\n(?P<command>.*)$

HTH!

[lsolberg]

Thanks! That worked just like advertised! I am using monitor, so I ended up using props.conf.
As a bonus-question, is there an easy way to create a field for the 2nd line in the event? I want a field named command, but I am not that familiar on how regex works with multiline in Splunk. How can I use re.MULTILINE?

[glitchcowboy]

Here's a scriptlet if that helps (it'll also grab sh_history, csh_history, etc):
It's not looking at /etc/passwd, but at the filesystem, which should work.


for file in $(find /home|grep -E "\.\w*sh_history$"; find /root|grep -E "\.\w*sh_history$")
do
echo "[monitor://$file]">>inputs.conf
echo "sourcetype = shell_history">>inputs.conf
done


Obviously I haven't answered your first question -- still thinking on that one.

[lsolberg]

I thought it was an easier more elegant way.. We are using puppet, so I think I am ending up with a custom fact generated on the server every hour or so, and then puppet can generate the inputs.conf based on that.

Thanks anyway 🙂