All Apps and Add-ons

Splunking bash_history

Path Finder

Hi

I am trying to add the bash_history file but have runned into some problems.

  • The bash_history is timestamped (but not my very oldest entries)
  • The timestamp is placed above the command, in the format #12345678 (unix-time)
  • Old entries gets pushed out (after eg. 2000 entries).

Example entries:

cd /root/backup
ls -la
nano /root/.bashrc
exit
#1348736649
ls
#1348736654
cd /etc
#1348737978
ps aux

How can I make splunk index this nicely? And is there a way I can include both /root/.bash_history, and /home/.../.bash_history in the same stanza? Or even better, populate the files included from /etc/passwd (or a script).

1 Solution

Builder

lsolberg,
If you don't mind adding the entire file as an event, you can use the file change monitor with fullEvent=true. This will work around the problem with the oldest entries being pushed out.

[fschange:/home/*/.bash_history]
index=bash
pollPeriod=600
fullEvent=true
sourcetype=bash_history

Another option would be to set HISTFILESIZE to be very very big, so the entries won't be pushed out. However, you eventually you'll hit that limit and your users would have to clear the history and start a new one, maybe saving the old history to a different location. If you don't like this option, you'll probably need to create a scripted input.

As for the timestamps, the following settings in props.conf should help you, if you use monitor rather than fschange:

[bash_history]
BREAK_ONLY_BEFORE = #(?=\d+)
MAX_TIMESTAMP_LOOKAHEAD = 11
SHOULD_LINEMERGE = true
TIME_FORMAT = %s
TIME_PREFIX = #

HTH

View solution in original post

Explorer

Hello,

I used the following modification in /etc/profile

PROMPT_COMMAND='logger -i -p local5.info -t bash "$USER $(tty): $(history 1)"'
or
trap 'logger -i -p local5.info -t bash "$USER $(tty): $(fc -ln -1)"' DEBUG

And rsyslog.d/shell.conf to store it on a common file for all users :

local5.* -/var/log/shell.log

My only issue is that I can't get sh or ksh commands with this.

any suggestions ?

Rgds

0 Karma

Path Finder

What I did:
Adding the following line:

PROMPT_COMMAND="history -a;$PROMPT_COMMAND";

to /etc/bashrc This causes the .bash_history file to be written to immediately after a command finishes, which makes each command its own entry with its own time stamp.

Entries like the ones below in splunk inputs.conf files will cause the .bash_history files to be indexed in splunk, thus keeping a record of commands run on linux servers.

[monitor:///root/.bash_history]
index = os
sourcetype = cmdhistory
source = root
disabled = 0
followTail = 1
ignoreOlderThan = 1d

For other accounts I want to monitory I add another stanza:

[monitor:///home/oracle/.bash_history]
index = os
sourcetype = cmdhistory
source = oracle
disabled = 0
followTail = 1
ignoreOlderThan = 1d

That works for me because I only have a small number of accounts that I want to monitor, but /etc/passwd is delimited, you could process it with a shell script that would add stanzas for each user.

Builder

lsolberg,
If you don't mind adding the entire file as an event, you can use the file change monitor with fullEvent=true. This will work around the problem with the oldest entries being pushed out.

[fschange:/home/*/.bash_history]
index=bash
pollPeriod=600
fullEvent=true
sourcetype=bash_history

Another option would be to set HISTFILESIZE to be very very big, so the entries won't be pushed out. However, you eventually you'll hit that limit and your users would have to clear the history and start a new one, maybe saving the old history to a different location. If you don't like this option, you'll probably need to create a scripted input.

As for the timestamps, the following settings in props.conf should help you, if you use monitor rather than fschange:

[bash_history]
BREAK_ONLY_BEFORE = #(?=\d+)
MAX_TIMESTAMP_LOOKAHEAD = 11
SHOULD_LINEMERGE = true
TIME_FORMAT = %s
TIME_PREFIX = #

HTH

View solution in original post

Builder

No worries. 🙂

I did some further playing around with your idea and found that these settings in .bashrc are nice:

HISTFILESIZE=1000000
HISTTIMEFORMAT='[%c] '
export HISTFILESIZE HISTTIMEFORMAT
export PROMPT_COMMAND='history -a'

The PROMPT_COMMAND setting will append to your history after each command, so it will end up in the index quicker. And makes it just a little bit harder for someone to cover up the tracks. Not impossible, mind you.

0 Karma

Path Finder

Thanks, I was able to create my own regex, but yours was more elegant.

Added this to props.conf

[bash_history]
EXTRACT-command = #\d+\n(?P<command>.*)$
0 Karma

Builder

Hi,
Using the field-extraction feature, you should be able to do what you want with a regex something like this:

#\d+\n(?P<command>.*)$

HTH!

0 Karma

Path Finder

Thanks! That worked just like advertised! I am using monitor, so I ended up using props.conf.
As a bonus-question, is there an easy way to create a field for the 2nd line in the event? I want a field named command, but I am not that familiar on how regex works with multiline in Splunk. How can I use re.MULTILINE?

0 Karma

Path Finder

Here's a scriptlet if that helps (it'll also grab sh_history, csh_history, etc):
It's not looking at /etc/passwd, but at the filesystem, which should work.


for file in $(find /home|grep -E "\.\w*sh_history$"; find /root|grep -E "\.\w*sh_history$")
do
echo "[monitor://$file]">>inputs.conf
echo "sourcetype = shell_history">>inputs.conf
done

Obviously I haven't answered your first question -- still thinking on that one.

0 Karma

Path Finder

I thought it was an easier more elegant way.. We are using puppet, so I think I am ending up with a custom fact generated on the server every hour or so, and then puppet can generate the inputs.conf based on that.

Thanks anyway 🙂

0 Karma