What's best practice for monitoring bash_history of all users in the system?

[templier]

Hello, all!
Maybe someone has set up tracking bash_history file from all users in /home/*/.bash_history

I experimented with fschange, but splunkforwarder don't send data to server.
Splunk user can access to read .bash_history files.

Can anybody help me with this question?
Thanks!

[PickleRick]

Monitoring .bash_history is one thing. But remember that just by monitoring what's getting written there does not give you history of users' activity. Just of what's written into .bash_history.

There are other ways of monitoring what users do if that's what you're after.

If you understand why bash history might be incomplete and unreliable however, be my guest.

[jackjack]

Hi PickleRick,

Great name haha. Can you recommend me the other ways you would monitor users?

These are servers in a data center so /var/log and .bash_history were the first two I was told to focus on but would love to bring a few more ideas to the table.

I was able to get the monitoring on .bash_history working. I think I just had to restart splunk or something. My system seems to have a lag but maybe that's normal. Idk I'm still learning 🙂 

Thanks,

jack

[michaelnorup]

How did you get the permissions to work?

Having to trouble allowing the splunk user to read all other users bash_history

[PickleRick]

It is a way of monitoring users, just not a very "complete" one. It's simply very easy to circumvent (by switching off history, changing shell or running commands from within other programs to name but a few ways).

There are different ways depending on what you want to achieve and how much you want to spend 😉 But seriously, the typical approach is to use auditd with a proper set of rules but this is way out of scope of this forum. But it's good to know that there is a nice app for analyzing auditd logs.

[woodcock · Accepted solution]

Use this:

[monitor:///home/*/.bash_history]
disabled = false
sourcetype = bash_history
index = bash_history

Neither followTail nor ignoreOlderThan will do what you expect them to.
But you can use followTail JUST ONCE when you first get it working and then IMMEDIATELY turn it off and restart Splunk on the forwarder. This will cause it to start at the end of the file, write this mark to the fishbucket database and then you will pick up new stuff after that. Definitely do not hardcode source because that is the only place that you can find out which user owns these events!

[jackjack]

Do you happen to know the required permissions settings to make this work?

I tried `sudo setfacl -R -m u:splunk:r-x /home` but I am seeing "Insufficient Permissions to read file" for some of the .bash_history files

[templier]

Yes, it work fine.
Many thanks for help!

[inventsekar]

bit not clear...
You/someone wanted to monitor /home/*/.bash_history, but UF is not sending the .bash_history? ---- is this is the question or something else?

Can you show us the inputs.conf setup for this bash history monitor?

[templier]

Hello, yes we want monitoring bash_history of all users on system.
My inputs.conf (on splunkforwarder):

[fschange:/home/*/.bash_history]
index=bash_history
pollPeriod=600
fullEvent=true
sourcetype=bash_history

Splunk user can read .bash_history file of users, but nothing on my server.
If i modify inputs.conf (on splunkforwarder) for specific .bash_history:

[monitor:///home/user/.bash_history]
disabled = false
source = user
sourcetype = bash_history
followTail = 1
ignoreOlderThan = 1d
index = bash_history

All work fine.
But it is necessary that you do not have to manually add records for each user in inputs.conf on splunkforwarder.

[inventsekar]

may I know, why you are using fschange monitor, why not simply a directory monitor?!?

On a quick thought, let's check this....
Not tested this, please cross verify -

[monitor:///home/*/.bash_history]
disabled = false
source = user
sourcetype = bash_history
followTail = 1
ignoreOlderThan = 1d
index = bash_history

[templier]

Tested this - not working.

About using fschange:
This was one of the assumptions, if there is another option - I will be glad to hear it.
i don't know how monitor few directory, now i have only 3 users:
/home/user1/.bash_history
/home/user2/.bash_history
/home/user3/.bash_history

But In the future, there may be several more new users

[robert_miller]

I am surprised this didn't work because the Add-On for Unix and Linux have the lines below in Splunk_TA_nix/default/inputs.conf. These stanzas work for us but we do see odd behavior where it doesn't know how to line break the results which causes some of our events to be 257 lines.

bash history

[monitor:///root/.bash_history]
disabled = true
sourcetype = bash_history

[monitor:///home/*/.bash_history]
disabled = true
sourcetype = bash_history