All Apps and Add-ons

What's best practice for monitoring bash_history of all users in the system?

templier
Communicator

Hello, all!
Maybe someone has set up tracking bash_history file from all users in /home/*/.bash_history

I experimented with fschange, but splunkforwarder don't send data to server.
Splunk user can access to read .bash_history files.

Can anybody help me with this question?
Thanks!

1 Solution

woodcock
Esteemed Legend

Use this:

[monitor:///home/*/.bash_history]
disabled = false
sourcetype = bash_history
index = bash_history

Neither followTail nor ignoreOlderThan will do what you expect them to.
But you can use followTail JUST ONCE when you first get it working and then IMMEDIATELY turn it off and restart Splunk on the forwarder. This will cause it to start at the end of the file, write this mark to the fishbucket database and then you will pick up new stuff after that. Definitely do not hardcode source because that is the only place that you can find out which user owns these events!

View solution in original post

PickleRick
SplunkTrust
SplunkTrust

Monitoring .bash_history is one thing. But remember that just by monitoring what's getting written there does not give you history of users' activity. Just of what's written into .bash_history.

There are other ways of monitoring what users do if that's what you're after.

If you understand why bash history might be incomplete and unreliable however, be my guest.

jackjack
Path Finder

Hi PickleRick,

Great name haha. Can you recommend me the other ways you would monitor users?

These are servers in a data center so /var/log and .bash_history were the first two I was told to focus on but would love to bring a few more ideas to the table.

I was able to get the monitoring on .bash_history working. I think I just had to restart splunk or something. My system seems to have a lag but maybe that's normal. Idk I'm still learning 🙂 

Thanks,

jack

0 Karma

michaelnorup
Communicator

How did you get the permissions to work?

Having to trouble allowing the splunk user to read all other users bash_history

0 Karma

PickleRick
SplunkTrust
SplunkTrust

It is a way of monitoring users, just not a very "complete" one. It's simply very easy to circumvent (by switching off history, changing shell or running commands from within other programs to name but a few ways).

There are different ways depending on what you want to achieve and how much you want to spend 😉 But seriously, the typical approach is to use auditd with a proper set of rules but this is way out of scope of this forum. But it's good to know that there is a nice app for analyzing auditd logs.

woodcock
Esteemed Legend

Use this:

[monitor:///home/*/.bash_history]
disabled = false
sourcetype = bash_history
index = bash_history

Neither followTail nor ignoreOlderThan will do what you expect them to.
But you can use followTail JUST ONCE when you first get it working and then IMMEDIATELY turn it off and restart Splunk on the forwarder. This will cause it to start at the end of the file, write this mark to the fishbucket database and then you will pick up new stuff after that. Definitely do not hardcode source because that is the only place that you can find out which user owns these events!

jackjack
Path Finder

Do you happen to know the required permissions settings to make this work?

I tried `sudo setfacl -R -m u:splunk:r-x /home` but I am seeing "Insufficient Permissions to read file" for some of the .bash_history files

0 Karma

templier
Communicator

Yes, it work fine.
Many thanks for help!

0 Karma

inventsekar
SplunkTrust
SplunkTrust

bit not clear...
You/someone wanted to monitor /home/*/.bash_history, but UF is not sending the .bash_history? ---- is this is the question or something else?

Can you show us the inputs.conf setup for this bash history monitor?

0 Karma

templier
Communicator

Hello, yes we want monitoring bash_history of all users on system.
My inputs.conf (on splunkforwarder):

[fschange:/home/*/.bash_history]
index=bash_history
pollPeriod=600
fullEvent=true
sourcetype=bash_history

Splunk user can read .bash_history file of users, but nothing on my server.
If i modify inputs.conf (on splunkforwarder) for specific .bash_history:

[monitor:///home/user/.bash_history]
disabled = false
source = user
sourcetype = bash_history
followTail = 1
ignoreOlderThan = 1d
index = bash_history

All work fine.
But it is necessary that you do not have to manually add records for each user in inputs.conf on splunkforwarder.

0 Karma

inventsekar
SplunkTrust
SplunkTrust

may I know, why you are using fschange monitor, why not simply a directory monitor?!?

On a quick thought, let's check this....
Not tested this, please cross verify -

[monitor:///home/*/.bash_history]
disabled = false
source = user
sourcetype = bash_history
followTail = 1
ignoreOlderThan = 1d
index = bash_history

0 Karma

templier
Communicator

Tested this - not working.

About using fschange:
This was one of the assumptions, if there is another option - I will be glad to hear it.
i don't know how monitor few directory, now i have only 3 users:
/home/user1/.bash_history
/home/user2/.bash_history
/home/user3/.bash_history

But In the future, there may be several more new users

0 Karma

robert_miller
Path Finder

I am surprised this didn't work because the Add-On for Unix and Linux have the lines below in Splunk_TA_nix/default/inputs.conf. These stanzas work for us but we do see odd behavior where it doesn't know how to line break the results which causes some of our events to be 257 lines.

bash history

[monitor:///root/.bash_history]
disabled = true
sourcetype = bash_history

[monitor:///home/*/.bash_history]
disabled = true
sourcetype = bash_history

0 Karma
Get Updates on the Splunk Community!

New in Observability - Improvements to Custom Metrics SLOs, Log Observer Connect & ...

The latest enhancements to the Splunk observability portfolio deliver improved SLO management accuracy, better ...

Improve Data Pipelines Using Splunk Data Management

  Register Now   This Tech Talk will explore the pipeline management offerings Edge Processor and Ingest ...

3-2-1 Go! How Fast Can You Debug Microservices with Observability Cloud?

Register Join this Tech Talk to learn how unique features like Service Centric Views, Tag Spotlight, and ...