How can i monitor linux commands in splunk

[kalpesh11]

I have 2 servers in place (linux), and would like to monitor the command hit at putty. Is there any way where we can do this?
Also what is the file path were all these logs get stored so that i can monitor it. (I have splunk app and add-on for unix and linux)
In simple language, whatever im doing in putty in want to monitor that.

[codebuilder]

If you add the following lines to bash_profile or bashrc, it will cause Linux to log all commands as they are executed, rather than writing them out when the user logs off (the default behavior). This will allow you to capture commands in real time, and also avoid the potential HISTFILESIZE=0 scenario.

shopt -s histappend
PROMPT_COMMAND="history -a;$PROMPT_COMMAND"

[vliggio]

True, though if someone is going to be smart enough to know HISTFILESIZE=0, they are probably smart enough to change their prompt and shell options :). Always a challenge to keep ahead of the nefarious!

All goes back to the original poster's question - is this for just general command tracking, or for actual security concerns. Relying on bash history for security purposes is risky. That's why running screen or sudo-io would be preferred in security situations, as the end user cannot override those.

[vliggio]

Capturing bash history isn't quite a correct answer. Any user can set their HISTFILESIZE environment variable to 0, and then their history will not be saved. You need to force user sessions to occur within something like the "screen" application to capture their session, if you want to guarantee from a security standpoint that you're capturing all their input.

If you're looking to see who runs privileged commands only via sudo, then you can use something like sudi-io to capture all commands issued via sudo (either individually or even if someone sudo's into a shell).

Note - if you are capturing the history file (a good thing to do irrespective of my comment above), set the following environment variable in your system defaults, which will put time in your history file (otherwise Splunk won't have accurate times that the commands were executed).

export HISTTIMEFORMAT="%h/%d -- %H:%M:%S "

[nickhills]

You raise an interesting point about user trust, however, the original question was about recording commands run on the terminal and bash history collection addresses this need.

However in the context that you do not trust your users and are fearful that they may take steps to prevent such recording, then that is a requirement which Splunk does not address on its own.
I have worked in organisations where history files are stored outside of the users home directory, and settings are enforced and user modification of the file is also impossible (I do not have specifics on how this was achieved, i just was responsible for collecting them 🙂 )

Your note on adding timestamps is good advice though!

[nickhills]

Use the Splunk Unix Add-On
https://splunkbase.splunk.com/app/833/

Configure it to collect bash history, which will record all the commands run via ssh.
You will need to configure ACLs to allow you to read all users bash history on the OS, unless you run Splunk as root (which is ill advised)

[kalpesh11]

Thanks for answer. I configured the input to collect bash history and its working.
For ACL, could you please share more on this?

[nickhills]

Well it very much depends on your OS and distribution.
Your windows or linux admins will be able to advise best how to configure this in your environment.

On rpm linux i know this works, but check with local expertise that this wont cause complications or be overwritten and for advice on a way to automate it

sudo setfacl -Rm g:yourSplunkUserGroup:rx,d:g:yourSplunkUserGroup:rx /root/.bash_history
sudo setfacl -Rm g:yourSplunkUserGroup:rx,d:g:yourSplunkUserGroup:rx /home/username/.bash_history