I am currently indexing OS data from my Linux data sources using the Universal Forwarder with the TA (Technology Add-on). One of the things I would like to monitor is shell command history (i.e. .bash_history, .sh_history). I have the following stanzas in my inputs.conf files:
### Bash History
### KSH History
Based on initial indexed data I'm receiving, however, every change to the history files triggers the forwarder to send the entire contents of the files. The history files currently don't timestamp each command record. What I wanted to know is would it be enough to just add the following lines in my stanzas if I was only interested in recording the day's commands?
Hope to hear from the experts out there soon. Thanks!
It turns out that there are quite a few problems with monitoring the bash history in this way. Aside from the issues you mentioned, bash history is often not flushed to disk until the user logs off or the session ends, which gives the user plenty of opportunities to clear history.
To answer your question, you could try setting followTail and using DATETIME_CONFIG=CURRENT and SHOULD_LINEMERGE=False to break up each line and assign each the time that the indexer received it, but this would be highly inaccurate and would suffer from data integrity issues as I mentioned above.
There are some hits in google about how to modify history logging to use syslog, but I've never tried the approaches described therein. It seems like it would need to be some daemon or script that receives command input and sends it to Splunk or a log file in real time.
I'm actually running into a number of problems right now, which include search failures (i.e. can't complete search because of more than 1000000 events found) and increased memory utilization for the splunkd process in one of my servers.
Would you have a more elegant, less problematic way to monitor both bash history and "ksh" history (we use korn shell for custom users and majority of tasks are done through this)?