All Apps and Add-ons

Splunking Command Line History in Linux

ejaypaz
New Member

Hi there!

I am currently indexing OS data from my Linux data sources using the Universal Forwarder with the TA (Technology Add-on). One of the things I would like to monitor is shell command history (i.e. .bash_history, .sh_history). I have the following stanzas in my inputs.conf files:

### Bash History
[monitor:///root/.bash_history]
disabled=0
sourcetype=bash_history
index=os

[monitor:///home/.../.bash_history]
disabled=0
sourcetype=bash_history
index=os

### KSH History
[monitor:///.../.sh_history]
disabled=0
sourcetype=ksh_history
index=os

Based on initial indexed data I'm receiving, however, every change to the history files triggers the forwarder to send the entire contents of the files. The history files currently don't timestamp each command record. What I wanted to know is would it be enough to just add the following lines in my stanzas if I was only interested in recording the day's commands?

followTail=1
ignoreOlderThan=1d

Hope to hear from the experts out there soon. Thanks!

0 Karma

araitz
Splunk Employee
Splunk Employee

It turns out that there are quite a few problems with monitoring the bash history in this way. Aside from the issues you mentioned, bash history is often not flushed to disk until the user logs off or the session ends, which gives the user plenty of opportunities to clear history.

To answer your question, you could try setting followTail and using DATETIME_CONFIG=CURRENT and SHOULD_LINEMERGE=False to break up each line and assign each the time that the indexer received it, but this would be highly inaccurate and would suffer from data integrity issues as I mentioned above.

0 Karma

araitz
Splunk Employee
Splunk Employee

There are some hits in google about how to modify history logging to use syslog, but I've never tried the approaches described therein. It seems like it would need to be some daemon or script that receives command input and sends it to Splunk or a log file in real time.

0 Karma

ejaypaz
New Member

@araitz, you're right.

I'm actually running into a number of problems right now, which include search failures (i.e. can't complete search because of more than 1000000 events found) and increased memory utilization for the splunkd process in one of my servers.

Would you have a more elegant, less problematic way to monitor both bash history and "ksh" history (we use korn shell for custom users and majority of tasks are done through this)?

0 Karma
Get Updates on the Splunk Community!

Observability Highlights | January 2023 Newsletter

 January 2023New Product Releases Splunk Network Explorer for Infrastructure MonitoringSplunk unveils Network ...

Security Highlights | January 2023 Newsletter

January 2023 Splunk Security Essentials (SSE) 3.7.0 ReleaseThe free Splunk Security Essentials (SSE) 3.7.0 app ...

Platform Highlights | January 2023 Newsletter

 January 2023Peace on Earth and Peace of Mind With Business ResilienceAll organizations can start the new year ...