I'm dealing with bash_history files in the following format. I would like to extract the timestamp and use that as the event timestamp, but I'm having some issues doing so.
#1579207583
whoami
#1579207584
cd /var/log
#1579207590
cat messages
#1579207595
id
#1579207598
exit
I'm using the following thread as reference: https://answers.splunk.com/answers/60015/splunking-bash-history.html
[bash_history]
BREAK_ONLY_BEFORE = #(?=\d+)
MAX_TIMESTAMP_LOOKAHEAD = 11
SHOULD_LINEMERGE = true
TIME_FORMAT = %s
TIME_PREFIX = #
We've changed a number of variables (set TIME_PREFIX = ^#, set MAX_TIMESTAMP_LOOKAHEAD to a higher value, etc.), but nothing seems to be working correctly.
The events do break in the correct place (#), and they do merge, so we get "groups" of events like:
#1579207583
whoami
However, the timestamp for the event isn't set to that value. All events are set to the date/time that history was written on, so everything for any given session is the same.
That props.conf configuration -appears- correct, and our sourcetype is named bash_history (we've also tried source::/root/.bash_history, without success). I'm not sure where we are going wrong, but any suggestions would be welcome.
I figured it out. The "default/props.conf" in Splunk_TA_nix contains several lines that affect the timestamp. I copied these to "local/props.conf" and unset them (didn't provide a value), and now it's working. Final props.conf looks like...
[bash_history]
BREAK_ONLY_BEFORE = #(?=\d+)
MAX_TIMESTAMP_LOOKAHEAD = 10
SHOULD_LINEMERGE = true
TIME_FORMAT = %s
TIME_PREFIX = ^#
EVENT_BREAKER_ENABLE =
DATETIME_CONFIG =
I also added a field extraction for the command itself:
^#\d+\s+(?P<command>.+)
TL;DR - It was working from the beginning, but other values in default were affecting the final result.