I'm dealing with bash_history files in the following format. I would like to extract the timestamp and use that as the event timestamp, but I'm having some issues doing so.
#1579207583 whoami #1579207584 cd /var/log #1579207590 cat messages #1579207595 id #1579207598 exit
I'm using the following thread as reference: https://answers.splunk.com/answers/60015/splunking-bash-history.html
[bash_history] BREAK_ONLY_BEFORE = #(?=\d+) MAX_TIMESTAMP_LOOKAHEAD = 11 SHOULD_LINEMERGE = true TIME_FORMAT = %s TIME_PREFIX = #
We've changed a number of variables (set TIMEPREFIX = ^#, set MAXTIMESTAMP_LOOKAHEAD to a higher value, etc.), but nothing seems to be working correctly.
The events do break in the correct place (#), and they do merge, so we get "groups" of events like:
However, the timestamp for the event isn't set to that value. All events are set to the date/time that history was written on, so everything for any given session is the same.
That props.conf configuration -appears- correct, and our sourcetype is named bashhistory (we've also tried source::/root/.bashhistory, without success). I'm not sure where we are going wrong, but any suggestions would be welcome.
Can you check the errors and warning you are receiving for date time parsing on the receiving SPLUNK instance
After looking in a few logs where I would expect and error to be (if there was one) I did a grep of -all- logs in /opt/splunk/var/log/splunk/ for "bash" and found nothing. Is there a specific log and/or keyword you know to check for?
It was deployed from the deployment server within the SplunkTAnix app to the UF's (so /opt/splunk/etc/deployment-apps/SplunkTAnix/local/)
I wonder if you replaced your entire props config as posted with just the below if this would cover both the line breaking and the timestamping? Maybe test and let me know?
LINE_BREAKER = (^\#)\d+
No luck, it's breaking... weird. So one event comes in as
hi this is a text #1579273320 exit
And the previous one as:
(the timestamp minux the #). It appears to alternate like this. Neither appears to be actually using this as the timestamp for the event though.
# props.conf [bash_history] # define event breaking behavior LINE_BREAKER = ([\r\n]+)\#\d+ SHOULD_LINEMERGE = false # define time parsing behavior TIME_PREFIX = # TIME_FORMAT = %s MAX_TIMESTAMP_LOOKAHEAD = 12
No luck, it appears to be line breaking at the correct place, as my original props.conf did. However, it's still not parsing the timestamp.
| makeresults | eval _raw="#1579207583 whoami #1579207584 cd /var/log #1579207590 cat messages #1579207595 id #1579207598 exit" `comment("this is sample you provide")` | rex max_match=100 "(?:#)(?<time>\w+)" | rex max_match=100 "(?m)^(?=[^#])(?<command>.+)$" | eval tmp=mvzip(time,command) | stats count by tmp | eval _time=mvindex(split(tmp,","),0), command=mvindex(split(tmp,","),1) | table _time command
props.conf doesn't work, you can extract it with this query.