Deployment Architecture
Highlighted

How to extract time from bash_history # timestamp?

Explorer

I'm dealing with bash_history files in the following format. I would like to extract the timestamp and use that as the event timestamp, but I'm having some issues doing so.

#1579207583
whoami
#1579207584
cd /var/log
#1579207590
cat messages
#1579207595
id
#1579207598
exit

I'm using the following thread as reference: https://answers.splunk.com/answers/60015/splunking-bash-history.html

 [bash_history]
 BREAK_ONLY_BEFORE = #(?=\d+)
 MAX_TIMESTAMP_LOOKAHEAD = 11
 SHOULD_LINEMERGE = true
 TIME_FORMAT = %s
 TIME_PREFIX = #

We've changed a number of variables (set TIMEPREFIX = ^#, set MAXTIMESTAMP_LOOKAHEAD to a higher value, etc.), but nothing seems to be working correctly.

The events do break in the correct place (#), and they do merge, so we get "groups" of events like:

#1579207583
 whoami

However, the timestamp for the event isn't set to that value. All events are set to the date/time that history was written on, so everything for any given session is the same.

That props.conf configuration -appears- correct, and our sourcetype is named bashhistory (we've also tried source::/root/.bashhistory, without success). I'm not sure where we are going wrong, but any suggestions would be welcome.

0 Karma
Highlighted

Re: How to extract time from bash_history # timestamp?

Communicator

Can you check the errors and warning you are receiving for date time parsing on the receiving SPLUNK instance

0 Karma
Highlighted

Re: How to extract time from bash_history # timestamp?

Explorer

After looking in a few logs where I would expect and error to be (if there was one) I did a grep of -all- logs in /opt/splunk/var/log/splunk/ for "bash" and found nothing. Is there a specific log and/or keyword you know to check for?

0 Karma
Highlighted

Re: How to extract time from bash_history # timestamp?

Path Finder

Where did you place your props.conf ?

0 Karma
Highlighted

Re: How to extract time from bash_history # timestamp?

Explorer

It was deployed from the deployment server within the SplunkTAnix app to the UF's (so /opt/splunk/etc/deployment-apps/SplunkTAnix/local/)

0 Karma
Highlighted

Re: How to extract time from bash_history # timestamp?

SplunkTrust
SplunkTrust

I wonder if you replaced your entire props config as posted with just the below if this would cover both the line breaking and the timestamping? Maybe test and let me know?

[bash_history]
LINE_BREAKER = (^\#)\d+

0 Karma
Highlighted

Re: How to extract time from bash_history # timestamp?

Explorer

No luck, it's breaking... weird. So one event comes in as

hi this is a text
#1579273320
exit

And the previous one as:

1579273315

(the timestamp minux the #). It appears to alternate like this. Neither appears to be actually using this as the timestamp for the event though.

0 Karma
Highlighted

Re: How to extract time from bash_history # timestamp?

Path Finder

Try

 # props.conf
 [bash_history]
 # define event breaking behavior
 LINE_BREAKER = ([\r\n]+)\#\d+
 SHOULD_LINEMERGE = false

 # define time parsing behavior
 TIME_PREFIX = #
 TIME_FORMAT = %s
 MAX_TIMESTAMP_LOOKAHEAD = 12
0 Karma
Highlighted

Re: How to extract time from bash_history # timestamp?

Explorer

No luck, it appears to be line breaking at the correct place, as my original props.conf did. However, it's still not parsing the timestamp.

0 Karma
Highlighted

Re: How to extract time from bash_history # timestamp?

Ultra Champion
| makeresults
| eval _raw="#1579207583
whoami
#1579207584
cd /var/log
#1579207590
cat messages
#1579207595
id
#1579207598
exit"
 `comment("this is sample you provide")`
| rex max_match=100 "(?:#)(?<time>\w+)"
| rex max_match=100 "(?m)^(?=[^#])(?<command>.+)$"
| eval tmp=mvzip(time,command)
| stats count by tmp
| eval _time=mvindex(split(tmp,","),0), command=mvindex(split(tmp,","),1)
| table _time command

If props.conf doesn't work, you can extract it with this query.

0 Karma
Speak Up for Splunk Careers!

We want to better understand the impact Splunk experience and expertise has has on individuals' careers, and help highlight the growing demand for Splunk skills.