topic Re: How can we index complete BSM log data? in Getting Data In

How can we index complete BSM log data?

Hemnaath — Tue, 29 Sep 2020 16:47:11 GMT

Hi All, Currently we are facing an issue in getting the complete BSM logs data in to splunk.

We have two remote host test01 and test02, test01 is running with older operating system (Solaris 9), so it's an old BSM version. Whereas test02 is running with Solaris 10.

Problem Detail: When user is trying to fetch an login information for a particular user id in splunk console, by executing the simple query "index=unix host=test01 sourcetype="unix:host:bsm" 007". Getting no result found.

But we could see the user id information present in the actual BSM.logs from the remote node test01.

At the same time, when same search query is executed with the host=test02, we could get the particular user id 007 information in splunk console.

index=unix host="test02" sourcetype="unix:host:bsm" 007

11/13/17
11:59:09.635 AM
2017-11-13 11:59:09.635 -05:00 zone=global event="logout" audit-uid=007 uid=007 tid="11911 196630 10.151.225.181" sid="4137709539"
eventtype = nix-all-logs host = test02 source = /var/bsm/20171113.bsm.log sourcetype = unix:host:bsm

When investigated the issue I could notice that both log "/var/bsm/20171113.bsm.log" data are in different format.

In test01 under /var/bsm/20171113.bsm.log
event="login - ssh" audit-uid=007 uid=007 tid="15615 22 test01.xxx.com" sid="4094245480" retval="0"
event="logout" audit-uid=007 uid=007 tid="15615 22 test01.xxx.com" sid="4094245480"
xsl_error="no_XSL_match" event="rsh access"
xsl_error="no_XSL_match" event="rsh access"

In test02 under /var/bsm/20171113.bsm.log

2017-11-13 08:14:49.826 -05:00 zone=global event="execve(2)" uid=root ruid=007 path=/usr/sbin/usr/lib/fs/ufs/quota retval=0 args="/usr/sbin/quota" tid="6323 196630 host.xxx.com" pid=622 sid="1060063057"
2017-11-13 08:14:49.851 -05:00 zone=global event="execve(2)" uid=007 ruid=007 path=/usr/bin/cat retval=0 args="/bin/cat -s /etc/motd" tid="6323 196630 host.xxx.com" pid=623 sid="1060063057"

User has raised a concern why the UID=007 is not being captured in the splunk from the node test01 and he wants this information to be captured in test01 server as well.

So kindly let me know how to fix this issue.

thanks in advance.

Re: How can we index complete BSM log data?

HiroshiSatoh — Tue, 14 Nov 2017 12:27:56 GMT

Because timestamp does not exist in the log, I think that _time is the capture time.
I think that it can be retrieved with a delay of a few seconds if it is captured in real time.

How Splunk software assigns timestamps
http://docs.splunk.com/Documentation/SplunkCloud/6.6.3/Data/HowSplunkextractstimestamps

※It is best if you can add a timestamp to the original(Solaris 9) log.

Re: How can we index complete BSM log data?

Hemnaath — Tue, 29 Sep 2020 16:47:14 GMT

Hi Hiroshisatoh, thanks for your effort on this, I am not sure whether its possible to add the time stamp in the actual logs. so could you please let me know how to add the time stamp via splunk, actually we have the below inputs.conf stanza configured for almost 400 + nodes.

[monitor:///var/bsm]
sourcetype = unix:host:bsm
crcSalt =
index = unix
disabled = 0

And also tried to execute the below query in real time All Time but could not fetch the output.
"index=unix host=test01 sourcetype="unix:host:bsm" 007"

var/bsm/20171113.bsm.log : Log Format with out time stamp.

event="ftp logout" xsl_error="no_XSL_match" event="ftp access" event="su" audit-uid=root uid=root text="success for user mqm" tid="9195 131094 test02.xxx.com" sid="59616287 xsl_error="no_XSL_match" event="ftp logout" xsl_error="no_XSL_match" event="ftp access" xsl_error="no_XSL_match" event="ftp access"xsl_error="no_XSL_match"event="ftp logout" event="login - ssh" audit-uid=007 uid=007 tid="15615 22 test02.xxx.com" sid="4094245480" retval="0" event="logout" audit-uid=007 uid=007 tid="15615 22 test02.xxx.com" sid="4094245480" xsl_error="no_XSL_match" event="rsh access" xsl_error="no_XSL_match"

Kindly guide me on how to add the time stamp in the events.

Re: How can we index complete BSM log data?

HiroshiSatoh — Tue, 29 Sep 2020 16:47:17 GMT

The timestamp is set by default.

As a last resort, Splunk software sets the timestamp to the current system time when indexing each event.

Please check _time and _indextime.
index=unix host=test01 sourcetype="unix:host:bsm" "uid=007"
| eval indextime=strftime(_indextime,"%Y-%m-%d %H:%M:%S")

Re: How can we index complete BSM log data?

Hemnaath — Tue, 14 Nov 2017 16:03:14 GMT

Hi Hiroshisatoh, thanks for your effort on this, Hey I had executed the above query but could not fetch any output.

index=unix host=test01 sourcetype="unix:host:bsm" "uid=007" | eval indextime=strftime(_indextime,"%Y-%m-%d %H:%M:%S") | table _time indextime

When executed without the uid=007 getting an output.

Kindly guide me how to set a timestamp when splunk reads the data from the source .

Re: How can we index complete BSM log data?

HiroshiSatoh — Wed, 15 Nov 2017 04:38:35 GMT

First of all, please check what time is in _time.

Splunk sets the timestamp by default.
If there is no time stamp in the log, the system time is set.

If you have a time stamp you want to set separately, present the sample log. There is no time stamp in the current sample log.

Re: How can we index complete BSM log data?

Hemnaath — Tue, 29 Sep 2020 16:48:11 GMT

Hi Hiroshisatoh, I had executed the below query to get the _time details

index=unix host="test01" sourcetype="unix:host:bsm" | table _time source host uid

11/15/2017 2:30 /var/bsm/20171115.bsm.log test01 solarwinds
11/15/2017 2:30 /var/bsm/20171115.bsm.log test01 solarwinds
11/15/2017 2:30 /var/bsm/20171115.bsm.log test01 mqm
11/15/2017 2:30 /var/bsm/20171115.bsm.log test01 solarwinds
11/15/2017 2:30 /var/bsm/20171115.bsm.log test01 solarwinds
11/15/2017 2:30 /var/bsm/20171115.bsm.log test01 solarwinds
11/15/2017 2:30 /var/bsm/20171115.bsm.log test01 solarwinds
11/15/2017 2:30 /var/bsm/20171115.bsm.log test01 solarwinds
11/15/2017 4:30 /var/bsm/20171115.bsm.log test01 mercator
11/15/2017 4:30 /var/bsm/20171115.bsm.log test01 mercator
11/15/2017 4:30 /var/bsm/20171115.bsm.log test01 solarwinds
11/15/2017 4:30 /var/bsm/20171115.bsm.log test01 solarwinds
11/15/2017 4:30 /var/bsm/20171115.bsm.log test01 root
11/15/2017 1:30 /var/bsm/20171115.bsm.log test01 mercator
11/15/2017 1:30 /var/bsm/20171115.bsm.log test01 mercator
11/15/2017 1:30 /var/bsm/20171115.bsm.log test01 mercator
11/15/2017 1:30 /var/bsm/20171115.bsm.log test01 mercator

From the above result above, when we run the query with the time frame set for 4 hours, we could see hardly few events with interval of 1 hour time difference between the each events in splunk.
System time was Wed Nov 15 05:26:26 EST 2017 when this query executed.

"Actual Log from the source file /var/bsm/2017115.bsm.log"

xsl_error="no_XSL_match" event="ftp access"
xsl_error="no_XSL_match" event="ftp logout"
xsl_error="no_XSL_match" event="ftp access"
xsl_error="no_XSL_match" event="ftp access"
xsl_error="no_XSL_match" event="ftp logout"
xsl_error="no_XSL_match" event="ftp access"
xsl_error="no_XSL_match" event="ftp logout"
xsl_error="no_XSL_match" event="ftp logout"
event="su" audit-uid=root uid=root text="success for user mercator" tid="9195 131094 test02.xxx.com" sid="596162876"
xsl_error="no_XSL_match" event="ftp access"
xsl_error="no_XSL_match" event="ftp logout"
xsl_error="no_XSL_match" event="ftp access"
event="su" audit-uid=root uid=root text="success for user mqm" tid="9195 131094 test02.xxx.com" sid="596162876"
xsl_error="no_XSL_match" event="ftp logout"
xsl_error="no_XSL_match" event="ftp access"
xsl_error="no_XSL_match" event="ftp logout"
event="su" audit-uid=root uid=root text="success for user mercator" tid="9195 131094 test02.xxx.com" sid="596162876"
event="su" audit-uid=root uid=root text="success for user mercator" tid="9195 131094 test02.xxx.com" sid="596162876"
event="su" audit-uid=root uid=root text="success for user mercator" tid="9195 131094 test02.xxx.com" sid="596162876"
event="su" audit-uid=root uid=root text="success for user mercator" tid="9195 131094 test02.xxx.com" sid="596162876"
xsl_error="no_XSL_match" event="ftp access"
event="su" audit-uid=root uid=root text="success for user mercator" tid="9195 131094 test02.xxx.com" sid="596162876"
event="login - ssh" audit-uid=mercator uid=mercator tid="14610 22 host01.xxx.com" sid="2978615533" retval="0"
event="su" audit-uid=root uid=root text="success for user mercator" tid="9195 131094 test02.xxx.com" sid="596162876"
event="su" audit-uid=root uid=root text="success for user mercator" tid="9195 131094 test02.xxx.com" sid="596162876"
xsl_error="no_XSL_match" event="ftp access"
event="logout" audit-uid=mercator uid=mercator tid="14610 22 host01.xxx.com" sid="2978615533"
xsl_error="no_XSL_match" event="ftp logout"
xsl_error="no_XSL_match" event="ftp logout"
event="login - ssh" audit-uid=mercator uid=mercator tid="14611 22 host01.xxx.com" sid="2083583931" retval="0"
xsl_error="no_XSL_match" event="ftp access"
event="logout" audit-uid=mercator uid=mercator tid="14611 22 host01.xxx.com" sid="2083583931"
xsl_error="no_XSL_match" event="ftp access"
xsl_error="no_XSL_match" event="ftp logout"
xsl_error="no_XSL_match" event="ftp logout"

Note : Attached partial logs in this comment, but most of the content in the
2017115.bsm.log is the same format.

Kindly guide me on how to add the time stamp in the events.

Re: How can we index complete BSM log data?

HiroshiSatoh — Tue, 29 Sep 2020 16:48:51 GMT

11/15/2017 2:30
11/15/2017 4:30
11/15/2017 1:30
This time is the time Splunk captured the log. Please check it against _indextime.
| eval indextime=strftime(_indextime,"%Y-%m-%d %H:%M:%S")

If there is no time stamp in the log, use the time taken in Splunk as the time stamp.

This time is valid when the log is captured in real time, but it can not be used if it is taken in periodically.

This log seems to be acquired periodically. Please check how logs are generated.

＞Kindly guide me on how to add the time stamp in the events.
I would like to teach but if there is no time stamp in the log, I can not set it.
If the default setting can not be used, it is necessary to change the log generation method.

Re: How can we index complete BSM log data?

Hemnaath — Tue, 29 Sep 2020 16:49:13 GMT

Hi Hiroshisatoh, thanks for your effort on this, I have asked the application owner to change the log generation method, but I had come across this stanza, in case if there is no data, time in the actual log we can configure the below stanza and pull the entire logs.
But I am really confused as the actual logs are multi line and I am not sure where the new events starts and ends in the actual logs.

Props.conf

[host::test01]
DATETIME_CONFIG = CURRENT
SHOULD_LINEMERGE = true
MUST_BREAK_AFTER =
MUST_NOT_BREAK_AFTER =

Can I use this stanza to get entire log in splunk.

Re: How can we index complete BSM log data?

HiroshiSatoh — Fri, 17 Nov 2017 05:50:51 GMT

When a timestamp is added to the log, Splunk splits the log with a timestamp.

If you want to combine multiple timestamp rows into one event, you must set the condition as a regular expression.

Can I present conditions for dividing events?

Re: How can we index complete BSM log data?

Hemnaath — Fri, 17 Nov 2017 15:51:31 GMT

Hi Hiroshisatoh, I could not understand the above comment. Anyway I had requested the application owner to change the log generation method in such a way to include the time stamp in the actual logs.

thanks for your effort.