How to configure a time stamp when the actual log data does not have date, time in it?

[Hemnaath]

Hi All, Currently I am facing an issue in getting the complete BSM logs data in to splunk. We have two remote host test01 and test02, test01 is running with older operating system (Solaris 9), so it's an old BSM version. Whereas test02 is running with Solaris 10.

On investigation we have identified that in test01 the actual log data does not have time stamp in it, so when we search with the below query, it gets very less events information.

index=unix host=test01 sourcetype="unix:host:bsm"

Example: when we run the query with the time frame set for 4 hours, we could see hardly 22 events with interval of 1 hour time difference between the each events in splunk.

11/14/17
8:30:01.000 AM

event="logout" audit-uid=mqm uid=mqm tid="13416 22 host05.xxx.com" sid="1266159348"
event="su" audit-uid=root uid=root text="success for user mercator" tid="9195 131094 test02.xxx.com" sid="596162876"
event="su" audit-uid=root uid=root text="success for user mercator" tid="9195 131094 test02.xxx.com" sid="596162876"
event="su" audit-uid=root uid=root text="success for user mercator" tid="9195 131094 test02.xxx.com" sid="596162876"
xsl_error="no_XSL_match" event="ftp access"
Show all 38 lines
eventtype = err0r error eventtype = nix-all-logs eventtype = nix_errors error host = test01 source = /var/bsm/20171114.bsm.log sourcetype = unix:host:bsm

11/14/17
7:30:03.000 AM

event="logout" audit-uid=solarwinds uid=solarwinds tid="12506 196630 host05.xxx.com" sid="1117761561"
event="login - ssh" audit-uid=solarwinds uid=solarwinds tid="12517 131094 host05.xxx.com" sid="2100926268" retval="0"
event="logout" audit-uid=solarwinds uid=solarwinds tid="12506 131094 host05.xxx.com" sid="1833882761"
eventtype = nix-all-logs host = test01 source = /var/bsm/20171114.bsm.log sourcetype = unix:host:bsm

11/14/17
6:30:01.000 AM

event="logout" audit-uid=mercator uid=mercator tid="10626 131094 host01.xxx.com" sid="1196470503"
xsl_error="no_XSL_match" event="ftp access"
xsl_error="no_XSL_match" event="ftp logout"
event="logout" audit-uid=mercator uid=mercator tid="10626 196630 host01.xxx.com" sid="2088481153"
xsl_error="no_XSL_match" event="ftp access"
Show all 28 lines
eventtype = err0r error eventtype = nix-all-logs eventtype = nix_errors error host = test01 source = /var/bsm/20171114.bsm.log sourcetype = unix:host:bsm

Actual Logs Format : /var/bsm/20171114.bsm.log with out time stamp.

event="ftp logout" xsl_error="no_XSL_match" event="ftp access" event="su" audit-uid=root uid=root text="success for user mqm" tid="9195 131094 test02.xxx.com" sid="59616287 xsl_error="no_XSL_match" event="ftp logout" xsl_error="no_XSL_match" event="ftp access" xsl_error="no_XSL_match" event="ftp access"xsl_error="no_XSL_match"event="ftp logout" event="login - ssh" audit-uid=007 uid=007 tid="15615 22 test02.xxx.com" sid="4094245480" retval="0" event="logout" audit-uid=007 uid=007 tid="15615 22 test02.xxx.com" sid="4094245480" xsl_error="no_XSL_match" event="rsh access" xsl_error="no_XSL_match"

In this case how to configure props.conf to assign the time stamp when splunk reads the data from the source "/var/bsm/20171114.bsm.log"

Kindly guide me on this.

thanks in advance.

[somesoni2]

You need to configure DATETIME_CONFIG = CURRENT in your sourcetype definition in props.conf on Indexer/Heavy forwarder, whichever comes first in data flow. See this link for more details on Timestamp parsing configurations.
http://docs.splunk.com/Documentation/Splunk/6.6.3/Data/Configuretimestamprecognition#Timestamp_attri...

[Hemnaath]

hi somesoni2, thanks for your effort on this, We have almost 400 + servers configured with the below inputs.conf stanzas via UF and deployed from deployment servers. We have technology add-on placed in all the five Indexer instances. So in this case if I want to apply the below prop.conf stanza details only to this host=test01 when it get indexed in splunk, can I add the stanza like this so that configuration is applied only to this host=test01 not to the other hosts.

App name: Test-IA-unix
Inputs.conf
[monitor:///var/bsm]
sourcetype = unix:host:bsm
crcSalt =
index = unix
disabled = 0

Example:

App name: Test-TA-unix
Props.conf
host::test01 --> where is the host value for an event.
DATETIME_CONFIG = CURRENT

Will that work ?
Note : We have already other configuration details already configured in props.conf and deployed in all the indexer. All the indexer are at EST time zone " Tue Nov 14 12:14:55 EST 2017 "

Kindly guide me on this

[Hemnaath]

Hi Somesoni2, Can you please guide me on this. As I told you we have other details in props.conf for this app and same app is applied for almost 400+ unix nodes in our environment.

App name: Test-IA-unix ( To monitor the file from the source and same app is placed in 400 + nodes)
Inputs.conf
[monitor:///var/bsm]
sourcetype = unix:host:bsm
crcSalt =
index = unix
disabled = 0

We are facing an issue with this node test01 as the actual log data "/var/bsm/2017115.bsm.log" does not have time stamp in it. So I add the stanza in props.conf like this so that configuration is applied only to this host=test01 not to the other hosts.

App name: Test-TA-unix ( Technology add-on app is placed in all the indexer instances )

Props.conf
host::test01 --> where is the host value for an event.
DATETIME_CONFIG = CURRENT

"Actual Log from the source file /var/bsm/2017115.bsm.log" - There is no time stamp in it.

xsl_error="no_XSL_match" event="ftp access"
xsl_error="no_XSL_match" event="ftp logout"
xsl_error="no_XSL_match" event="ftp access"
xsl_error="no_XSL_match" event="ftp access"
xsl_error="no_XSL_match" event="ftp logout"
xsl_error="no_XSL_match" event="ftp access"
xsl_error="no_XSL_match" event="ftp logout"
xsl_error="no_XSL_match" event="ftp logout"
event="su" audit-uid=root uid=root text="success for user mercator" tid="9195 131094 test02.xxx.com" sid="596162876"
xsl_error="no_XSL_match" event="ftp access"
xsl_error="no_XSL_match" event="ftp logout"
xsl_error="no_XSL_match" event="ftp access"
event="su" audit-uid=root uid=root text="success for user mqm" tid="9195 131094 test02.xxx.com" sid="596162876"
xsl_error="no_XSL_match" event="ftp logout"
xsl_error="no_XSL_match" event="ftp access"
xsl_error="no_XSL_match" event="ftp logout"
event="su" audit-uid=root uid=root text="success for user mercator" tid="9195 131094 test02.xxx.com" sid="596162876"
event="su" audit-uid=root uid=root text="success for user mercator" tid="9195 131094 test02.xxx.com" sid="596162876"

Note : Attached partial logs in this comment, but most of the content in the
2017115.bsm.log is the same format.

Kindly guide me on this.

[Hemnaath]

Hi Somesoni2, Can you please guide me on this.

thanks in advance.

[somesoni2]

If you want all logs from the host use DATETIME_CONFIG=CURRENT , you can use this

Props.conf on Test-TA-unix

[host::test01]
DATETIME_CONFIG = CURRENT

[Hemnaath]

Hi Somesoni2, thanks for your effort on this. Hey since the actual logs are multi line and as per the splunk documents we need to have this configured when we use the DATETIME_CONFIG stanza.

• Both "CURRENT" and "NONE" explicitly disable the per-text timestamp identification, so the default event boundary detection (BREAK_ONLY_BEFORE_DATE = true) is likely to not work as desired. When using these settings, use SHOULD_LINEMERGE and/or the BREAK_ONLY_* , MUST_BREAK_* settings to control event merging.

Props.conf

[host::test01]
DATETIME_CONFIG = CURRENT
SHOULD_LINEMERGE = true
MUST_BREAK_AFTER =
MUST_NOT_BREAK_AFTER =

Can I use this stanza to get entire log in splunk.

[pradeepkumarg]

use DATETIME_CONFIG = CURRENT in your props.conf to force splunk to use the index time as the time stamp of the event

[Hemnaath]

hi Pradeep, thanks for your effort on this, We have almost 400 + servers configured with the below inputs.conf stanzas via UF and deployed from deployment servers. We have technology add-on placed in all the five Indexer instances. So in this case if I want to apply the below prop.conf stanza details only to this host=test01 when it get indexed in splunk, can I add the stanza like this so that configuration is applied only to this host not to the other hosts.

App name: Test-IA-unix

Inputs.conf
[monitor:///var/bsm]
sourcetype = unix:host:bsm
crcSalt =
index = unix
disabled = 0

Example:

App name: Test-TA-unix
Props.conf
host::test01 --> where is the host value for an event.
DATETIME_CONFIG = CURRENT

Will that work ?

Note all the indexer are place at Tue Nov 14 12:14:55 EST 2017

Kindly guide me on this.

[Hemnaath]

Hi Pradeep, Can you please guide me on this

thanks in advance

[Hemnaath]

Hi Pradeep, Can you please guide me on this .

thanks in advance.