topic Re: Multiple log formats in a single log file in Getting Data In

Multiple log formats in a single log file

ChhayaV — Mon, 28 Sep 2020 14:30:43 GMT

hi,
I've log file with multiple log formats.
sample.log file

Type 1:

[Thu May 31 13:27:14 2012] FATAL: WARNING: The current transaction ID is Oxfwq3SgJS. Run local database
repair with rebuild database option enabled to fix it before it reaches allowed limit cf transactions
(Oxqfhhe320)

[Thu May 31 14:01:38 2012] FATAL: WARNING: The current transaction ID is Oxeap54lh7. Run local database
repair with rebuild database option enabled to fix it before it reaches allowed limit cf transactions
(Oxfqfhhe 020)

Type 2:

3996491294 ZONE: 2012/06/01 9:54:21.599 (Oxjun3l:0x63) Sending search result entry
“docsismacaddr—1\,6\,00:l5:dl:al:ed:c9,ou—IKV,ou—NorthWest,o--General” to connection 0x973fw410

3996491294 ZONE: [2012/06/01 9:54:21.599) (90.21.103.1:42009) (Oxjun3l:0x63) Sending operation result
0:MM:1 to connection 3x972fw410

Type 3:

[-- DRost Logging STARTED Fri Jun 1 02:47:47 2012 -- ]

[-- DRost Logging STARTED Fri Jun 1 03:07:35 2012 -- ]

Type 4:

Jun 01 02:45:35 NDS iMonitor for Novell eDirectory 9.9.5 SP5 v20506.01 SI’S started successfully.

Jun 01 03:09:27

conf files:

inputs.conf in system/local directory to load the sample.log in splunk

[monitor://D:\sample.log]

props.conf

[Mysourcetype]

DATETIME_CONFIG = /etc/system/datetime.xml

LINE_BREAKER = ([\r\n])+(?=([\w{3}\s(\w{3})\s(\d{1,2})\s(\d{2}):(\d{2}):(\d{2})\s(\d{4})]|\d{10}:?\s\w{4}:|[\s--\s|\w{3}\s\d{2}\s\d{2}:\d{2}:\d{2}\s\s))

SHOULD_LINEMERGE=false

datetime.xml in /etc/system/datetime.xml

datetime>
!-- [Sat May 31 13:27:14 2012] -->
define name="_datetimeformat1" extract="litmonth, day, hour, minute, second, year">

text>[\w{3}\s(\w{3})\s(\d{1,2})\s(\d{2}):(\d{2}):(\d{2})\s(\d{4})]

/define>
!-- [2012/06/01 8:54:21.599] -->
define name="_datetimeformat2" extract="year, month, day, hour, minute, second, subsecond">

text>[(\d{4})/(\d{2})/(\d{2})\s\s(\d{1,2}):(\d{2}):(\d{2}).(\d{3})]

/define>

!-- Fri Jun 1 02:47:47 2012 -->

define name="_datetimeformat3" extract="litmonth, day, hour, minute, second, year">

text>\s\w{3}\s(\w{3})\s\s?(\d{1,2})\s(\d{2}):(\d{2}):(\d{2})\s(\d{4})

/define>

!-- Jun 01 02:45:35 NDS iMonitor for Novell eDirectory 8.8.5 SP5 v20506.01 SP5 started successfully. -->

define name="_datetimeformat4" extract="month, day, hour, minute, second">

text>(\w{3})\s(\d{2})\s(\d{2}):(\d{2}):(\d{2})\s\s

/define>

timePatterns>

use name="_datetimeformat1"/>

use name="_datetimeformat2"/>

use name="_datetimeformat3"/>

use name="_datetimeformat4"/>
/timePatterns>
datePatterns>
use name="_datetimeformat1"/>

use name="_datetimeformat2"/>

use name="_datetimeformat3"/>

use name="_datetimeformat4"/>

/datePatterns>

/datetime>

while restarting splunk i'm getting error as

error while parsing 'C:\program files....\datetime.xml

[Errno 13] permission denied: 'C:\program files....\datetime.xml

Please tell me where is the problem?
where should put datetime.xml if i dont have custom app
IS this way to load log file with multiple formats?

Thank you

Re: Multiple log formats in a single log file

linu1988 — Tue, 06 Aug 2013 13:14:51 GMT

You can put in the local system folder of splunk.

Please follow the documentation:
http://blogs.splunk.com/2009/12/02/configure-splunk-to-pull-a-date-out-of-a-non-standard-filename/

Re: Multiple log formats in a single log file

sowings — Tue, 06 Aug 2013 15:03:26 GMT

Note that even with datetime.xml properly sited with the right permissions, Splunk will develop an affinity for the format rule for a given input stream. This means that a combined log file, with several different time stamp formats, will end up having correct time stamps for those that match its preferred time stamp, and "weird" or "matched to adjacent log event" time stamps for those that are in a different format.

If at all possible, you should split the log streams, or agree upon a consistent time format.

Re: Multiple log formats in a single log file

ChhayaV — Wed, 07 Aug 2013 10:59:24 GMT

Even after placing it in local system folder the error
[Error 13] Permission Denied : \path\ datetime.xml persists.
Is there any other way to load this kind of log file(with multiple formats) without using datetime.xml

Re: Multiple log formats in a single log file

sowings — Wed, 07 Aug 2013 12:28:26 GMT

Check the file permissions on the file; if it's not readable by Splunk, it won't be read, regardless of where it is located.

And even with datetime.xml, it's likely that you won't be able to load this multi-format log file successfully, per my response below.

Re: Multiple log formats in a single log file

ChhayaV — Thu, 08 Aug 2013 05:44:31 GMT

So there is no other way to load multi-format log file?
We don't want split the log file into 4 different files(one file for each format).

Re: Multiple log formats in a single log file

ChhayaV — Wed, 14 Aug 2013 09:16:14 GMT

/etc/system/datetime.xml is this the way to specify DATETIME_CONFIG in windows?