Can you help me with Line Breaker and Event Time?

[damucka]

Hello,

I have a database crashdump file, which has the following structure (from the beginning):

=========================================
Crash protocol for PID 289251
Rundirectory: /usr/sap/CWW/HDB02/ls5433
Process start time: 2018-11-29 15:24:13 782 Local
Exception time: 2018-11-30 21:00:50 859 Local
SectionTimeout: 30sec
KillTimeout: 300sec
=========================================

Table of contents:
   [BUILD]  Build information
   [SYSTEMINFO]  System information
   [CRASH_SHORTINFO]  Exception short info
   ...
   [DISASSEMBLY]  Disassembly of frames in callstack
   [REGISTER_OBJECTS]  Objects registers point to
   [THREADS]  Running threads
WARNING: could not suspend 1 threads

[BUILD]  Build information: (2018-11-30 21:01:01 972 Local)
Version       : 2.00.024.06.1538035880 (fa/hana2sp02)
Build host    : ld4552
Build time    : 2018-09-27 10:26:47
Platform      : linuxx86_64
Compiler      : gcc (SAP release 20170307, based on SUSE gcc6-6.2.1+r239768-2.4) 6.2.1 20160826 [gcc-6-branch revision 239773]
Maketype      : rel
Branch        : fa/hana2sp02
Git hash      : 4d5d5af986c75c6f2d61bde8b289b9c8ca078032
Git mergetime : 2018-09-27 10:11:20
Weekstone     : 0000.00.0
[OK]
--

[SYSTEMINFO]  System information: (2018-11-30 21:01:01 972 Local)
Instance CWW/02, OS Linux ls5433 4.4.121-92.98-default #1 SMP Fri Oct 19 07:52:13 UTC 2018 (e4d85ce) x86_64
[OK]
--

Now, I would like each section to be a separate event. The sections are separated with the [OK], except the heading one, but here I would not have anything against that it is combined together with the [BUILD] section together. This means I would like to set the line breaker to the [OK].

Also, the event time should be set to the timestamp coming in brackets:

[BUILD]  Build information: (2018-11-30 21:01:01 972 Local)
...
[SYSTEMINFO]  System information: (2018-11-30 21:01:01 972 Local)

etc. Unfortunately these timestamps do not come in the first line of the event, at least in case of the first event / header.

How would I achieve both?

The line breaker set to [OK] and the proper event time setting?

Could you please advice the corresponding props.conf entries?

Kind Regards,
Kamil

[harsmarvania57 · Accepted solution]

Hi,

Please try below configuration

props.conf

[yoursourcetype]
BREAK_ONLY_BEFORE_DATE = false
LINE_BREAKER = ([\r\n+])\[\w+\]\s{2}\w+
MAX_TIMESTAMP_LOOKAHEAD = 23
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
TIME_FORMAT = %Y-%m-%d %H:%M:%S %3N
TIME_PREFIX = information: \(
disabled = false

Above configuration will break sample event provided by you in 3 different events
First Event

=========================================
Crash protocol for PID 289251
Rundirectory: /usr/sap/CWW/HDB02/ls5433
Process start time: 2018-11-29 15:24:13 782 Local
Exception time: 2018-11-30 21:00:50 859 Local
SectionTimeout: 30sec
KillTimeout: 300sec
=========================================

Table of contents:
   [BUILD]  Build information
   [SYSTEMINFO]  System information
   [CRASH_SHORTINFO]  Exception short info
   ...
   [DISASSEMBLY]  Disassembly of frames in callstack
   [REGISTER_OBJECTS]  Objects registers point to
   [THREADS]  Running threads
WARNING: could not suspend 1 threads

Second Event

[BUILD]  Build information: (2018-11-30 21:01:01 972 Local)
Version       : 2.00.024.06.1538035880 (fa/hana2sp02)
Build host    : ld4552
Build time    : 2018-09-27 10:26:47
Platform      : linuxx86_64
Compiler      : gcc (SAP release 20170307, based on SUSE gcc6-6.2.1+r239768-2.4) 6.2.1 20160826 [gcc-6-branch revision 239773]
Maketype      : rel
Branch        : fa/hana2sp02
Git hash      : 4d5d5af986c75c6f2d61bde8b289b9c8ca078032
Git mergetime : 2018-09-27 10:11:20
Weekstone     : 0000.00.0
[OK]
--

Third Event

[SYSTEMINFO]  System information: (2018-11-30 21:01:01 972 Local)
Instance CWW/02, OS Linux ls5433 4.4.121-92.98-default #1 SMP Fri Oct 19 07:52:13 UTC 2018 (e4d85ce) x86_64
[OK]
--

[damucka]

Hello harsmarvania57,

Thank you.
Could you perhaps say how the first event will get the event time set in this case?
The string "information: (" is not there.
Also, the rest of the log entries look like the 3-rd one, separated with the [OK]. Will they be translated properly split and given the timestamp like in the case of the 3-rd one?

Kind Regards,
Kamil

[harsmarvania57]

Hi Kamil,

For the First event, splunk will not able to find timestamp because I have explicitly given TIME_PREFIX = information: \( so in this case splunk will assign system time when event actually parsed by splunk.

If you want to set timestamp for the First Event then you can change TIME_PREFIX as given below.

If you want Process start time: then TIME_PREFIX = (information: \(|Process start time: )

If you want Exception time: then TIME_PREFIX = (information: \(|Exception time: )

I didn't get what you are trying to say for 3rd one but splunk will assign timestamp as 2018-11-30 21:01:01 972 to 3rd event. Is this what you are looking for?

[damucka]

Hello,

Thank you, I think the
TIME_PREFIX = (information: (|Exception time: )

is the option to choose. With the 3rd event I meant, that the file is big and has many sections/logs/events which look precisely like the 3rd log. I wanted to just reassure that the configuration above will correctly get them, but I guess this became obvious to me now.

Thank you, I will check the configuration proposed by you.

Kind Regards,
Kamil

[harsmarvania57]

Best to test this configuration in Standalone or Test environment and then implement it in production. I have converted my comment to answer and if it will work then you can accept it.

[damucka]

Hello,

One more question.
Unfortunately I noticed first now that the date prefix is not always "information: (", so sometimes the event time is set wrong. For example:
[CRASH_EMERGENCYSTACK] Emergency stacktrace: (2018-11-30 21:01:01 973 Local)

Also when I would like to define the prefix as only ": (" it will not work as there are other strings with this suffix. In principle I would need to define the prefix using the regex somehow, telling that this would be the line with the [] brackets and having the ": (" as a prefix.
Is it possible?

Kind regards,
Kamil

[harsmarvania57]

Hi Kamil,

Please try below config in props.conf

[yoursourcetype]
BREAK_ONLY_BEFORE_DATE=false
CHARSET=UTF-8
LINE_BREAKER=([\r\n+])\[\w+\]\s+\w+
MAX_TIMESTAMP_LOOKAHEAD=23
NO_BINARY_CHECK=true
SHOULD_LINEMERGE=false
TIME_FORMAT=%Y-%m-%d %H:%M:%S %3N
TIME_PREFIX=(?:\[\w+\]\s+\w+(?:\s+\w+)?\:\s\(|Exception\stime\:\s)
disabled=false

[damucka]

Hello harsmanvania57,

Thank you.
The above configuration works for all sections of the crashdump file but one and I do not why.
For the section below the timestamps seems not to be recognized correctly and the event time is set wrongly:

[CRASH_EXTINFO]  Extended exception info: (2018-12-03 12:29:34 058 Local)
----> Dump of siginfo contents <----
  signal:      11(SIGSEGV)
  code:        STACK OVERFLOW: 2(SEGV_ACCERR: invalid permissions for mapped object)

The event time set is 12:29:34.015 PM, so it is taken from the previous section.
The next section after that gets recognized already correctly (timestamp) and the event time is set in a right way:

[CRASH_CONTEXT]  Context info: (2018-12-03 12:29:34 058 Local)
----> Crashing context information <----
  ContextStack at (0x00007f3b4e31c460)

The event time set in this case is 12:29:34.058 PM, so it is correct.

Any hint why in this one case the timestamp does not get recognized correctly?

Kind Regards,
Kamil

[harsmarvania57]

As you don't have fixed number of words after [] and before : (, TIME_PREFIX regex didn't work for below content.

[CRASH_EXTINFO]  Extended exception info: (2018-12-03 12:29:34 058 Local)

Try TIME_PREFIX = (?:\[\w+\]\s+\w+(?:\s+\w+)*?\:\s\(|Exception\stime\:\s)

[gcusello]

Hi damucka,
let me understand: you want to break your file using [OK] as Line Breaker and you want as timestamp the date and time that is between round brackets, is it correct?

If this is your requirements, try this props.conf:

[ my_sourcetype]
SHOULD_LINEMERGE=true
NO_BINARY_CHECK=true
BREAK_ONLY_BEFORE=\[OK\]
TIME_FORMAT=%Y-%m-%d %H:%M:%S
TIME_PREFIX=\(

Bye.
Giuseppe