Getting Data In
Highlighted

Why is line breaking on certain events?

Path Finder

Hi!

I am currently having some problems breaking certain events from an Oracle log correctly. The log is being onboarded via a HF, and so the props.conf file containing the sourcetype of the log is located there. This is my current props.conf configuration for the sourcetype:

[application:env:oracle:alert]
TRUNCATE = 0
TIME_PREFIX = ^
MAX_TIMESTAMP_LOOKAHEAD = 32
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%6Q%:z
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\n\r]+)\d{4}-\d{2}-\d{2}[T]\d{2}:\d{2}:\d{2}.\d{6}\+\d{2}:\d{2}

A sample of the log file:

2019-04-23T02:00:00.234792+02:00
Closing scheduler window
Closing Resource Manager plan via scheduler window
Clearing Resource Manager plan via parameter
2019-04-23T04:07:57.219629+02:00
ALTER SYSTEM ARCHIVE LOG
2019-04-23T04:07:57.258800+02:00
Thread 1 advanced to log sequence 867 (LGWR switch)
  Current log# 3 seq# 867 mem# 0: +DATA/FWEB1/ONLINELOG/group_3.260.979816057
  Current log# 3 seq# 867 mem# 1: +FRA/FWEB1/ONLINELOG/group_3.259.979816057
2019-04-23T04:07:57.426428+02:00
TT02: Standby redo logfile selected for thread 1 sequence 867 for destination LOG_ARCHIVE_DEST_2
2019-04-23T04:07:58.682079+02:00
Archived Log entry 1382 added for T-1.S-866 ID 0x49172f7c LAD:1
2019-04-23T04:57:18.076346+02:00
Incremental restore complete of datafile 4 to datafile copy +FRA/FWEB1/DATAFILE/users.553.999450151
  checkpoint is 213563471108
  last deallocation scn is 202352337426
Incremental restore complete of datafile 3 to datafile copy +FRA/FWEB1/DATAFILE/undotbs1.552.999450151
  checkpoint is 213563471108
  last deallocation scn is 213563368804
Incremental restore complete of datafile 1 to datafile copy +FRA/FWEB1/DATAFILE/system.269.999450149
  checkpoint is 213563471108
  last deallocation scn is 201963629285
Incremental restore complete of datafile 2 to datafile copy +FRA/FWEB1/DATAFILE/sysaux.432.999450147
  checkpoint is 213563471108
  last deallocation scn is 213563209794

How it is being parsed in "Add Data" preview on the HF:

alt text

How it is being parsed when "live" (host, source and sourcetype have been anonymized):

alt text

As you can see the last log message is not being parsed correctly according to the props.conf specifications. Any ideas on how to fix this so that the log is onboarded correctly?

PS: I know I could probably fix this by installing the Splunk TA for Oracle, but in this case that is not an option.

0 Karma
Highlighted

Re: Why is line breaking on certain events?

Motivator

I don't know why your live environment is showing different results. Perhaps parts of the events arrive with a delay and thus form a new event.

Instead of modifying LINEBREAKER and disabling line merging, you can also leave LINEBREAKER as it is (by default every newline) and set SHOULDLINEMERGE=true with BREAKONLYBEFOREDATE=true:

[oracle]
category = Custom
pulldown_type = true
NO_BINARY_CHECK = true
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%6N%:z

(I left out SHOULDLINEMERGE=true and BREAKONLYBEFOREDATE=true because these are the default settings.)

alt text

You can ignore the orange warnings, since not all lines contain a timestamp. The lines which do not contain a timestamp are merged.

I also noticed my TIME_FORMAT is slightly different than yours.
Check out https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Commontimeformatvariables

0 Karma
Highlighted

Re: Why is line breaking on certain events?

Path Finder

Hi, and thanks for your response! I tried your variant of the props.conf configuration, but with little success. So unfortunately I am stuck with the same problem.

0 Karma
Highlighted

Re: Why is line breaking on certain events?

Ultra Champion

How are you ingesting this data? Through a network input or reading a file? As @whrg mentions, it could be that the line that is shown separately comes in with a small delay, causing splunk to fail in including it in the previous multiline event. Depending on the ingest method, there might be some tricks to resolve that.

0 Karma
Highlighted

Re: Why is line breaking on certain events?

Path Finder

This is ingested via reading a file, so there shouldn't be any delay problems =/

0 Karma