Using ASCII FS/GS Control Characters as Delimiters

[nboscia]

Hello! I'm having such a hard time with this but I know it is super-simple to do.   Our log files are structured to use  RS (\x1E) and GS (\x1D).   I'm trying to configure the props.conf for this sourcetype but it's just not properly picking up the lines/fields:

BREAK_ONLY_BEFORE_DATE =
DATETIME_CONFIG =
LINE_BREAKER = \x1E
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
category = Application
pulldown_type = 1
description = Logs that contain the FS/RS characters
disabled = false
FIELD_DELIMITER = \x1D

An example of a log (converting ascii character codes as human-readable for this post):

\x1E2021-05-28T12:00:35.489-0700 \x1DINFO \x1Dservice \x1DBlah blah this is the main log message with possible newline characters 

What stupid thing am I doing? 😞

[richgalloway · Accepted solution]

The LINE_BREAKER setting must contain at least one capture group.  The FIELD_DELIMITER setting only applies when INDEXED_EXTRACTION is set.  BREAK_ONLY_BEFORE_DATE only applies when SHOULD_LINEMERGE is true.  Try these settings, which include an EXTRACT to pull out the fields at search time.

DATETIME_CONFIG =
LINE_BREAKER = (\x1E+)
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
description = Logs that contain the FS/RS characters
disabled = false
TIME_PREFIX = ^
TIME_FORMAT = %Y-m-%dT%H:%M:%S.%3N%z
MAX_TIMESTAMP_LOOKAHEAD = 23
EXTRACT-fields = \x1D(?<log_level>\w+)\s\x1D(?<service>\w+)\s\x1D(?<message>.*)

[nboscia]

Oh my, I was REALLY off.  Thank you so very much!!