topic Using ASCII FS/GS Control Characters as Delimiters in Getting Data In

Using ASCII FS/GS Control Characters as Delimiters

nboscia — Sat, 29 May 2021 03:55:44 GMT

Hello! I'm having such a hard time with this but I know it is super-simple to do. Our log files are structured to use RS (\x1E) and GS (\x1D). I'm trying to configure the props.conf for this sourcetype but it's just not properly picking up the lines/fields:

BREAK_ONLY_BEFORE_DATE = DATETIME_CONFIG = LINE_BREAKER = \x1E NO_BINARY_CHECK = true SHOULD_LINEMERGE = false category = Application pulldown_type = 1 description = Logs that contain the FS/RS characters disabled = false FIELD_DELIMITER = \x1D

An example of a log (converting ascii character codes as human-readable for this post):

\x1E2021-05-28T12:00:35.489-0700 \x1DINFO \x1Dservice \x1DBlah blah this is the main log message with possible newline characters

What stupid thing am I doing? 😞

Re: Using ASCII FS/GS Control Characters as Delimiters

richgalloway — Sat, 29 May 2021 17:33:17 GMT

The LINE_BREAKER setting must contain at least one capture group. The FIELD_DELIMITER setting only applies when INDEXED_EXTRACTION is set. BREAK_ONLY_BEFORE_DATE only applies when SHOULD_LINEMERGE is true. Try these settings, which include an EXTRACT to pull out the fields at search time.

DATETIME_CONFIG = LINE_BREAKER = (\x1E+) NO_BINARY_CHECK = true SHOULD_LINEMERGE = false description = Logs that contain the FS/RS characters disabled = false TIME_PREFIX = ^ TIME_FORMAT = %Y-m-%dT%H:%M:%S.%3N%z MAX_TIMESTAMP_LOOKAHEAD = 23 EXTRACT-fields = \x1D(?<log_level>\w+)\s\x1D(?<service>\w+)\s\x1D(?<message>.*)

Re: Using ASCII FS/GS Control Characters as Delimiters

nboscia — Sun, 30 May 2021 04:16:45 GMT

Oh my, I was REALLY off. Thank you so very much!!