Getting Data In

Using ASCII FS/GS Control Characters as Delimiters

nboscia
Engager

Hello! I'm having such a hard time with this but I know it is super-simple to do.   Our log files are structured to use  RS (\x1E) and GS (\x1D).   I'm trying to configure the props.conf for this sourcetype but it's just not properly picking up the lines/fields:

 

BREAK_ONLY_BEFORE_DATE =
DATETIME_CONFIG =
LINE_BREAKER = \x1E
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
category = Application
pulldown_type = 1
description = Logs that contain the FS/RS characters
disabled = false
FIELD_DELIMITER = \x1D

 

 

An example of a log (converting ascii character codes as human-readable for this post):

\x1E2021-05-28T12:00:35.489-0700 \x1DINFO \x1Dservice \x1DBlah blah this is the main log message with possible newline characters 

What stupid thing am I doing? 😞

Labels (3)
0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

The LINE_BREAKER setting must contain at least one capture group.  The FIELD_DELIMITER setting only applies when INDEXED_EXTRACTION is set.  BREAK_ONLY_BEFORE_DATE only applies when SHOULD_LINEMERGE is true.  Try these settings, which include an EXTRACT to pull out the fields at search time.

DATETIME_CONFIG =
LINE_BREAKER = (\x1E+)
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
description = Logs that contain the FS/RS characters
disabled = false
TIME_PREFIX = ^
TIME_FORMAT = %Y-m-%dT%H:%M:%S.%3N%z
MAX_TIMESTAMP_LOOKAHEAD = 23
EXTRACT-fields = \x1D(?<log_level>\w+)\s\x1D(?<service>\w+)\s\x1D(?<message>.*)

 

---
If this reply helps you, an upvote would be appreciated.

View solution in original post

richgalloway
SplunkTrust
SplunkTrust

The LINE_BREAKER setting must contain at least one capture group.  The FIELD_DELIMITER setting only applies when INDEXED_EXTRACTION is set.  BREAK_ONLY_BEFORE_DATE only applies when SHOULD_LINEMERGE is true.  Try these settings, which include an EXTRACT to pull out the fields at search time.

DATETIME_CONFIG =
LINE_BREAKER = (\x1E+)
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
description = Logs that contain the FS/RS characters
disabled = false
TIME_PREFIX = ^
TIME_FORMAT = %Y-m-%dT%H:%M:%S.%3N%z
MAX_TIMESTAMP_LOOKAHEAD = 23
EXTRACT-fields = \x1D(?<log_level>\w+)\s\x1D(?<service>\w+)\s\x1D(?<message>.*)

 

---
If this reply helps you, an upvote would be appreciated.

View solution in original post

nboscia
Engager

Oh my, I was REALLY off.  Thank you so very much!!

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!