Getting Data In

Using ASCII FS/GS Control Characters as Delimiters

nboscia
Engager

Hello! I'm having such a hard time with this but I know it is super-simple to do.   Our log files are structured to use  RS (\x1E) and GS (\x1D).   I'm trying to configure the props.conf for this sourcetype but it's just not properly picking up the lines/fields:

 

BREAK_ONLY_BEFORE_DATE =
DATETIME_CONFIG =
LINE_BREAKER = \x1E
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
category = Application
pulldown_type = 1
description = Logs that contain the FS/RS characters
disabled = false
FIELD_DELIMITER = \x1D

 

 

An example of a log (converting ascii character codes as human-readable for this post):

\x1E2021-05-28T12:00:35.489-0700 \x1DINFO \x1Dservice \x1DBlah blah this is the main log message with possible newline characters 

What stupid thing am I doing? 😞

Labels (3)
0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

The LINE_BREAKER setting must contain at least one capture group.  The FIELD_DELIMITER setting only applies when INDEXED_EXTRACTION is set.  BREAK_ONLY_BEFORE_DATE only applies when SHOULD_LINEMERGE is true.  Try these settings, which include an EXTRACT to pull out the fields at search time.

DATETIME_CONFIG =
LINE_BREAKER = (\x1E+)
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
description = Logs that contain the FS/RS characters
disabled = false
TIME_PREFIX = ^
TIME_FORMAT = %Y-m-%dT%H:%M:%S.%3N%z
MAX_TIMESTAMP_LOOKAHEAD = 23
EXTRACT-fields = \x1D(?<log_level>\w+)\s\x1D(?<service>\w+)\s\x1D(?<message>.*)

 

---
If this reply helps you, Karma would be appreciated.

View solution in original post

richgalloway
SplunkTrust
SplunkTrust

The LINE_BREAKER setting must contain at least one capture group.  The FIELD_DELIMITER setting only applies when INDEXED_EXTRACTION is set.  BREAK_ONLY_BEFORE_DATE only applies when SHOULD_LINEMERGE is true.  Try these settings, which include an EXTRACT to pull out the fields at search time.

DATETIME_CONFIG =
LINE_BREAKER = (\x1E+)
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
description = Logs that contain the FS/RS characters
disabled = false
TIME_PREFIX = ^
TIME_FORMAT = %Y-m-%dT%H:%M:%S.%3N%z
MAX_TIMESTAMP_LOOKAHEAD = 23
EXTRACT-fields = \x1D(?<log_level>\w+)\s\x1D(?<service>\w+)\s\x1D(?<message>.*)

 

---
If this reply helps you, Karma would be appreciated.

nboscia
Engager

Oh my, I was REALLY off.  Thank you so very much!!

0 Karma
Get Updates on the Splunk Community!

How I Instrumented a Rust Application Without Knowing Rust

As a technical writer, I often have to edit or create code snippets for Splunk's distributions of ...

Splunk Community Platform Survey

Hey Splunk Community, Starting today, the community platform may prompt you to participate in a survey. The ...

Observability Highlights | November 2022 Newsletter

 November 2022Observability CloudEnd Of Support Extension for SignalFx Smart AgentSplunk is extending the End ...