Getting Data In

Using ASCII FS/GS Control Characters as Delimiters

nboscia
Engager

Hello! I'm having such a hard time with this but I know it is super-simple to do.   Our log files are structured to use  RS (\x1E) and GS (\x1D).   I'm trying to configure the props.conf for this sourcetype but it's just not properly picking up the lines/fields:

 

BREAK_ONLY_BEFORE_DATE =
DATETIME_CONFIG =
LINE_BREAKER = \x1E
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
category = Application
pulldown_type = 1
description = Logs that contain the FS/RS characters
disabled = false
FIELD_DELIMITER = \x1D

 

 

An example of a log (converting ascii character codes as human-readable for this post):

\x1E2021-05-28T12:00:35.489-0700 \x1DINFO \x1Dservice \x1DBlah blah this is the main log message with possible newline characters 

What stupid thing am I doing? 😞

Labels (3)
0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

The LINE_BREAKER setting must contain at least one capture group.  The FIELD_DELIMITER setting only applies when INDEXED_EXTRACTION is set.  BREAK_ONLY_BEFORE_DATE only applies when SHOULD_LINEMERGE is true.  Try these settings, which include an EXTRACT to pull out the fields at search time.

DATETIME_CONFIG =
LINE_BREAKER = (\x1E+)
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
description = Logs that contain the FS/RS characters
disabled = false
TIME_PREFIX = ^
TIME_FORMAT = %Y-m-%dT%H:%M:%S.%3N%z
MAX_TIMESTAMP_LOOKAHEAD = 23
EXTRACT-fields = \x1D(?<log_level>\w+)\s\x1D(?<service>\w+)\s\x1D(?<message>.*)

 

---
If this reply helps you, Karma would be appreciated.

View solution in original post

richgalloway
SplunkTrust
SplunkTrust

The LINE_BREAKER setting must contain at least one capture group.  The FIELD_DELIMITER setting only applies when INDEXED_EXTRACTION is set.  BREAK_ONLY_BEFORE_DATE only applies when SHOULD_LINEMERGE is true.  Try these settings, which include an EXTRACT to pull out the fields at search time.

DATETIME_CONFIG =
LINE_BREAKER = (\x1E+)
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
description = Logs that contain the FS/RS characters
disabled = false
TIME_PREFIX = ^
TIME_FORMAT = %Y-m-%dT%H:%M:%S.%3N%z
MAX_TIMESTAMP_LOOKAHEAD = 23
EXTRACT-fields = \x1D(?<log_level>\w+)\s\x1D(?<service>\w+)\s\x1D(?<message>.*)

 

---
If this reply helps you, Karma would be appreciated.

nboscia
Engager

Oh my, I was REALLY off.  Thank you so very much!!

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...