Line breaking behavior questions

romainbouajila — Wed, 22 Jan 2020 05:08:45 GMT

My propfs.conf file for my app looks like the following :

[bit_fuse_log]
pulldown_type = true
NO_BINARY_CHECK = true
TIME_PREFIX = ^
TIME_FORMAT = %H:%M:%S\,%3Q
MAX_TIMESTAMP_LOOKAHEAD = 12
SHOULD_LINEMERGE = false
BREAK_ONLY_BEFORE_DATE = true
TRUNACTE = 0

[bit_fuse_access_proxy]
SHOULD_LINEMERGE = false
NO_BINARY_CHECK = true
MAX_TIMESTAMP_LOOKAHEAD = 32
disabled = false
TRUNCATE = 0
MAX_EVENTS = 10240

and my log files look like the following (note that every line does not start with a blank space)

        <texttexttext>
          <text>
            <text>texttexttext</text>
            <text>text text text</text>
          </text>
        </texttexttext>
      </text>
    </text>
  </text>
</texttext>
--------------------------------------
11:35:11,715 | INFO  | 12345678-12345 | texttexttexttext              | 107 - texttexttexttexttexttext | Outbound Message
---------------------------
ID: xxxxxxx
Address: http://url
Encoding: UTF-8
Http-Method: POST
Content-Type: text/xml
Headers: {Accept=[*/*], SOAPAction=[""]}
Payload: <texttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttext>
  <texttexttext>
    <texttexttexttexttexttexttexttexttexttexttexttext"texttexttexttexttexttexttexttexttexttext">
      <texttext>
        <text>text</text>
        <text>text</text>
        <text/>
        <text/>
        <text>text</text>
        <text>001</text>
        <text/>
        <text>texttext</text>
        <OPERATION>QueryCardDtlsLst</OPERATION>
        <SOURCE_OPERATION/>
        <SOURCE_USERID/>
[...]
--------------------------------------
08:20:22,972 | INFO  | 12345678-234567890 | texttexttexttext            | texttexttexttexttexttexttexttexttext | Outbound Message
---------------------------
ID: 1234
Response-Code: 200
Content-Type: application/json
Headers: {Content-Type=[application/json], Date=[Mon, 06 Jan 2020 01:20:22 GMT]}
Payload: {"texttexttext":{"texttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttext}
--------------------------------------
08:20:24,862 | INFO  | 12345678-234567890 | texttexttexttexttexttext | blablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablabla
08:20:24,862 | INFO  | 12345678-234567890 | texttexttexttexttexttext | blablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablabla
08:20:24,865 | INFO  | 12345678-234567890 | texttexttexttexttexttext | blablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablabla
08:20:24,865 | WARN  | 12345678-234567890 | texttexttexttexttexttext | blablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablabla
08:20:24,865 | INFO  | 12345678-234567890 | texttexttexttexttexttext | blablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablabla
08:20:24,865 | INFO  | 12345678-234567890 | texttexttexttexttexttext | blablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablabla
08:20:24,865 | INFO  | 12345678-234567890 | texttexttexttexttexttext | blablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablabla
08:20:24,866 | INFO  | 12345678-234567890 | texttexttexttexttexttext | blablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablabla
08:20:24,867 | INFO  | 12345678-234567890 | texttexttexttexttexttext | blablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablabla
08:20:24,867 | INFO  | 12345678-234567890 | texttexttexttexttexttext | blablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablabla

----------------------------
ID: 123456
Address: texttexttexttexttexttexttexttext
Encoding: texttext
Http-Method: POST
Content-Type: application/json
Headers: {Accept=[application/json], texttexttexttexttexttexttexttexttexttexttexttexttexttexttext}
Payload: {"startDate":"2020-01-01T00:00:00.000+0700","endDate":"2020-01-06T23:59:59.000+0700","pageNumber":1,"pageSize":300}
--------------------------------------
08:20:24,862 | INFO  | 12345678-234567890 | texttexttexttexttexttext | blablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablabla
08:20:24,862 | INFO  | 12345678-234567890 | texttexttexttexttexttext | blablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablabla
08:20:24,865 | INFO  | 12345678-234567890 | texttexttexttexttexttext | blablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablabla
08:20:24,865 | WARN  | 12345678-234567890 | texttexttexttexttexttext | blablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablabla
<text>

I would like to break at every timestamp at the beginning of line (every arrow in the following picture).
I don't understand Splunk's behavior considering my props.conf file.

My issue is that Splunk sometimes breaks as expected and sometimes not. The most common behavior is :

Breaking before Header
Not breaking single line consecutive events

I can't find a pattern for this behavior, do you have an idea ?

Re: Line breaking behavior questions

PavelP — Wed, 22 Jan 2020 09:57:44 GMT

try:

SHOULD_LINEMERGE=false
LINE_BREAKER=([\r\n]+)\d\d:\d\d:\d\d,
NO_BINARY_CHECK=true
TIME_FORMAT=%H:%M:%S,%3N
TIME_PREFIX=^
MAX_TIMESTAMP_LOOKAHEAD=15

topic Line breaking behavior questions in Splunk Search

Line breaking behavior questions

Re: Line breaking behavior questions