topic Re: BREAK_ONLY_BEFORE and BREAK_ONLY_BEFORE_DATE=false in the same props.conf in Getting Data In

BREAK_ONLY_BEFORE and BREAK_ONLY_BEFORE_DATE=false in the same props.conf

cweissnike — Wed, 27 Mar 2019 17:38:39 GMT

We're trying to break up some log entries that look like:

2019-03-27 17:11:59.942 Request was not matched as were no stubs registered:
{
  "url" : "/",
  "absoluteUrl" : "http://localhost:8080/",
  "method" : "GET",
  "clientIp" : "127.0.0.1",
  "headers" : {
    "User-Agent" : "Wget",
    "Connection" : "close",
    "Host" : "localhost:8080"
  },
  "cookies" : { },
  "browserProxyRequest" : false,
  "loggedDate" : 1553706719942,
  "bodyAsBase64" : "",
  "body" : "",
  "loggedDateString" : "2019-03-27T17:11:59Z",
  "queryParams" : { }
}

Our props.conf looks like:

SHOULD_LINEMERGE = true
BREAK_ONLY_BEFORE = \d\d\d\d[-]\d\d[-]\d\d[ ]\d\d[:]\d\d[:]\d\d[.]\d\d\d

The trouble is, our Splunk result is breaking on every date. The above regex matches only the top date format in my regex tester.
So with the above data, Splunk is parsing it into:
Record 1:

2019-03-27 17:11:59.942 Request was not matched as were no stubs registered:
    {
      "url" : "/",
      "absoluteUrl" : "http://localhost:8080/",
      "method" : "GET",
      "clientIp" : "127.0.0.1",
      "headers" : {
        "User-Agent" : "Wget",
        "Connection" : "close",
        "Host" : "localhost:8080"
      },
      "cookies" : { },
      "browserProxyRequest" : false,

Record 2:

      "loggedDate" : 1553706719942,
      "bodyAsBase64" : "",
      "body" : "",

Record 3:

      "loggedDateString" : "2019-03-27T17:11:59Z",
      "queryParams" : { }
    }

That sure looks like it's using both BREAK_ONLY_BEFORE and BREAK_ONLY_BEFORE_DATE = true. We've tried adding BREAK_ONLY_BEFORE_DATE = false to the props.conf and it's still breaking up at all dates. I read in another Q that you cannot use both BREAK_ONLY_BEFORE and BREAK_ONLY_BEFORE_DATE in the same props.conf, how do we set it to not break at the other date instances?

Re: BREAK_ONLY_BEFORE and BREAK_ONLY_BEFORE_DATE=false in the same props.conf

somesoni2 — Wed, 27 Mar 2019 18:08:50 GMT

I find this configuration works correct consistently

[YourSourcetype]
#making it false as we're changing line boundary in LINE_BREAKER
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)(?=\d{4}-\d{2}-\d{2}\s+\d{2})
TIME_PREFIX = ^
TIME_FORMAT = %Y-%m-%d %H:%M:%S.%3N
MAX_TIMESTAMP_LOOKAHEAD = 23

Re: BREAK_ONLY_BEFORE and BREAK_ONLY_BEFORE_DATE=false in the same props.conf

cweissnike — Wed, 27 Mar 2019 20:20:57 GMT

Hmmm. That didn't seem to change the results. Just to verify, I threw some garbage into the file and it did correctly complain, so I know it's actually reading the file I'm editing.

Re: BREAK_ONLY_BEFORE and BREAK_ONLY_BEFORE_DATE=false in the same props.conf

cweissnike — Tue, 29 Sep 2020 23:54:08 GMT

Nevermind - There's something far weirder going on - Splunk is not actually processing any of the directives in our props.conf file, it's just defaulting to the BREAK_ONLY_BEFORE_DATE=true.

Re: BREAK_ONLY_BEFORE and BREAK_ONLY_BEFORE_DATE=false in the same props.conf

simontam — Thu, 30 May 2019 06:46:56 GMT

I have the same problem. No matter what I changed in the props.conf, Splunk still break the lines by date time. Did you fix your problem yet?

Re: BREAK_ONLY_BEFORE and BREAK_ONLY_BEFORE_DATE=false in the same props.conf

cweissnike — Thu, 30 May 2019 15:33:30 GMT

In our case, it turned out to be server-side filters that were overriding our local configs. Since our instance is managed by a central team, we did not have visibility into the server-side filters.