In props.conf, why is BREAK_ONLY_BEFORE_DATE not properly line breaking my events?

[yqifan83]

My props.conf is like:

BREAK_ONLY_BEFORE_DATE = true
TIME_PREFIX = GMT
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N
MAX_DAYS_HENCE = 5
MAX_TIMESTAMP_LOOKAHEAD = 24
SHOULD_LINEMERGE = true

and my events is like this:

41785:11 INFO [machine] 150 GMT2016-11-16T22:31:07.330Z (18 ms) [uuid] 13683279 [firm] 9001 [sn] 866562 onRequestExpired: request id: 6353697407667535883

41785:11 INFO [machine] 150 GMT2016-11-16T22:31:07.330Z (18 ms) [uuid] 13683279 [firm] 9001 [sn] 866562 postApplicationDataEvent roomId BCAST-fs:582CDE21190C000D data: {"retractEvent":{"retractType":"BY_TIMER"}}

41785:11 INFO [machine] 150 GMT2016-11-16T22:31:07.689Z (59 ms) [uuid] 13683279 [firm] 9001 [sn] 866562 BCAST-fs:582CDE21190C000D processRetractEvent

41785:11 INFO [machine] 150 GMT2016-11-16T22:31:07.845Z (22 ms) [uuid] 13683279 [firm] 9001 [sn] 866562 scrape: 
{
    "requestId": "6353697450617208879",
    "chatId": "BCAST-fs:582CDE21190C000D",
    "operationTypeEnum": "EXPIRED",
    "initiator": 13683279,
    "capturer": 13683279,
    "counterPartyUser": 0,
    "counterPartyUserIdUrn": null,
    "events": [
        {
            "idUrn": "urn:identity-ib-bloomberg-net:1:0:urn%3Afb-ib-bloomberg-net%3ABGEU%3Ain%3Df:uuid%3D13683279",
            "content": "hi=5,\n",
            "eventTypeEnum": "CHAT"
        }
    ],
    "ibdRequestId": "6353697407667535883",
    "takerDealCode": "BGEU",
    "makerDealCode": "QA01",
    "text": "",
    "pointX": 0,
    "pointY": 0,
    "height": 100,
    "width": 100,

I would like to break the events with time. But they take all the above 4 events as one event.
How should I fix this?

[yqifan83]

I have changed to this setting:
TZ=UTC
TRUNCATE = 0
BREAK_ONLY_BEFORE_DATE = false
BREAK_ONLY_BEFORE = ^\d+:\d+\s(INFO|ERROR|FATAL|WARN|DEBUG|TRACE)\s[machine]\s\d+\sGMT
TIME_PREFIX = ^\d+:\d+\s(INFO|ERROR|FATAL|WARN|DEBUG|TRACE)\s[machine]\s\d+\sGMT
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N
MAX_DAYS_HENCE = 5
MAX_TIMESTAMP_LOOKAHEAD = 24
SHOULD_LINEMERGE = true

Now it only break here:
"dealTime": 1479323911
}
,
undefined
,

And it never break at something like 41785:11 INFO [machine] 150 GMT2016-11-16T22:31:07.330Z now .
Somebody knows why this happen? Thank you.

[yqifan83]

I changed to
TZ=UTC
TRUNCATE = 0
BREAK_ONLY_BEFORE_DATE = false
BREAK_ONLY_BEFORE = ^\d+:\d+\s(INFO|ERROR|FATAL|WARN|DEBUG|TRACE)\s[machine]\s\d+\sGMT
TIME_PREFIX = ^\d+:\d+\s(INFO|ERROR|FATAL|WARN|DEBUG|TRACE)\s[machine]\s\d+\sGMT
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N
MAX_DAYS_HENCE = 5
MAX_TIMESTAMP_LOOKAHEAD = 24
SHOULD_LINEMERGE = true

But the problem has not be solved: now it only break at :

"dealTime": 1479481957

}
,
undefined
,
{
"rcodeResponse": 0
}

This is now is took as one event. and

41785:11 INFO [machine] 150 GMT2016-11-16T22:31:07.689Z (59 ms) [uuid] 13683279 [firm] 9001 [sn] 866562 BCAST-fs:582CDE21190C000D processRetractEvent
now is not taken as an event.

[rodrigorsilva · Accepted solution]

Hi,

You can try this in the file props.conf:

SHOULD_LINEMERGE=true
NO_BINARY_CHECK=true
BREAK_ONLY_BEFORE=(\s\d\d\d\d\d:\d\d\sINFO)

Tks

Rodrigo Ribeiro

[yqifan83]

Thank you Rodrigo,
Sometimes the begging of the event is 41785:11 ERROR [machine]
How I could express this after BREAK_ONLY_BEFORE? Thank you!

[yqifan83]

Is this correct? BREAK_ONLY_BEFORE=(\s\d\d\d\d\d:\d\d\s\d{1,5}\s[machine])
And what is NO_BINARY_CHECK=true?

[rodrigorsilva]

This is no problem, I use the following site to test my regular expressions:

https://regex101.com/r/YNDBcR/1

So it should look something like this:
(\s\d\d\d\d\d:\d\d\s(INFO|ERROR))

Note: It is worth noting that this is not a rule, it can be improved.

This option (NO_BINARY_CHECK), according to the link:

http://docs.splunk.com/Documentation/Splunk/6.5.0/Admin/Propsconf

NO_BINARY_CHECK = [true|false]
* When set to true, Splunk processes binary files.
* Can only be used on the basis of [], or [source::],
not [host::].
* Defaults to false (binary files are ignored).
* This setting applies at input time, when data is first read by Splunk.
The setting is used on a Splunk system that has configured inputs
acquiring the data.

Tks

Rodrigo Ribeiro