Hi Everyone,
Requesting small help with configuring props.conf which can help me to break the multiline events correctly. These are two types of events which I am trying to ingest for the first one either a part is being ingested or the event is broken for the second one(in a single line) that is ingesting without any issues.
I tried below props.conf but no luck, I am just a newbie therefore requesting your help. For BREAK_ONLY_BEFORE I added the rex so that it can capture and break both types of events.
[testing]
BREAK_ONLY_BEFORE={(\s+|)"transaction-id"(\s+|):(\s+|)"
SHOULD_LINEMERGE=false
NO_BINARY_CHECK=1
TRUNCATE=0
MAX_EVENTS=1024
{
"transaction-id" : "steve-123",
"usecase-id" : "123",
"timestamp" : "2021-03-07T06:51:27,188+0100",
"timestamp-out" : "2021-03-07T06:51:27,188+0100",
"component" : "A",
"payload" : "{\"error\":\"Internal server error\",\"message\":\"Internal server error\",\"description\":\"The server encountered an unexpected condition that prevented it from fulfilling the request\"}",
"country-code" : "IN",
"status" : "error",
"error-code" : "500",
"error" : "Internal Server Error",
"message-size" : 176,
"logpoint" : "response"
}
{"transaction-id":"steve-456","usecase-id":"456","timestamp":"2021-03-07T06:51:27,188+0100","timestamp-out":"2021-03-07T06:51:27,188+0100","component":"B","payload":"{\"error\":\"Internalservererror\",\"message\":\"Internalservererror\",\"description\":\"The server encountered an unexpected condition that prevented it from fulfilling the request\"}","country-code":"IN","status":"error","error-code":"500","error":"Internal Server Error","message-size":176,"logpoint":"response"}
Thanks,
Sunny
Hi @sunnyb147,
You can use builtin _json sourcetype, it will ingest correctly;
[ _json ]
CHARSET=UTF-8
INDEXED_EXTRACTIONS=json
KV_MODE=none
SHOULD_LINEMERGE=true
category=Structured
description=JavaScript Object Notation format. For more information, visit http://json.org/
disabled=false
pulldown_type=true
@scelikok Thanks for the suggestion but still the event which is being ingested is broken.
{
"transaction-id" : "novotel-123",
"usecase-id" : "123",
"timestamp" : "2021-03-22T06:51:27,188+0100",
"timestamp-out" : "2021-03-22T06:51:27,188+0100",
"component" : "A",
"payload" : "{\"error\":\"Internal server error\",\"message\":\"Internal server error\",\"description\":\"The server encountered an unexpected condition that prevented it from fulfilling the request\"}",
"country-code" : "IN",
"status" : "error",
"error-code" : "500",
"error" : "Internal Server Error",
"caller-id" : "",
"message-size" : 176,
"logpoint" : "response-out"
}
I appended above event in the log file but over index I received broken one:
{
"transaction-id" : "novotel-123",
"usecase-id" : "123",
"timestamp" : "2021-03-22T06:51:27,188+0100",
"timestamp-out" : "2021-03-22T06:51:27,188+0100",
"component" : "A",
Collapse
[testing]
LINE_BREAKER=([\r\n]+){\"transaction-id
SHOULD_LINEMERGE=false
NO_BINARY_CHECK=1
TRUNCATE=0
MAX_EVENTS=1024
LINE_BREAKER is better.
@to4kawa Thanks for your response, but unfortunately its still the same, the event which is being ingested is broken.
I tried changing the limit of MAX_EVENTS but then too it is not helping 😕