topic Re: Issues with multiline events props.conf in Getting Data In

Issues with multiline events props.conf

sunnyb147 — Fri, 19 Mar 2021 18:52:14 GMT

Hi Everyone,

Requesting small help with configuring props.conf which can help me to break the multiline events correctly. These are two types of events which I am trying to ingest for the first one either a part is being ingested or the event is broken for the second one(in a single line) that is ingesting without any issues.

I tried below props.conf but no luck, I am just a newbie therefore requesting your help. For BREAK_ONLY_BEFORE I added the rex so that it can capture and break both types of events.

[testing] BREAK_ONLY_BEFORE={(\s+|)"transaction-id"(\s+|):(\s+|)" SHOULD_LINEMERGE=false NO_BINARY_CHECK=1 TRUNCATE=0 MAX_EVENTS=1024

{ "transaction-id" : "steve-123", "usecase-id" : "123", "timestamp" : "2021-03-07T06:51:27,188+0100", "timestamp-out" : "2021-03-07T06:51:27,188+0100", "component" : "A", "payload" : "{\"error\":\"Internal server error\",\"message\":\"Internal server error\",\"description\":\"The server encountered an unexpected condition that prevented it from fulfilling the request\"}", "country-code" : "IN", "status" : "error", "error-code" : "500", "error" : "Internal Server Error", "message-size" : 176, "logpoint" : "response" } {"transaction-id":"steve-456","usecase-id":"456","timestamp":"2021-03-07T06:51:27,188+0100","timestamp-out":"2021-03-07T06:51:27,188+0100","component":"B","payload":"{\"error\":\"Internalservererror\",\"message\":\"Internalservererror\",\"description\":\"The server encountered an unexpected condition that prevented it from fulfilling the request\"}","country-code":"IN","status":"error","error-code":"500","error":"Internal Server Error","message-size":176,"logpoint":"response"}

Thanks,

Sunny

Re: Issues with multiline events props.conf

to4kawa — Sat, 20 Mar 2021 00:01:34 GMT

[testing] LINE_BREAKER=([\r\n]+){\"transaction-id SHOULD_LINEMERGE=false NO_BINARY_CHECK=1 TRUNCATE=0 MAX_EVENTS=1024

LINE_BREAKER is better.

Re: Issues with multiline events props.conf

sunnyb147 — Sat, 20 Mar 2021 10:40:21 GMT

@to4kawa Thanks for your response, but unfortunately its still the same, the event which is being ingested is broken.

I tried changing the limit of MAX_EVENTS but then too it is not helping 😕

Re: Issues with multiline events props.conf

scelikok — Sat, 20 Mar 2021 14:38:50 GMT

Hi @sunnyb147,

You can use builtin _json sourcetype, it will ingest correctly;

[ _json ] CHARSET=UTF-8 INDEXED_EXTRACTIONS=json KV_MODE=none SHOULD_LINEMERGE=true category=Structured description=JavaScript Object Notation format. For more information, visit http://json.org/ disabled=false pulldown_type=true

Re: Issues with multiline events props.conf

sunnyb147 — Mon, 22 Mar 2021 11:19:55 GMT

@scelikok Thanks for the suggestion but still the event which is being ingested is broken.

{ "transaction-id" : "novotel-123", "usecase-id" : "123", "timestamp" : "2021-03-22T06:51:27,188+0100", "timestamp-out" : "2021-03-22T06:51:27,188+0100", "component" : "A", "payload" : "{\"error\":\"Internal server error\",\"message\":\"Internal server error\",\"description\":\"The server encountered an unexpected condition that prevented it from fulfilling the request\"}", "country-code" : "IN", "status" : "error", "error-code" : "500", "error" : "Internal Server Error", "caller-id" : "", "message-size" : 176, "logpoint" : "response-out" }

I appended above event in the log file but over index I received broken one:

{ "transaction-id" : "novotel-123", "usecase-id" : "123", "timestamp" : "2021-03-22T06:51:27,188+0100", "timestamp-out" : "2021-03-22T06:51:27,188+0100", "component" : "A", Collapse