topic Why is this linebreak is not working with JSON data? in Getting Data In

Why is this linebreak is not working with JSON data?

fisuser1 — Fri, 20 Apr 2018 16:44:18 GMT

Any ideas why this linebreak is not working with JSON data? I've even set the sourcetype to _json, but still no luck.

[prd_thousand_eyes_rest]
INDEXED_EXTRACTIONS = json
**BREAK_ONLY_BEFORE={|**
CHARSET=UTF-8
MAX_TIMESTAMP_LOOKAHEAD=300
SHOULD_LINEMERGE=false
category=Structured
disabled=false
pulldown_type=true

_raw
[{|type=HTTP Server||ruleName=Default HTTP Alert Rule||ResponseCode=200||testId=126574||testName=RAPIDReviewer - Prod||active=1||alertId=13123086||ruleExpression=Error Type is any||dateStart=2018-03-11 06:10:38||ruleId=387415||violationCount=3|}{|type=HTTP Server||ruleName=Default HTTP Alert Rule||ResponseCode=200||testId=126574||testName=RAPIDReviewer - Prod||active=1||alertId=13123086||ruleExpression=Error Type is any||dateStart=2018-03-11 06:10:38||ruleId=387415||violationCount=3|}{|type=HTTP Server||ruleName=Default HTTP Alert Rule||ResponseCode=200||testId=126574||testName=RAPIDReviewer - Prod||active=1||alertId=13123086||ruleExpression=Error Type is any||dateStart=2018-03-11 06:10:38||ruleId=387415||violationCount=3|}]

[{|type=HTTP Server||ruleName=Default HTTP Alert Rule||ResponseCode=200||testId=126574||testName=RAPIDReviewer - Prod||active=1||alertId=13123086||ruleExpression=Error Type is any||dateStart=2018-03-11 06:10:38||ruleId=387415||violationCount=3|}{|type=HTTP Server||ruleName=Default HTTP Alert Rule||ResponseCode=200||testId=126574||testName=RAPIDReviewer - Prod||active=1||alertId=13123086||ruleExpression=Error Type is any||dateStart=2018-03-11 06:10:38||ruleId=387415||violationCount=3|}{|type=HTTP Server||ruleName=Default HTTP Alert Rule||ResponseCode=200||testId=126574||testName=RAPIDReviewer - Prod||active=1||alertId=13123086||ruleExpression=Error Type is any||dateStart=2018-03-11 06:10:38||ruleId=387415||violationCount=3|}]

[{|type=HTTP Server||ruleName=Default HTTP Alert Rule||ResponseCode=200||testId=126574||testName=RAPIDReviewer - Prod||active=1||alertId=13123086||ruleExpression=Error Type is any||dateStart=2018-03-11 06:10:38||ruleId=387415||violationCount=3|}{|type=HTTP Server||ruleName=Default HTTP Alert Rule||ResponseCode=200||testId=126574||testName=RAPIDReviewer - Prod||active=1||alertId=13123086||ruleExpression=Error Type is any||dateStart=2018-03-11 06:10:38||ruleId=387415||violationCount=3|}{|type=HTTP Server||ruleName=Default HTTP Alert Rule||ResponseCode=200||testId=126574||testName=RAPIDReviewer - Prod||active=1||alertId=13123086||ruleExpression=Error Type is any||dateStart=2018-03-11 06:10:38||ruleId=387415||violationCount=3|}]

[{|type=HTTP Server||ruleName=Default HTTP Alert Rule||ResponseCode=200||testId=126574||testName=RAPIDReviewer - Prod||active=1||alertId=13123086||ruleExpression=Error Type is any||dateStart=2018-03-11 06:10:38||ruleId=387415||violationCount=3|}{|type=HTTP Server||ruleName=Default HTTP Alert Rule||ResponseCode=200||testId=126574||testName=RAPIDReviewer - Prod||active=1||alertId=13123086||ruleExpression=Error Type is any||dateStart=2018-03-11 06:10:38||ruleId=387415||violationCount=3|}{|type=HTTP Server||ruleName=Default HTTP Alert Rule||ResponseCode=200||testId=126574||testName=RAPIDReviewer - Prod||active=1||alertId=13123086||ruleExpression=Error Type is any||dateStart=2018-03-11 06:10:38||ruleId=387415||violationCount=3|}]

[{|type=HTTP Server||ruleName=Default HTTP Alert Rule||ResponseCode=200||testId=126574||testName=RAPIDReviewer - Prod||active=1||alertId=13123086||ruleExpression=Error Type is any||dateStart=2018-03-11 06:10:38||ruleId=387415||violationCount=3|}{|type=HTTP Server||ruleName=Default HTTP Alert Rule||ResponseCode=200||testId=126574||testName=RAPIDReviewer - Prod||active=1||alertId=13123086||ruleExpression=Error Type is any||dateStart=2018-03-11 06:10:38||ruleId=387415||violationCount=3|}{|type=HTTP Server||ruleName=Default HTTP Alert Rule||ResponseCode=200||testId=126574||testName=RAPIDReviewer - Prod||active=1||alertId=13123086||ruleExpression=Error Type is any||dateStart=2018-03-11 06:10:38||ruleId=387415||violationCount=3|}]

[{|type=HTTP Server||ruleName=Default HTTP Alert Rule||ResponseCode=200||testId=126574||testName=RAPIDReviewer - Prod||active=1||alertId=13123086||ruleExpression=Error Type is any||dateStart=2018-03-11 06:10:38||ruleId=387415||violationCount=3|}{|type=HTTP Server||ruleName=Default HTTP Alert Rule||ResponseCode=200||testId=126574||testName=RAPIDReviewer - Prod||active=1||alertId=13123086||ruleExpression=Error Type is any||dateStart=2018-03-11 06:10:38||ruleId=387415||violationCount=3|}{|type=HTTP Server||ruleName=Default HTTP Alert Rule||ResponseCode=200||testId=126574||testName=RAPIDReviewer - Prod||active=1||alertId=13123086||ruleExpression=Error Type is any||dateStart=2018-03-11 06:10:38||ruleId=387415||violationCount=3|}]

[{|type=HTTP Server||ruleName=Default HTTP Alert Rule||ResponseCode=200||testId=126574||testName=RAPIDReviewer - Prod||active=1||alertId=13123086||ruleExpression=Error Type is any||dateStart=2018-03-11 06:10:38||ruleId=387415||violationCount=3|}{|type=HTTP Server||ruleName=Default HTTP Alert Rule||ResponseCode=200||testId=126574||testName=RAPIDReviewer - Prod||active=1||alertId=13123086||ruleExpression=Error Type is any||dateStart=2018-03-11 06:10:38||ruleId=387415||violationCount=3|}{|type=HTTP Server||ruleName=Default HTTP Alert Rule||ResponseCode=200||testId=126574||testName=RAPIDReviewer - Prod||active=1||alertId=13123086||ruleExpression=Error Type is any||dateStart=2018-03-11 06:10:38||ruleId=387415||violationCount=3|}]

Re: Why is this linebreak is not working with JSON data?

ssadanala1 — Tue, 29 Sep 2020 19:09:35 GMT

SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)[\{|

Re: Why is this linebreak is not working with JSON data?

fisuser1 — Fri, 20 Apr 2018 18:00:56 GMT

thanks, unfortunetly still seeing the same behavior. not breaking before "{|type" with in the json bracket.

[{|type=HTTP Server||ruleName=Default HTTP Alert Rule||ResponseCode=200||testId=126574||testName=RAPIDReviewer - Prod||active=1||alertId=13123086||ruleExpression=Error Type is any||dateStart=2018-03-11 06:10:38||ruleId=387415||violationCount=3|}{|type=HTTP Server||ruleName=Default HTTP Alert Rule||ResponseCode=200||testId=126574||testName=RAPIDReviewer - Prod||active=1||alertId=13123086||ruleExpression=Error Type is any||dateStart=2018-03-11 06:10:38||ruleId=387415||violationCount=3|}{|type=HTTP Server||ruleName=Default HTTP Alert Rule||ResponseCode=200||testId=126574||testName=RAPIDReviewer - Prod||active=1||alertId=13123086||ruleExpression=Error Type is any||dateStart=2018-03-11 06:10:38||ruleId=387415||violationCount=3|}]

Re: Why is this linebreak is not working with JSON data?

richgalloway — Fri, 20 Apr 2018 18:12:33 GMT

BREAK_ONLY_BEFORE only applies when SHOULD_LINEMERGE=true.

Re: Why is this linebreak is not working with JSON data?

jconger — Fri, 20 Apr 2018 18:21:12 GMT

There are 2 main problems here:

1) The _raw data is not JSON. JSON should look like

{"key1":"value1", "key2":"value2"}

Instead, you have

{|field1=value1||field2=value2|}

2) The _raw data looks like it has arrays of collections. Are you collecting this data programmatically, or reading it from a file? If reading from a file, you could use props.conf to strip out the square brackets using SEDCMD, set up your line breaker, and regex the keys and values. If you are getting this data programmatically, you could format it before streaming it over to Splunk.