Splunk Add-on for Microsoft Cloud Services not line breaking JSON docs from event hub

[hughkelley]

The add-on fails to line break JSON docs into separate events/logs when pulling from an event hub.

Certain Azure services seem to write multiple JSON docs to a single event hub message.

Is there an option to correct this parsing?

{"body":{"records": {"DataCenterName": "East US 2", "DeploymentUnit": "xyz", "EventId": 160, "EventName": "AzureBackupCentralReport", "properties": {"VaultUniqueId": ".........
{"body":{"records": {"DataCenterName": "East US 2", "DeploymentUnit": "xyz", "EventId": 160, "EventName": "AzureBackupCentralReport", "properties": {"VaultUniqueId": ".........
{"body":{"records": {"DataCenterName": "East US 2", "DeploymentUnit": "xyz", "EventId": 160, "EventName": "AzureBackupCentralReport", "properties": {"VaultUniqueId": ".........
{"body":{"records": {"DataCenterName": "East US 2", "DeploymentUnit": "xyz", "EventId": 160, "EventName": "AzureBackupCentralReport", "properties": {"VaultUniqueId": ".........
{"body":{"records": {"DataCenterName": "East US 2", "DeploymentUnit": "xyz", "EventId": 160, "EventName": "AzureBackupCentralReport", "properties": {"VaultUniqueId": ".........
{"body":{"records": {"DataCenterName": "East US 2", "DeploymentUnit": "xyz", "EventId": 160, "EventName": "AzureBackupCentralReport", "properties": {"VaultUniqueId": ".........
{"body":{"records": {"DataCenterName": "East US 2", "DeploymentUnit": "xyz", "EventId": 160, "EventName": "AzureBackupCentralReport", "properties": {"VaultUniqueId": ".........
{"body":{"records": {"DataCenterName": "East US 2", "DeploymentUnit": "xyz", "EventId": 160, "EventName": "AzureBackupCentralReport", "properties": {"VaultUniqueId": ".........
{"body":{"records": {"DataCenterName": "East US 2", "DeploymentUnit": "xyz", "EventId": 160, "EventName": "AzureBackupCentralReport", "properties": {"VaultUniqueId": ".........
{"body":{"records": {"DataCenterName": "East US 2", "DeploymentUnit": "xyz", "EventId": 160, "EventName": "AzureBackupCentralReport", "properties": {"VaultUniqueId": ".........
{"body":{"records": {"DataCenterName": "East US 2", "DeploymentUnit": "xyz", "EventId": 160, "EventName": "AzureBackupCentralReport", "properties": {"VaultUniqueId": ".........

[JkNo]

Add the following to  props.conf

[yoursourcetypename]
CHARSET=UTF-8
INDEXED_EXTRACTIONS=json
KV_MODE=none
SHOULD_LINEMERGE=true
category=Structured
description=JavaScript Object Notation format. For more information, visit http://json.org/
disabled=false
pulldown_type=true

[dfronck]

We get these too but only when the add-on first starts. Then it seems like everything line breaks correctly.

The suggested props config did not fix this for us. 

[vmhenard]

Hello,

We have the same issue, we are currently using a Regex line breaker to remove the outer layers of json added by the event-hub (as well as the x-opt-sequence-number, x-opt-offset and x-opt-enqueued-time fields) and only get the events themselves.

It is not ideal, but it works so far.

(\s*\{\"body\"\:\{\"records\"\:\s*\[)|((?<=\}),(?=\{\s*\"))|((?<=\})\]\},\"x-opt.*\}\s*\{\"body\"\:\{\"records\"\:\s*\[)|((?<=\})\]\},\"x-opt.*\})

First group catches the first of new messages, second group catches the events nested in records, third groups catches the end of a message and the start of a new one, fourth group catches the end of the last message.
Hope this helps, it might need to be tweaked depending on the resource.