Unable to break the multi-line events into single event from kinesis log?

[Hemnaath]

Hi All, We are trying to break the multi-line events into single events by building the customizing the configuration provided in the Splunk_TA_AWS Add-on.

Reason for doing this is testing as we want to break the json payload uploaded as individual events ( { id: , timestamp:, message: } ), extract the payload level logGroup: and map it to source meta field and send the payload level unnecessary data to nullQueue.

When we test the below configuration in the live stream of data, the Splunk is unable to break the multiple events in to single Events.

Props.conf

[aws:kinesis]
SHOULD_LINEMERGE=false
LINE_BREAKER=(\[|,\s*|\], )({"id":|"logGroup":)
disabled=false
MAX_TIMESTAMP_LOOKAHEAD=13
TIME_FORMAT=%s%3Q
TIME_PREFIX="timestamp":\s+
TZ=UTC
TRUNCATE=100000

In aws_kinesis_tasks.conf

[unify_timestamp_test]
account = splunk-TA-aws-instance-role
aws_iam_role = test_acc_np
index = unify_main
init_stream_position = LATEST
region = ap-southeast-2
sourcetype = aws:kinesis
stream_names = test-kin-splunkSharpIngestionLogStream
disabled = 1

But it perfectly working fine when we upload sample raw data from the Live stream into the test environment and splunk breaking the multiple events into single events. I have attached the snap shot for the reference.

Sample data: 

{ "owner": "111111111111", "logGroup": "CloudTrail", "logStream": "111111111111_CloudTrail_us-east-1", "subscriptionFilters": [ "Destination" ], "messageType": "DATA_MESSAGE", "logEvents": [ { "id": "31953106606966983378809025079804211143289615424298221568", "timestamp": 1432826855000, "message": "{\"eventVersion\":\"1.03\",\"userIdentity\":{\"type\":\"Root\"}" }, { "id": "31953106606966983378809025079804211143289615424298221569", "timestamp": 1432826855000, "message": "{\"eventVersion\":\"1.03\",\"userIdentity\":{\"type\":\"Root\"}" }, { "id": "31953106606966983378809025079804211143289615424298221570", "timestamp": 1432826855000, "message": "{\"eventVersion\":\"1.03\",\"userIdentity\":{\"type\":\"Root\"}" } ] }

PIC-1 -- Displays events when splunk parsing and ingesting the live stream of data from Kinesis.

PIC-2 -- When same sample data is uploaded in the test environment, it is breaking the multiple event into each single events, using the Line_Break stanza =(\[|,\s*|\], )({"id":|"logGroup":)

[marcosartini]

Hi,
did you manage to solve this issue?

Thanks,
Marco

[marcosartini]

Looking at the add-on code, my current solution is to set in the file aws_kinesis_tasks.conf the parameter

format = CloudWatchLogs

 You can set the value also from GUI in the Record Format field while filling the New Input form.
This setting generates an event for each log message, printing the content of the message field, instead of producing a logEvents list in JSON format.