Getting Data In

unable to break the multi-line events into single event from kinesis log?

Hemnaath
Motivator
Hi All, We are trying to break the multi-line events into single events by building the customizing the configuration provided in the Splunk_TA_AWS Add-on.
 

Reason for doing this is testing as we want to break the json payload uploaded as individual events ( { id: , timestamp:, message: } ), extract the payload level logGroup: and map it to source meta field and send the payload level unnecessary data to nullQueue.

When we test the below configuration in the live stream of data, the Splunk is unable to break the multiple events in to single Events.

 

Props.conf

[aws:kinesis]
SHOULD_LINEMERGE=false
LINE_BREAKER=(\[|,\s*|\], )({"id":|"logGroup":)
disabled=false
MAX_TIMESTAMP_LOOKAHEAD=13
TIME_FORMAT=%s%3Q
TIME_PREFIX="timestamp":\s+
TZ=UTC
TRUNCATE=100000

In aws_kinesis_tasks.conf

[unify_timestamp_test]
account = splunk-TA-aws-instance-role
aws_iam_role = test_acc_np
index = unify_main
init_stream_position = LATEST
region = ap-southeast-2
sourcetype = aws:kinesis
stream_names = test-kin-splunkSharpIngestionLogStream
disabled = 1

But it perfectly working fine when we upload sample raw data from the Live stream into the test environment and splunk breaking the multiple events into single events. I have attached the snap shot for the reference.

Sample data: 

{ "owner": "111111111111", "logGroup": "CloudTrail", "logStream": "111111111111_CloudTrail_us-east-1", "subscriptionFilters": [ "Destination" ], "messageType": "DATA_MESSAGE", "logEvents": [ { "id": "31953106606966983378809025079804211143289615424298221568", "timestamp": 1432826855000, "message": "{\"eventVersion\":\"1.03\",\"userIdentity\":{\"type\":\"Root\"}" }, { "id": "31953106606966983378809025079804211143289615424298221569", "timestamp": 1432826855000, "message": "{\"eventVersion\":\"1.03\",\"userIdentity\":{\"type\":\"Root\"}" }, { "id": "31953106606966983378809025079804211143289615424298221570", "timestamp": 1432826855000, "message": "{\"eventVersion\":\"1.03\",\"userIdentity\":{\"type\":\"Root\"}" } ] }

PIC-1 -- Displays events when splunk parsing and ingesting the live stream of data from Kinesis.

PIC-2 -- When same sample data is uploaded in the test environment, it is breaking the multiple event into each single events, using the Line_Break stanza =(\[|,\s*|\], )({"id":|"logGroup":)

 

 
 
 
 
 
 
 
 
Labels (3)
0 Karma
Register for .conf21 Now! Go Vegas or Go Virtual!

How will you .conf21? You decide! Go in-person in Las Vegas, 10/18-10/21, or go online with .conf21 Virtual, 10/19-10/20.