Splunk Add-on for Amazon Web Services: Issue with event line breaking JSON messages without using mvexpand

[ash2l]

Using Kinesis, AWS sends multiple messages in a single JSON body. How to event break JSON messages without using "mvexpand" command?
When we use Kinesis, AWS is sending multiple messages in a single JSON body to Splunk. I need help with breaking the messages in JSON body of an event and create separate events for each message.

Here is the search:

index=aws_xxx sourcetype="aws:kinesis-*-kinesis-*"   | table _time,  logEvents{}.message

When I run this search, it shows multiple records of logEvents{}.message per event. What I need is a single row for each message.

New search to fix this issue with mvexpand command:

index=aws_xxx sourcetype="aws:kinesis-*-kinesis-*"  | mvexpand logEvents{}.message  | table _time,  logEvents{}.message

The above works fine but the mvexpand command has limits of 6500 records which can be updated with limits.conf and other parameters but it will impact the Indexer RAM drastically and I feel that is not a performant way to resolve this issue. Instead I am looking for any substitute of the mvexpand command or trying to find if there is a simpler way to break the events for JSON body so that each message is tagged as a new line within Splunk (may be using props.conf).

[polab]

I am also having the same issue where AWS is sending multiple messages in a single JSON body to Splunk. our Splunk Add-on for AWS is 4.0.0; I wanted to know if upgrading to 4.4.0 will fix the issue. Also, will I be able to easily downgrade from 4.4.0 to 4.0.0 if needed? Thanks

[hunters_splunk]

Hi ash21,

Which version of AWS Add-on are you using to collect Kinesis data? The latest release is 4.2.1 available on SplunkBase:
http://splunkbase.splunk.com/app/1876

Please make sure you configure the Kinesis input correctly and the Splunk Add-on for AWS will ingest Kinesis events for you without the need to customize any configurations:
http://docs.splunk.com/Documentation/AddOns/released/AWS/Kinesis

Hope this helps. Thanks!
Hunter

[polab]

Hi,

We are having a similar issue where AWS is sending multiple messages in a single JSON body to Splunk. We are currently on splunk 4.0.0; I wanted to know if upgrading to splunk 4.4.0 will fix the issue?

if upgrading to splunk 4.4.0 will indeed fix the issue, and we choose to upgrade and upgrading causes other unforeseen issues, will it be fairly easy to downgrade? Can we easily downgrade back to 4.4.0 ourselves?

[ash2l]

The version of Splunk Add-on for AWS is 4.0.0

Is this specific issue fixed in 4.2.1?

Kinesis output is correctly configured based on the document. We see data coming in from Kinesis stream however the JSON events are embedded with multiple messages in it.

The workaround we currently have to break the events by messages within JSON body is:
index=aws_xxx sourcetype="aws:kinesis--kinesis-" | stats count by _time, logEvents{}.message | table _time, logEvents{}.message