Getting Data In

Why logstream field from Cloudwatch is being rejected by Kinesis Firehose when using HEC with event data?

David1
Loves-to-Learn

I've setup Kinesis Firehose to push to Splunk HEC which is ingesting fine, however, I would like to add the logstream field from Cloudwatch to Splunk.   The code used in the Lambda can be found here: https://github.com/ptdavies17/CloudwatchFH2HEC 

 

 

 

sourcetype = os.environ['SPLUNK_SOURCETYPE']
return_message = '{"time": ' + str(log_event['timestamp']) + ',"host": "' + \
arn + '","source": "' + filterName + ':' + loggrp + '"'
return_message = return_message + ',"sourcetype":"' + sourcetype + '"'
return_message = return_message + ',"event": ' + \
json.dumps(log_event['message']) + '}\n'
return return_message + '\n'

 

 

 

 
The code block above works and it returns the formatted message:

 

 

 

{
    "time": 1234567891011,
    "host": "arn",
    "source": "some_source",
    "sourcetype": "some_source_type",
    "event": "event_data"
}

 

 

 

 When I add the logstream to the code block as such:

 

 

 

sourcetype = os.environ['SPLUNK_SOURCETYPE']
return_message = '{"time": ' + str(log_event['timestamp']) + ',"host": "' + \
arn + '","source": "' + filterName + ':' + loggrp + '"'
return_message = return_message + ',"logstream":"' + logstrm + '"'
return_message = return_message + ',"sourcetype":"' + sourcetype + '"'
return_message = return_message + ',"event": ' + \
json.dumps(log_event['message']) + '}\n'
return return_message + '\n'

 

 

 

The changes get processed by the transformation Lambda correctly( it is a valid json and the logs in Cloudwatch confirm that):

 

 

 

{
    "time": 1234567891011,
    "host": "some_host",
    "source": "some_source",
    "logstream": "logstream_in_amazon",
    "sourcetype": "some_source_type",
    "event": "some_event_data"
}

 

 

 

But the Kinesis Delivery Stream to Splunk errors out with:

 

 

 

The data is not formatted correctly. To see how to properly format data for Raw or Event HEC endpoints, see Splunk Event Data (http://dev.splunk.com/view/event-collector/SP-CAAAE6P#data). HecServerErrorResponseException{serverRespObject=HecErrorResponseValueObject{text=Invalid data format, code=6, invalidEventNumber=0}, httpBodyAndStatus=HttpBodyAndStatus{statusCode=400, body={"text":"Invalid data format","code":6,"invalid-event-number":0}...

 

 

 

Is there something I'm missing? I've tried tons of changes to the Lambda code, but all of them return this error. Maybe I cannot add a field from CW to Splunk this way?

I am new to Splunk, so I might not be asking the right question or might be missing something, but any help would be much appreciated.

Labels (1)
Tags (2)
0 Karma
Get Updates on the Splunk Community!

Splunk Forwarders and Forced Time Based Load Balancing

Splunk customers use universal forwarders to collect and send data to Splunk. A universal forwarder can send ...

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...