I've setup Kinesis Firehose to push to Splunk HEC which is ingesting fine, however, I would like to add the logstream field from Cloudwatch to Splunk. The code used in the Lambda can be found here: https://github.com/ptdavies17/CloudwatchFH2HEC
sourcetype = os.environ['SPLUNK_SOURCETYPE']
return_message = '{"time": ' + str(log_event['timestamp']) + ',"host": "' + \
arn + '","source": "' + filterName + ':' + loggrp + '"'
return_message = return_message + ',"sourcetype":"' + sourcetype + '"'
return_message = return_message + ',"event": ' + \
json.dumps(log_event['message']) + '}\n'
return return_message + '\n'
The code block above works and it returns the formatted message:
{
"time": 1234567891011,
"host": "arn",
"source": "some_source",
"sourcetype": "some_source_type",
"event": "event_data"
}
When I add the logstream to the code block as such:
sourcetype = os.environ['SPLUNK_SOURCETYPE']
return_message = '{"time": ' + str(log_event['timestamp']) + ',"host": "' + \
arn + '","source": "' + filterName + ':' + loggrp + '"'
return_message = return_message + ',"logstream":"' + logstrm + '"'
return_message = return_message + ',"sourcetype":"' + sourcetype + '"'
return_message = return_message + ',"event": ' + \
json.dumps(log_event['message']) + '}\n'
return return_message + '\n'
The changes get processed by the transformation Lambda correctly( it is a valid json and the logs in Cloudwatch confirm that):
{
"time": 1234567891011,
"host": "some_host",
"source": "some_source",
"logstream": "logstream_in_amazon",
"sourcetype": "some_source_type",
"event": "some_event_data"
}
But the Kinesis Delivery Stream to Splunk errors out with:
The data is not formatted correctly. To see how to properly format data for Raw or Event HEC endpoints, see Splunk Event Data (http://dev.splunk.com/view/event-collector/SP-CAAAE6P#data). HecServerErrorResponseException{serverRespObject=HecErrorResponseValueObject{text=Invalid data format, code=6, invalidEventNumber=0}, httpBodyAndStatus=HttpBodyAndStatus{statusCode=400, body={"text":"Invalid data format","code":6,"invalid-event-number":0}...
Is there something I'm missing? I've tried tons of changes to the Lambda code, but all of them return this error. Maybe I cannot add a field from CW to Splunk this way?
I am new to Splunk, so I might not be asking the right question or might be missing something, but any help would be much appreciated.