All Apps and Add-ons

Amazon Kinesis Firehose: Set index depending on Cloudwatch Log Group name in Lambda Function

knam
Explorer

I recently setup Kinesis Firehose to push to Splunk HEC which is ingesting fine, however, I would like to have the logs sent to "nonprod" or "prod" index depending on the cloudwatch log group name.

For example, if cloudwatch log group name is like "prod", go to prod-index. Else, go to nonprod-index.

I'm utilizing this python script for my lambda function (Cloudwatch2FH2HEC.py) and tried editing the area below but I believe this is nested in the lower JSON and will not matter.
Edit: Hyperlink didn't work
https://github.com/ptdavies17/CloudwatchFH2HEC
https://github.com/ptdavies17/CloudwatchFH2HEC/blob/master/Cloudwatch2FH2HEC.py

Working version:

if "CloudTrail" in loggrp:
    sourcetype="aws:cloudtrail"
elif "VPC" in loggrp:
    sourcetype="aws:cloudwatchlogs:vpcflow"
else:
    sourcetype=os.environ['SPLUNK_SOURCETYPE']

return_message = '{"time": ' + str(log_event['timestamp']) + ',"host": "' + logstrm  + '","source": "' + loggrp + '"'
return_message = return_message + ',"sourcetype":"' + sourcetype  + '"'
return_message = return_message + ',"event": ' + json.dumps(log_event['message']) + '}\n'

return return_message + '\n'

Attempted version:

if "CloudTrail" in loggrp:
    sourcetype="aws:cloudtrail"
elif "VPC" in loggrp:
    sourcetype="aws:cloudwatchlogs:vpcflow"
else:
    sourcetype=os.environ['SPLUNK_SOURCETYPE']

if "prod" in str.lower(loggrp):
    index="prod-kinesis"
else:
    index="nonprod-kinesis"

return_message = '{"time": ' + str(log_event['timestamp']) + ',"host": "' + logstrm  + '","source": "' + loggrp + '"'
return_message = return_message + ',"sourcetype":"' + sourcetype  + '"'
return_message = return_message + ',"index":"' + index  + '"'
return_message = return_message + ',"event": ' + json.dumps(log_event['message']) + '}\n'

return return_message + '\n'

Any thoughts on how to achieve this?

0 Karma
1 Solution

knam
Explorer

Issue was with str. Lambda had the following error

descriptor 'lower' requires a 'str' object but received a 'unicode': TypeError
Traceback (most recent call last):
File "/var/task/lambda_function.py", line 213, in handler
records = list(processRecords(event['records'],streamARN))
File "/var/task/lambda_function.py", line 113, in processRecords
data = ''.join([transformLogEvent(e,data['owner'],arn,data['logGroup'],data['logStream'],data['subscriptionFilters'][0]) for e in data['logEvents']])
File "/var/task/lambda_function.py", line 82, in transformLogEvent
if "prod" in str.lower(loggrp):
TypeError: descriptor 'lower' requires a 'str' object but received a 'unicode'

Changed str.lower(loggrp): to unicode.lower(loggrp) and it is now working and sending to prod or nonprod index depending on cloudwatch log group.

View solution in original post

knam
Explorer

Issue was with str. Lambda had the following error

descriptor 'lower' requires a 'str' object but received a 'unicode': TypeError
Traceback (most recent call last):
File "/var/task/lambda_function.py", line 213, in handler
records = list(processRecords(event['records'],streamARN))
File "/var/task/lambda_function.py", line 113, in processRecords
data = ''.join([transformLogEvent(e,data['owner'],arn,data['logGroup'],data['logStream'],data['subscriptionFilters'][0]) for e in data['logEvents']])
File "/var/task/lambda_function.py", line 82, in transformLogEvent
if "prod" in str.lower(loggrp):
TypeError: descriptor 'lower' requires a 'str' object but received a 'unicode'

Changed str.lower(loggrp): to unicode.lower(loggrp) and it is now working and sending to prod or nonprod index depending on cloudwatch log group.

Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.
Get Updates on the Splunk Community!

Observe and Secure All Apps with Splunk

 Join Us for Our Next Tech Talk: Observe and Secure All Apps with SplunkAs organizations continue to innovate ...

What's New in Splunk Observability - August 2025

What's New We are excited to announce the latest enhancements to Splunk Observability Cloud as well as what is ...

Introduction to Splunk AI

How are you using AI in Splunk? Whether you see AI as a threat or opportunity, AI is here to stay. Lucky for ...