All Apps and Add-ons

"Unable to parse message." for aws:config logs from SQS

Splunk Employee
Splunk Employee


We have configured a topic SNS for AWS Config changes, which is bridged to an SQS queue. We do multiple see "messages in flight" on the SQS via the SQS Console. But, the AWS TA input config keep throwing "Unable to parse message." errors in the TA log.
We do see the messages are in json format in the SQS console. We have validated the json message through a validator.

Below are the errors thrown by the TA.

2017-11-16 19:32:57,591 level=CRITICAL pid=19233 tid=Thread-10 logger=splunk_ta_aws.modinputs.sqs_based_s3.handler | datainput="US_East_Virginia_Config" start_time=1510860606, created=1510860777.58 message_id="655d6019-1525-4647-8160-8a3584d71c59" ttl=300 job_id=8d332ee6-acbd-4628-960d-4642e08e79a4 | message="An error occurred while processing the message."
Traceback (most recent call last):
  File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/sqs_based_s3/", line 245, in _process
    records = self._parse(message)
  File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/sqs_based_s3/", line 312, in _parse
    raise ValueError("Unable to parse message.")
ValueError: Unable to parse message.

Any hints/clues are highly appreciated.



¿Does anyone know what could be happening ?

I have been checking this issue and I saw that python 3 lib, especially decoder json function does not recognize the message incoming. I wonder if the message changes between aws to Splunk or Json format is not being created properly.

Anyway,  I am receiving events nearly to 12h later.

I would like to create a case to aws but I do not know where I can do it. 

0 Karma

Loves-to-Learn Lots

could you please share your solution as i have the same issues.

well , there are 3 things we are talking about here

aws:config  - > aws config S3 bucket    -> am having trouble here 

aws:config:notification -> cloudwatch event -> firehose -> hec

aws:config:rule - > have to be per accoutn config , can not do through s3 bucket or sqs

Tags (1)
0 Karma

New Member

Bumping this thread as I am currently having the exact same issue. I really need help with this as I have both resources from AWS as well as Splunk telling me that they have never seen this happen before.

0 Karma


Were you able to resolve this problem as I am running into this with my config events as well.

0 Karma


I've had the same issue and ended up using a CloudWatch Event trigger -> Lambda -> HEC for config:notification events instead. This assumes you have HEC available and working. I'm not an AWS expert but this solution seem to work well.

Lambda blueprint used is splunk-logging and make sure you use only the Log event with user-specified request parameters section of the code (so comment out the 4 other options above that section) where you can specify sourcetype as aws:config:notification

The CloudWatch event pattern is:

  "source": [
  "detail-type": [
    "Config Configuration Item Change"

For CloudWatch event target, use the Lambda and under configure input choose "part of the matched event" and enter "$.detail"

In Config settings use the usual S3->SNS->SQS for Config history and Config snapshot logging but disable the Stream configuration changes and notifications to an Amazon SNS topic option or you'll get duplicate data.

0 Karma

Path Finder

this happens all the time when they truncate.

0 Karma

Path Finder

jtlittle, can you elaborate? I'm having the same issue and it's not clear to me what you mean.

0 Karma
Get Updates on the Splunk Community!

.conf23 Registration is Now Open!

Time to toss the .conf-etti 🎉 —  .conf23 registration is open!   Join us in Las Vegas July 17-20 for ...

Don't wait! Accept the Mission Possible: Splunk Adoption Challenge Now and Win ...

Attention everyone! We have exciting news to share! We are recruiting new members for the Mission Possible: ...

Unify Your SecOps with Splunk Mission Control

In today’s post, I'm excited to share some recent Splunk Mission Control innovations. With Splunk Mission ...