Hi,
We have configured a topic SNS for AWS Config changes, which is bridged to an SQS queue. We do multiple see "messages in flight" on the SQS via the SQS Console. But, the AWS TA input config keep throwing "Unable to parse message." errors in the TA log.
We do see the messages are in json format in the SQS console. We have validated the json message through a validator.
Below are the errors thrown by the TA.
2017-11-16 19:32:57,591 level=CRITICAL pid=19233 tid=Thread-10 logger=splunk_ta_aws.modinputs.sqs_based_s3.handler pos=handler.py:_process:265 | datainput="US_East_Virginia_Config" start_time=1510860606, created=1510860777.58 message_id="655d6019-1525-4647-8160-8a3584d71c59" ttl=300 job_id=8d332ee6-acbd-4628-960d-4642e08e79a4 | message="An error occurred while processing the message."
Traceback (most recent call last):
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/sqs_based_s3/handler.py", line 245, in _process
records = self._parse(message)
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/sqs_based_s3/handler.py", line 312, in _parse
raise ValueError("Unable to parse message.")
ValueError: Unable to parse message.
Any hints/clues are highly appreciated.
thanks!
¿Does anyone know what could be happening ?
I have been checking this issue and I saw that python 3 lib, especially decoder json function does not recognize the message incoming. I wonder if the message changes between aws to Splunk or Json format is not being created properly.
Anyway, I am receiving events nearly to 12h later.
I would like to create a case to aws but I do not know where I can do it.
could you please share your solution as i have the same issues.
well , there are 3 things we are talking about here
aws:config - > aws config S3 bucket -> am having trouble here
aws:config:notification -> cloudwatch event -> firehose -> hec
aws:config:rule - > have to be per accoutn config , can not do through s3 bucket or sqs
Bumping this thread as I am currently having the exact same issue. I really need help with this as I have both resources from AWS as well as Splunk telling me that they have never seen this happen before.
Were you able to resolve this problem as I am running into this with my config events as well.
I've had the same issue and ended up using a CloudWatch Event trigger -> Lambda -> HEC for config:notification events instead. This assumes you have HEC available and working. I'm not an AWS expert but this solution seem to work well.
Lambda blueprint used is splunk-logging and make sure you use only the Log event with user-specified request parameters section of the code (so comment out the 4 other options above that section) where you can specify sourcetype as aws:config:notification
The CloudWatch event pattern is:
{
"source": [
"aws.config"
],
"detail-type": [
"Config Configuration Item Change"
]
}
For CloudWatch event target, use the Lambda and under configure input choose "part of the matched event" and enter "$.detail"
In Config settings use the usual S3->SNS->SQS for Config history and Config snapshot logging but disable the Stream configuration changes and notifications to an Amazon SNS topic option or you'll get duplicate data.
this happens all the time when they truncate.
jtlittle, can you elaborate? I'm having the same issue and it's not clear to me what you mean.