All Apps and Add-ons
Highlighted

"Unable to parse message." for aws:config logs from SQS

Splunk Employee
Splunk Employee

Hi,

We have configured a topic SNS for AWS Config changes, which is bridged to an SQS queue. We do multiple see "messages in flight" on the SQS via the SQS Console. But, the AWS TA input config keep throwing "Unable to parse message." errors in the TA log.
We do see the messages are in json format in the SQS console. We have validated the json message through a validator.

Below are the errors thrown by the TA.

2017-11-16 19:32:57,591 level=CRITICAL pid=19233 tid=Thread-10 logger=splunk_ta_aws.modinputs.sqs_based_s3.handler pos=handler.py:_process:265 | datainput="US_East_Virginia_Config" start_time=1510860606, created=1510860777.58 message_id="655d6019-1525-4647-8160-8a3584d71c59" ttl=300 job_id=8d332ee6-acbd-4628-960d-4642e08e79a4 | message="An error occurred while processing the message."
Traceback (most recent call last):
  File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/sqs_based_s3/handler.py", line 245, in _process
    records = self._parse(message)
  File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/sqs_based_s3/handler.py", line 312, in _parse
    raise ValueError("Unable to parse message.")
ValueError: Unable to parse message.

Any hints/clues are highly appreciated.

thanks!

Highlighted

Re: "Unable to parse message." for aws:config logs from SQS

Path Finder

this happens all the time when they truncate.

0 Karma
Highlighted

Re: "Unable to parse message." for aws:config logs from SQS

Path Finder

jtlittle, can you elaborate? I'm having the same issue and it's not clear to me what you mean.

0 Karma
Highlighted

Re: "Unable to parse message." for aws:config logs from SQS

Engager

I've had the same issue and ended up using a CloudWatch Event trigger -> Lambda -> HEC for config:notification events instead. This assumes you have HEC available and working. I'm not an AWS expert but this solution seem to work well.

Lambda blueprint used is splunk-logging and make sure you use only the Log event with user-specified request parameters section of the code (so comment out the 4 other options above that section) where you can specify sourcetype as aws:config:notification

The CloudWatch event pattern is:

{
  "source": [
    "aws.config"
  ],
  "detail-type": [
    "Config Configuration Item Change"
  ]
}

For CloudWatch event target, use the Lambda and under configure input choose "part of the matched event" and enter "$.detail"

In Config settings use the usual S3->SNS->SQS for Config history and Config snapshot logging but disable the Stream configuration changes and notifications to an Amazon SNS topic option or you'll get duplicate data.

0 Karma
Highlighted

Re: "Unable to parse message." for aws:config logs from SQS

Explorer

Were you able to resolve this problem as I am running into this with my config events as well.

0 Karma