All Apps and Add-ons

"Unable to parse message." for aws:config logs from SQS

smitra_splunk
Splunk Employee
Splunk Employee

Hi,

We have configured a topic SNS for AWS Config changes, which is bridged to an SQS queue. We do multiple see "messages in flight" on the SQS via the SQS Console. But, the AWS TA input config keep throwing "Unable to parse message." errors in the TA log.
We do see the messages are in json format in the SQS console. We have validated the json message through a validator.

Below are the errors thrown by the TA.

2017-11-16 19:32:57,591 level=CRITICAL pid=19233 tid=Thread-10 logger=splunk_ta_aws.modinputs.sqs_based_s3.handler pos=handler.py:_process:265 | datainput="US_East_Virginia_Config" start_time=1510860606, created=1510860777.58 message_id="655d6019-1525-4647-8160-8a3584d71c59" ttl=300 job_id=8d332ee6-acbd-4628-960d-4642e08e79a4 | message="An error occurred while processing the message."
Traceback (most recent call last):
  File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/sqs_based_s3/handler.py", line 245, in _process
    records = self._parse(message)
  File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/sqs_based_s3/handler.py", line 312, in _parse
    raise ValueError("Unable to parse message.")
ValueError: Unable to parse message.

Any hints/clues are highly appreciated.

thanks!

saraque
Observer

¿Does anyone know what could be happening ?

I have been checking this issue and I saw that python 3 lib, especially decoder json function does not recognize the message incoming. I wonder if the message changes between aws to Splunk or Json format is not being created properly.

Anyway,  I am receiving events nearly to 12h later.

I would like to create a case to aws but I do not know where I can do it. 

0 Karma

Nanda
Loves-to-Learn Lots

could you please share your solution as i have the same issues.

well , there are 3 things we are talking about here

aws:config  - > aws config S3 bucket    -> am having trouble here 

aws:config:notification -> cloudwatch event -> firehose -> hec

aws:config:rule - > have to be per accoutn config , can not do through s3 bucket or sqs

Tags (1)
0 Karma

DennisCz42
New Member

Bumping this thread as I am currently having the exact same issue. I really need help with this as I have both resources from AWS as well as Splunk telling me that they have never seen this happen before.

0 Karma

arlombar
Explorer

Were you able to resolve this problem as I am running into this with my config events as well.

0 Karma

scoxspau
Engager

I've had the same issue and ended up using a CloudWatch Event trigger -> Lambda -> HEC for config:notification events instead. This assumes you have HEC available and working. I'm not an AWS expert but this solution seem to work well.

Lambda blueprint used is splunk-logging and make sure you use only the Log event with user-specified request parameters section of the code (so comment out the 4 other options above that section) where you can specify sourcetype as aws:config:notification

The CloudWatch event pattern is:

{
  "source": [
    "aws.config"
  ],
  "detail-type": [
    "Config Configuration Item Change"
  ]
}

For CloudWatch event target, use the Lambda and under configure input choose "part of the matched event" and enter "$.detail"

In Config settings use the usual S3->SNS->SQS for Config history and Config snapshot logging but disable the Stream configuration changes and notifications to an Amazon SNS topic option or you'll get duplicate data.

0 Karma

jtlittle
Path Finder

this happens all the time when they truncate.

0 Karma

kranzrm
Path Finder

jtlittle, can you elaborate? I'm having the same issue and it's not clear to me what you mean.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...