topic Re: AWS CloudTrail SQS Based S3 error in Getting Data In

AWS CloudTrail SQS Based S3 error?

leuorrouel — Wed, 15 Mar 2023 16:03:39 GMT

Hi,

I tried to configure CloudTrail SQS Based S3 and I got the following message:

"Warning: This message does not have a valid SNS Signature None None doesn't match required format '^https://sns\\.[-a-z0-9]+\\.amazonaws\\.com(?:\\.cn)?/'"

Sometimes I also get:

"Failed to delete message"

I have no clue where to look in order to solve this issue. I will appreciate any help!

Re: AWS CloudTrail SQS Based S3 error

tlanghals_il — Thu, 06 Jan 2022 18:07:35 GMT

Same issue for ALB access log collection using the SQS based S3 input after upgrading to 5.2.1. Not only is this not indexing the S3 logs, but it's also deleting the SQS messages so it's not appropriately handling the error.

Re: AWS CloudTrail SQS Based S3 error

parcflyer — Tue, 11 Jan 2022 17:13:38 GMT

Having the same issues in the cloud. Was upgraded to 5.2.1 last night and all inputs are doing the exact same thing.

Pulling data from the Que, warning, not indexing and the Que are cleared.

Re: AWS CloudTrail SQS Based S3 error

danint — Tue, 18 Jan 2022 20:18:31 GMT

Haven't looked into this too much, but I'm guessing it is related to the updates in version 5.2.1:

https://docs.splunk.com/Documentation/AddOns/released/AWS/Releasenotes

"""

Version 5.2.1 of the Splunk Add-on for AWS version contains the following new and changed features:

Added validation of the signature of the SNS message being sent from SQS queue to Splunk. The source of the logs is validated by matching the signature of the SNS message with the signature field.

"""

The setup docs for the CloudTrail input state S3->SNS->SQS now. I don't think this was the case before, but I don't have a copy of the old docs to check.

It looks like it is using this plugin to validate. https://pypi.org/project/validate-aws-sns-message/ Note, it "Requires message be no older than one hour, the maximum lifetime of an SNS message."

Splunk, it would be much better if this was an optional feature, as it breaks our infrastructure, especially bad since this is a PATCH update, not MAJOR.

Re: AWS CloudTrail SQS Based S3 error

leuorrouel — Wed, 19 Jan 2022 10:04:12 GMT

I checked Splunkbase, but they do not have version 5.2.1. I know they had this before, but for some reason they reverted it back to version 5.2.0

Re: AWS CloudTrail SQS Based S3 error

parcflyer — Wed, 19 Jan 2022 10:55:36 GMT

We backed out of the 5.2.1 update. 5.2.0 seems to be working.

Re: AWS CloudTrail SQS Based S3 error

splunkoptimus — Wed, 15 Mar 2023 05:48:00 GMT

Hello I'm having the same error, did you manage to fixing it?

2023-03-15 05:26:05,915 level=WARNING pid=2768 tid=Thread-7 logger=splunk_ta_aws.modinputs.sqs_based_s3.handler pos=handler.py:_process:390 | datainput="XXXXXX" start_time=1678855832, message_id=XXXXXXXXXXXXXXX ttl=300 job_id=XXXXXXXXXXXXX | message="Warning: This message does not have a valid SNS Signature Message is too old: 2 days, 16:30:10.556919"