Using SQS based S3 in a private VPC

[erocky]

We'd like to use the SQS based S3 method in the Splunk Add-on for AWS, but have trouble connecting to the endpoint. From my experience it only tries to connect to the legacy URL, which is not supported with the SQS endpoint:

"Private DNS doesn't support legacy endpoints such as queue.amazonaws.com or us-east-2.queue.amazonaws.com." source 

Legacy URL: REGION.queue.amazonaws.com 
New URL: sqs.REGION.amazonaws.com.

I was able to create a Band-Aid fix by adding the legacy URL to /etc/hosts with the new URL's IP, but that's a fragile solution and doesn't support multiple AZs.  

In testing this I found that the AWS CLI supports redirecting through the use of a --endpoint-url argument:

aws sqs receive-message --endpoint-url https://REGION.amazonaws.com/ --queue-url https://sqs.REGION.amazonaws.com/ACCT#/QUEUENAME

It also looks like S3 might support setting a host_name value in the inputs.conf file to allow it to connect to a different endpoint. 

Is there a method for setting the endpoint URL for SQS?

Thank you,

Erocky

[livehybrid]

When you've added your SQS input it should set the sqs_queue_url field, something like:

[aws_sqs_based_s3://YourInputName]
aws_account = loader
aws_iam_role = audit_cloudtrail
index = my_index
interval = 300
s3_file_decoder = ELBAccessLogs
sourcetype = aws:elb:accesslogs
sqs_batch_size = 10
sqs_queue_region = eu-west-2
sqs_queue_url = https://eu-west-2.queue.amazonaws.com/<yourAccountID>/<yourSQSQueue>

You should then be able to update the queue URL to

sqs_queue_url = https://sqs.REGION.amazonaws.com/<YourAccountID>/<YourQueueNamee>