All Apps and Add-ons

Splunk Add-on for Amazon Web Services - Custom Data - SQS Based

rkeq0515
Path Finder

My ultimate goal is to process csv files directly from an S3 bucket/prefix for splunk to index. I have the Splunk Add-on for Amazon Web Services installed and was attempting to create a new input Custom Data Type using SQS-Based S3. I have a bucket created and I have an SQS created. As I am filling out the AWS Input Configuration info, everything is good until I select the SQS Queue Name. I can see the SQS queue that I created, but when I select it the "Save" button turns grey and I cannot save the configuration. Also, I can no longer select the drop down for the SQS Queue Name as my mouse turns to a red circle with a line through it. I have not been able to figure this out. Any help would be appreciated.

0 Karma

venkatasri
SplunkTrust
SplunkTrust

Hi @rkeq0515 

That means you do not have enough permissions towards AWS account to list SQS Queue and other operations advised in add-on docs. Did you check the splunkd and sourcetype=aws:sqs* logs which logs the AWS error behind UI.

0 Karma

patilsonali1729
Path Finder

Hello, 

Did you figure out the solution for this one. Facing the same issue.

0 Karma

rkeq0515
Path Finder

I never got the SQS to work as I had to move forward.  I ended up going with a direct read from the S3 bucket and object, which seemed to work pretty well so far. 

 

Here is a sample input.conf entry.  

[aws_s3://testtype]
aws_account = s3accountname
bucket_name = testbucket
character_set = auto
ct_blacklist = ^$
host_name = s3.amazonaws.com
index = testindex
initial_scan_datetime = 2020-03-01T15:43:47Z
interval = 30
is_secure = True
key_name = csvdata1/
max_items = 100000
max_retries = 3
polling_interval = 60
recursion_depth = -1
sourcetype = aws:s3:testsrctype

0 Karma
Get Updates on the Splunk Community!

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud  In today’s fast-paced digital ...

Observability protocols to know about

Observability protocols define the specifications or formats for collecting, encoding, transporting, and ...

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...