All Apps and Add-ons

Splunk Add-on for Amazon Web Services - Custom Data - SQS Based

rkeq0515
Path Finder

My ultimate goal is to process csv files directly from an S3 bucket/prefix for splunk to index. I have the Splunk Add-on for Amazon Web Services installed and was attempting to create a new input Custom Data Type using SQS-Based S3. I have a bucket created and I have an SQS created. As I am filling out the AWS Input Configuration info, everything is good until I select the SQS Queue Name. I can see the SQS queue that I created, but when I select it the "Save" button turns grey and I cannot save the configuration. Also, I can no longer select the drop down for the SQS Queue Name as my mouse turns to a red circle with a line through it. I have not been able to figure this out. Any help would be appreciated.

0 Karma

venkatasri
SplunkTrust
SplunkTrust

Hi @rkeq0515 

That means you do not have enough permissions towards AWS account to list SQS Queue and other operations advised in add-on docs. Did you check the splunkd and sourcetype=aws:sqs* logs which logs the AWS error behind UI.

0 Karma

patilsonali1729
Path Finder

Hello, 

Did you figure out the solution for this one. Facing the same issue.

0 Karma

rkeq0515
Path Finder

I never got the SQS to work as I had to move forward.  I ended up going with a direct read from the S3 bucket and object, which seemed to work pretty well so far. 

 

Here is a sample input.conf entry.  

[aws_s3://testtype]
aws_account = s3accountname
bucket_name = testbucket
character_set = auto
ct_blacklist = ^$
host_name = s3.amazonaws.com
index = testindex
initial_scan_datetime = 2020-03-01T15:43:47Z
interval = 30
is_secure = True
key_name = csvdata1/
max_items = 100000
max_retries = 3
polling_interval = 60
recursion_depth = -1
sourcetype = aws:s3:testsrctype

0 Karma
Get Updates on the Splunk Community!

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...

Industry Solutions for Supply Chain and OT, Amazon Use Cases, Plus More New Articles ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Enterprise Security Content Update (ESCU) | New Releases

In November, the Splunk Threat Research Team had one release of new security content via the Enterprise ...