All Apps and Add-ons

Spluk Addon for AWS

ajith_sukumaran
Explorer

Hello

The addon configured for AWS runs form 3 HFs to get the data from SQS queue, however on the SQS, the Messages Available" grows to 999K+ and is not getting cleared. "Messages in Flight" appears to be around 30

Tried to increase the interval to 20 secs on the CloudTrail Input to see if that helps, but it did not.
The Queue still grows, dont see any errors on the splunk_ta_aws_cloudtrail_main.log

"processing 20 records in s3:logs*/AWSLogs/..json.gz"
"fetched 20 records, wrote 20, discarded 0, redirected 0 from s3:logs
/AWSLogs/*..json.gz"

Any suggestions on how to ensure the Queue is read to clear the Messages Available

Thanks

0 Karma
1 Solution

KranthiGhanta
Engager

Hi @ajith_sukumaran ,

In order to avoid the situation of SQS getting clogged , use more input pipelines from the HF on the same SQS (on the existing inputs, select clone and change the polling period to 90seconds), once the sqs queue is grabbed by one consumer(input) it will not be available for other , so you are increasing the ingestion levels by this method, you can grow as big as you want but make sure your HF resources are not fully throttled by the input processing. ( as its parallel processing)

hope this helps , thanks

View solution in original post

KranthiGhanta
Engager

Hi @ajith_sukumaran ,

In order to avoid the situation of SQS getting clogged , use more input pipelines from the HF on the same SQS (on the existing inputs, select clone and change the polling period to 90seconds), once the sqs queue is grabbed by one consumer(input) it will not be available for other , so you are increasing the ingestion levels by this method, you can grow as big as you want but make sure your HF resources are not fully throttled by the input processing. ( as its parallel processing)

hope this helps , thanks

ajith_sukumaran
Explorer

Thanks. This is exactly the suggested solution later found out from Splunk too.
Thus the config would look as:
eg:

[aws_cloudtrail://AWSCloudTrailData]

sqs_queue = AWS-Splunk

[aws_cloudtrail://AWSCloudTrailData0]

sqs_queue = AWS-Splunk

0 Karma
Get Updates on the Splunk Community!

Technical Workshop Series: Splunk Data Management and SPL2 | Register here!

Hey, Splunk Community! Ready to take your data management skills to the next level? Join us for a 3-part ...

Spotting Financial Fraud in the Haystack: A Guide to Behavioral Analytics with Splunk

In today's digital financial ecosystem, security teams face an unprecedented challenge. The sheer volume of ...

Solve Problems Faster with New, Smarter AI and Integrations in Splunk Observability

Solve Problems Faster with New, Smarter AI and Integrations in Splunk Observability As businesses scale ...