Deployment Architecture

Error Ingest Data AWS Cloudtrail "error=Traceback (most recent call last):"

zksvc
Contributor

Hi Everyone, 

I encountered an error while ingesting sourcetype=aws:cloudtrails in AWS Apps. I attempted to ingest data from the following sources: aws:waflogs, aws:network-firewall-log, aws:cloudtrails, aws:securityhub-log-group. However, upon checking, only aws:waflogs and aws:network-firewall-log were ingested. Attached below are the errors from the logs. 

zksvc_0-1749639515584.png

Also i screenshot inputs config from the apps side here : 

zksvc_2-1749639584109.png

Last i show you the proof if i only received that 2 sourctypes here : 

zksvc_3-1749639668960.png

 

If you have any experience from this issue, please give me the answer. 

 

Danke,

 

Zake

 

Labels (3)
0 Karma

livehybrid
SplunkTrust
SplunkTrust

Hi @zksvc 

It looks like the inputs are polling AWS Cloudwatch too frequently, which is giving your Rate Limit exception. 

If you have just set this up then it will be trying to pull logs back from whatever the only_after date you set was (see https://splunk.github.io/splunk-add-on-for-amazon-web-services/CloudWatchLogs/ for input config descriptions)

If you left this field blank then I believe it tries to load all the events in the Cloudwatch logs group in AWS. Ultimately it looks like its repeatedly querying CW Logs to get more logs which is why it is hitting the rate limit. The number of polls to CW Logs will reduce once it has caught up to the current date. It might be worth enabling one at a time to allow them to catch up gradually.

If you do not need the historic data then I would suggest cloning the inputs and setting the only_after date to a recent date and then deleting the old input. I dont think it is possible to change the only_after once created because of how the checkpoint of the current date/time is recorded, but I may be wrong here.

🌟 Did this answer help you? If so, please consider:

  • Adding karma to show it was useful
  • Marking it as the solution if it resolved your issue
  • Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

zksvc
Contributor

Thanks for your reply, 
I will try to change the interval time to 600 seconds first. 

0 Karma

zksvc
Contributor

Hi @livehybrid  I have changed the interval to 600 seconds, but the data is still not available. Is there any other solution that you know?

0 Karma
Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.
Get Updates on the Splunk Community!

Tech Talk Recap | Mastering Threat Hunting

Mastering Threat HuntingDive into the world of threat hunting, exploring the key differences between ...

Observability for AI Applications: Troubleshooting Latency

If you’re working with proprietary company data, you’re probably going to have a locally hosted LLM or many ...

Splunk AI Assistant for SPL vs. ChatGPT: Which One is Better?

In the age of AI, every tool promises to make our lives easier. From summarizing content to writing code, ...