Deployment Architecture

Error Ingest Data AWS Cloudtrail "error=Traceback (most recent call last):"

zksvc
Communicator

Hi Everyone, 

I encountered an error while ingesting sourcetype=aws:cloudtrails in AWS Apps. I attempted to ingest data from the following sources: aws:waflogs, aws:network-firewall-log, aws:cloudtrails, aws:securityhub-log-group. However, upon checking, only aws:waflogs and aws:network-firewall-log were ingested. Attached below are the errors from the logs. 

zksvc_0-1749639515584.png

Also i screenshot inputs config from the apps side here : 

zksvc_2-1749639584109.png

Last i show you the proof if i only received that 2 sourctypes here : 

zksvc_3-1749639668960.png

 

If you have any experience from this issue, please give me the answer. 

 

Danke,

 

Zake

 

Labels (3)
0 Karma

livehybrid
Super Champion

Hi @zksvc 

It looks like the inputs are polling AWS Cloudwatch too frequently, which is giving your Rate Limit exception. 

If you have just set this up then it will be trying to pull logs back from whatever the only_after date you set was (see https://splunk.github.io/splunk-add-on-for-amazon-web-services/CloudWatchLogs/ for input config descriptions)

If you left this field blank then I believe it tries to load all the events in the Cloudwatch logs group in AWS. Ultimately it looks like its repeatedly querying CW Logs to get more logs which is why it is hitting the rate limit. The number of polls to CW Logs will reduce once it has caught up to the current date. It might be worth enabling one at a time to allow them to catch up gradually.

If you do not need the historic data then I would suggest cloning the inputs and setting the only_after date to a recent date and then deleting the old input. I dont think it is possible to change the only_after once created because of how the checkpoint of the current date/time is recorded, but I may be wrong here.

🌟 Did this answer help you? If so, please consider:

  • Adding karma to show it was useful
  • Marking it as the solution if it resolved your issue
  • Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

zksvc
Communicator

Thanks for your reply, 
I will try to change the interval time to 600 seconds first. 

0 Karma

zksvc
Communicator

Hi @livehybrid  I have changed the interval to 600 seconds, but the data is still not available. Is there any other solution that you know?

0 Karma
Get Updates on the Splunk Community!

Splunk Answers Content Calendar, July Edition I

Hello Community! Welcome to another month of Community Content Calendar series! For the month of July, we will ...

Secure Your Future: Mastering Upgrade Readiness for Splunk 10

Spotlight: The Splunk Health Assistant Add-On  The Splunk Health Assistant Add-On is your ultimate companion ...

Observability Unlocked: Kubernetes & Cloud Monitoring with Splunk IM

Ready to master Kubernetes and cloud monitoring like the pros? Join Splunk’s Growth Engineering team on ...