Hi Everyone,
I encountered an error while ingesting sourcetype=aws:cloudtrails in AWS Apps. I attempted to ingest data from the following sources: aws:waflogs, aws:network-firewall-log, aws:cloudtrails, aws:securityhub-log-group. However, upon checking, only aws:waflogs and aws:network-firewall-log were ingested. Attached below are the errors from the logs.
Also i screenshot inputs config from the apps side here :
Last i show you the proof if i only received that 2 sourctypes here :
If you have any experience from this issue, please give me the answer.
Danke,
Zake
Hi @zksvc
It looks like the inputs are polling AWS Cloudwatch too frequently, which is giving your Rate Limit exception.
If you have just set this up then it will be trying to pull logs back from whatever the only_after date you set was (see https://splunk.github.io/splunk-add-on-for-amazon-web-services/CloudWatchLogs/ for input config descriptions)
If you left this field blank then I believe it tries to load all the events in the Cloudwatch logs group in AWS. Ultimately it looks like its repeatedly querying CW Logs to get more logs which is why it is hitting the rate limit. The number of polls to CW Logs will reduce once it has caught up to the current date. It might be worth enabling one at a time to allow them to catch up gradually.
If you do not need the historic data then I would suggest cloning the inputs and setting the only_after date to a recent date and then deleting the old input. I dont think it is possible to change the only_after once created because of how the checkpoint of the current date/time is recorded, but I may be wrong here.
🌟 Did this answer help you? If so, please consider:
Your feedback encourages the volunteers in this community to continue contributing
Thanks for your reply,
I will try to change the interval time to 600 seconds first.
Hi @livehybrid I have changed the interval to 600 seconds, but the data is still not available. Is there any other solution that you know?