Hi community, asking for your help and trying to make to you clear the context. We are ingesting access log about an AWS Elastic Load Balance from an S3 bucket by the Splunk AWS app.
It happens, in some specific time window, that the S3 log volume is high, this make the indexer consume a lot of resources, then creating other kind of issue in terms of performance.
Due the fact the ingestion of this log is not required to be "realtime", we would like to evaluate the case to ingest this ELB logs from S3 with a sort of delay,let say for example:
-ingest from that source just during the night
-control the ingestion troughput for that specific source
-ingest the log with a delay of 1 day (today the log of yestarday)
Could you please suggest to us some configuration or possibility we can look at?
We only find the polling period parameter from "http://docs.splunk.com/Documentation/AddOns/released/AWS/ConfigRules", it can work cause allow to control the frequency, but not the phase..so there is a risk that for some reason the ingestion from that source can occur is some time period for which it is not preferred.
Thanks in advance for support you can provide.
Regards.
Andrea
