Cloud trail was writing to an s3 bucket that just kept filling up and was never cleaned up. We wrote an s3 lifecycle policy you delete things older than x days. The aws ta was scanning the entire bucket and just couldn’t keep up once the bucket got over a certain size. ... View more

when searched for data out of (hotlist_recency_secs period ) the indexer will get the required warm/cold buckets from remote storage and stores in local cache , so it is recommended to have SSD's for your local storage for better cache management and performance. ... View more

well explained. ... View more

I don't really see anything wrong with your seach try adding _time to sistats. Also try using the bucket command, it will allow you create a more accurate hour based summary. Try adding the follwing in your Summary search. Summary search (run hourly): index=dns sourcetype=dnsdebuglog | bucket span=1h _time| sistats count by _time, host Scheduled summary search(I assume summary_dnsdebuglog is the name of your saved search): index=summary search_name=summary_dnsdebuglog | stats count by orig_host | sort -count ... View more

This is probably a good place to start: http://docs.splunk.com/Documentation/Splunk/latest/Admin/Wheretofindtheconfigurationfiles Also, you need to know that truncation of events take place in the parsing phase, which can happen on either a heavy forwarder or an indexer, so there is no need to push such configs to a Universal forwarder. For more information on that subject, see; http://wiki.splunk.com/Where_do_I_configure_my_Splunk_settings Hope this helps, Kristian ... View more