This issue is resolved by increasing the max number of results returned per API request. By default (AWS Documentation) the ListFindings maxResults returns 10 results per call. Increasing this small value to a higher one resolved our issue. Python Script to modify:   /opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/inspector/aws_inspector_data_loader.py   And the function to modify:   def _list_findings_by_time_window(self, begin, end): # boto3 do not accept unix timestamp on windows # cast to datetime by hand begin = datetime.datetime.utcfromtimestamp(begin) end = datetime.datetime.utcfromtimestamp(end) params = { "filter": { 'creationTimeRange': { 'beginDate': begin, 'endDate': end } }, "maxResults": 500 }   Resulted in far less API calls and no further rate-limiting. I've put in an enhancement request for this function's defaults to be modified. Hope this helps! API calls before/after change.   ... View more

We did a bunch of math to see how much storage space we'd need to do to determine drive sizes. For our setup/data ingestion we went with a 2 TB SSD (hot/warm) and a 16 TB (cold) storage solution. For the cold drives were using the new AWS cold storage as a cheaper alternative to long storage. We put our cluster in maintenance mode and took down one indexer at a time, did all the following steps and brought it back up. We left the master in maintenance mode until all three indexers had been updated. We also pushed out a new indexes.conf file with the added "homePath.maxDataSizeMB = [number]" stanza to make our warm buckets small enough to fit on our new 2 TB drive. Create 2 disks in the availability zone of the box (please note our COLD drive was cloned from a 8TB snapshot of our existing drive: SSD - 2000 GiB COLD - 16387 GiB (Max size) Create two new mount points: sudo mkdir /mnt/idx-storage-warm sudo mkdir /mnt/idx-storage-cold Determine which drives are where: lslbk - (please note ours were located at /dev/xvdg and /dev/xvdh, please make sure you use the right /dev location on your install) ONLY If your drives are new drives, add a file system. If they are clones of your old data, then DO NOT do this. sudo mkfs -t ext4 /dev/xvdg sudo mkfs -t ext4 /dev/xvdh Mount the drives: sudo mount /dev/xvdg /mnt/idx-storage-warm/ sudo mount /dev/xvdh /mnt/idx-storage-cold/ Modify the /etc/fstab to automount the drives when the box restarts: /dev/xvdg /mnt/idx-storage-warm ext4 defaults,nobootwait 0 0 /dev/xvdh /mnt/idx-storage-cold ext4 defaults,nobootwait 0 0 Migrate the data. The indexes are stored in five different folders based on the type of data there under each of your indexes. This makes for quite the mess. They're located on the filesystem at: /opt/splunk/var/lib/splunk/[name-of-index]/[colddb, thaweddb, db, summary, datamodel_summary]/* 7.5 - We found out later that you should roll your hot buckets to warm before copying them or you'll run into issues. We just deleted all the hot buckets and lost that data, but there's actually a CLI command to roll them all for you so you don't have to do this: More info here from Splunk docs What we ended up doing was snapshotting our existing entire 8TB drive and copying it to the cold storage then rsyncing the warm buckets ONLY (not hot buckets) to the warm drive location. rsync -avpR --progress --exclude '/opt/splunk/var/lib/splunk//colddb/' --exclude '/opt/splunk/var/lib/splunk//thaweddb/' --exclude '/opt/splunk/var/lib/splunk//db/hot*' /opt/splunk/ /mnt/idx-storage-warm/* We needed to resize the cold drive from the 8TB copy to the 16TB new size, which you can do live while mounted: sudo resize2fs /dev/xvdh Modify the splunk-launch.conf file to indicate where splunk should launch from: sudo nano /mnt/idx-storage-warm/opt/splunk/etc/splunk-launch.conf We changed /mnt/idx-storage to /mnt/idx-storage-warm/ for both SPLUNK_HOME and SPLUNK_DB Update the indexes.conf config locally (this would be the one normally pushed by the master), we'll fix this later on in the install. sudo nano /mnt/idx-storage-warm/opt/splunk/etc/slave-apps/_cluster/local/indexes.conf Here's a sample too on our how our config looks. Note how the cold path and thawed path both point to the cold storage and the home path points to the warm storage. Frozen time is our retention period before data is deleted and homepath states how many MB of hot/warm to keep before rolling it to the cold storage (sadly you can only do this by space and not time). [our_data_index] repFactor = auto coldPath = /mnt/idx-storage-cold/opt/splunk/var/lib/splunk/our_data_index/colddb homePath = /mnt/idx-storage-warm/opt/splunk/var/lib/splunk/our_data_index/db thawedPath = /mnt/idx-storage-cold/opt/splunk/var/lib/splunk/our_data_index/thaweddb frozenTimePeriodInSecs = 315569520 homePath.maxDataSizeMB = 5000 Resymlink the new warm storage to the normal splunk install location: sudo rm /opt/splunk sudo ln -s /mnt/idx-storage-warm/opt/splunk/ /opt/splunk Remove the old drive: sudo umount /mnt/idx-storage Update the init.d scripts: sudo /mnt/idx-storage-warm/opt/splunk/bin/splunk disable boot-start sudo /mnt/idx-storage-warm/opt/splunk/bin/splunk enable boot-start -user splunk Fix any permissions that might have gotten screwed up: sudo chown -R splunk:splunk /mnt/idx-storage-cold/opt/splunk sudo chown -R splunk:splunk /mnt/idx-storage-warm/opt/splunk Turn back on indexer, wait patiently (and I do mean patiently) for it to come back up and rejoin the cluster. Then move onto another indexer. Some other cleanup items: 1. Removing unneeded data from cold storage (since we cloned the drive) 2. Removing/Detaching the old 8TB drive. Good luck! Hope this helps! ... View more