DMC Alert - Near Critical Disk Usage

[nnimbe1]

Hi All, 

We are getting DMC Alert - Near Critical Disk Usage on all our four indexers, we are running Splunk on distributed environment on AWS windows servers, and now due AWS disk space 16 TB limit, we are unable to extend the drive, can you please suggest suitable solution for this issue.

[nickhills]

Assuming that you can't free space (by freezing events)...

If you need more storage, you need more disk. If you can't make the disks any bigger, then you need more of them.

You have some options:

- Add more indexers - This is the simplest way to add capacity. Once you have added new peer(s), rebalance your cluster and after the process completes you will have (approximately) even distribution of data across all your indexers. https://docs.splunk.com/Documentation/Splunk/8.1.0/Indexer/Addclusterpeer

- Add S3 Storage. If you are already in AWS, you could consider using SmartStore. This stores your oldest data in AWS S3, keeping your more recent data cached locally on your physical [sic] volumes. Unless you frequently access old data the performance impact is negligible if you tune things properly. https://docs.splunk.com/Documentation/Splunk/8.1.0/Indexer/AboutSmartStore

- [a potentially unhelpful remark about moving to linux was here. Self censored]

If it's an option, you can modify your SF/RF settings to reduce the number of duplicate copies of data to buy some time. Make sure you understand the implications of this first.