Why are indexers unable to start?

[damode1]

My Indexer cluster with smartstore was working fine with the below config

[default]
remotePath = volume:remote_store/$_index_name
repFactor = auto

# Configure the remote volume.
[volume:remote_store]
storageType = remote
path = s3://splunk_data/

However, suddenly now its unable to start and giving the following error on all indexers when Splunk starts due to which Splunk is unable to start

Problem parsing indexes.conf: Cannot load IndexConfig: Unable to load remote volume "remote_store" of scheme "s3" referenced by index "_audit": Could not get s3 region from the metadata endpoint
Validating databases (splunkd validatedb) failed with code '1'.  If you cannot resolve the issue(s) above after consulting documentation, please file a case online at http://www.splunk.com/page/submit_issue

To fix the above issue, I added below attribute to indexes.conf  in the hopes that it will get the region

remote.s3.endpoint = https://s3.<region_name>.amazonaws.com

but after that I am getting the below error

Problem parsing indexes.conf: Cannot load IndexConfig: Unable to load remote volume "remote_store" of scheme "s3" referenced by index "_audit": Could not find access_key and/or secret_key in a configuration file, in environment variables or via the AWS metadata endpoint.
Validating databases (splunkd validatedb) failed with code '1'.  If you cannot resolve the issue(s) above after consulting documentation, please file a case online at http://www.splunk.com/page/submit_issue

can someone please help fix this issue ?

[richgalloway]

Make sure the S3 bucket is still accessible by the indexer - no firewalls blocking connections, access keys haven't changed, AWS security hasn't changed, etc.  You may need to add remote.s3.access_key and remote.s3.secret_key settings to indexes.conf.

Please share the _audit stanza from indexes.conf (sanitized).