Thanks for this @rbal_splunk!
This post helped me with my adventure deploying smartstore on an existing cluster!
Here are some other useful items I found after this got me on track verifying my smart store config was working on 7.2.0:
index=_internal source=*splunkd.log component=S3Client ERROR OR WARN
11-23-2018 01:45:21.790 +0000 WARN S3Client - command=list transactionId=0x7f7d59d7b200 rTxnId=0x7f7d3f7f9350 status=completed success=N uri=https://s3.ca-central-1.amazonaws.com/mattymo/thisShouldntBeInTheURI statusCode=502 statusDescription="Error resolving: Name or service not known"
The above search helped find an incorrect path in the remote.s3.endpoint URI...among other config butchering.
Here are my indexers screaming at me for an hour as i bumblefutzed my way through config on my Splunk cluster deployed on Kubernetes:
index=_internal source=*splunkd.log component=S3client statusCode=*
| timechart span=1m count by statusCode
index=_internal source=*splunkd.log component=CacheManager status=*
| timechart span=5m count by status
Once I got the URI right ( around 01:30 GMT), the cluster settled down, all the fix-ups were good to go.
http://docs.splunk.com/Documentation/Splunk/latest/Indexer/TroubleshootSmartStore
Other interesting components in splunkd.log, see docs link above for more:
S3Client - Communication with S3.
StorageInterface - External storage activity (at a higher level than S3Client).
CacheManager - Activity of the cache manger component.
CacheManagerHandler - Cache manager REST endpoint activity (both server and client side).
Side Note: I threw in some feedback for docs, as the smart store example has the user putting repFactor = auto in the default indexes.conf stanza which triggers the bundle validation errors due to replication of _introspection
config I ended up with:
[default]
# Configure all indexes to use the SmartStore remote volume called
# "smartstore".
# Note: If you want only some of your indexes to use SmartStore,
# place this setting under the individual stanzas for each of the
# SmartStore indexes, rather than here.
remotePath = volume:smartstore/$_index_name
repFactor = auto
# Configure the remote volume
[volume:smartstore]
storageType = remote
# On the next line, the path attribute points to the remote storage location
# where indexes reside. Each SmartStore index resides directly below the location
# specified by the path attribute. The <scheme> identifies a supported remote
# storage system type, such as S3. The <remote-location-specifier> is a
# string specific to the remote storage system that specifies the location
# of the indexes inside the remote system.
# This is an S3 example: "path = s3://mybucket/some/path".
path = s3://somebucket/
# The following S3 settings are required only if you're using the access and secret
# keys. They are not needed if you are using AWS IAM roles.
remote.s3.access_key = someAccessKey
remote.s3.secret_key = someSecretKey
remote.s3.endpoint = https://s3.ca-central-1.amazonaws.com/
[_introspection]
repFactor = 0
