Knowledge Management

[SmartStore] How to verify splunk indexer connectivty to remote storage?

Splunk Employee
Splunk Employee

I have configured Splunk Remote storage on indexer. How can I verify connectivity ?

Tags (1)

Splunk Employee
Splunk Employee

Thanks for this @rbal_splunk!

This post helped me with my adventure deploying smartstore on an existing cluster!

Here are some other useful items I found after this got me on track verifying my smart store config was working on 7.2.0:

index=_internal source=*splunkd.log component=S3Client ERROR OR WARN

11-23-2018 01:45:21.790 +0000 WARN  S3Client - command=list transactionId=0x7f7d59d7b200 rTxnId=0x7f7d3f7f9350 status=completed success=N uri= statusCode=502 statusDescription="Error resolving: Name or service not known"

The above search helped find an incorrect path in the remote.s3.endpoint URI...among other config butchering.

Here are my indexers screaming at me for an hour as i bumblefutzed my way through config on my Splunk cluster deployed on Kubernetes:

alt text

index=_internal source=*splunkd.log component=S3client statusCode=*
| timechart span=1m count by statusCode

alt text

   index=_internal source=*splunkd.log component=CacheManager status=*
    | timechart span=5m count by status

alt text

Once I got the URI right ( around 01:30 GMT), the cluster settled down, all the fix-ups were good to go.

Other interesting components in splunkd.log, see docs link above for more:

S3Client - Communication with S3.
StorageInterface - External storage activity (at a higher level than S3Client).
CacheManager - Activity of the cache manger component.
CacheManagerHandler - Cache manager REST endpoint activity (both server and client side).

Side Note: I threw in some feedback for docs, as the smart store example has the user putting repFactor = auto in the default indexes.conf stanza which triggers the bundle validation errors due to replication of _introspection

config I ended up with:

# Configure all indexes to use the SmartStore remote volume called
# "smartstore".
# Note: If you want only some of your indexes to use SmartStore,
# place this setting under the individual stanzas for each of the
# SmartStore indexes, rather than here.

remotePath = volume:smartstore/$_index_name
repFactor = auto

# Configure the remote volume
storageType = remote

# On the next line, the path attribute points to the remote storage location
# where indexes reside. Each SmartStore index resides directly below the location
# specified by the path attribute. The <scheme> identifies a supported remote
# storage system type, such as S3. The <remote-location-specifier> is a
# string specific to the remote storage system that specifies the location
# of the indexes inside the remote system.
# This is an S3 example: "path = s3://mybucket/some/path".

path = s3://somebucket/

# The following S3 settings are required only if you're using the access and secret
# keys. They are not needed if you are using AWS IAM roles.

remote.s3.access_key = someAccessKey
remote.s3.secret_key = someSecretKey
remote.s3.endpoint =

repFactor = 0
- MattyMo

Splunk Employee
Splunk Employee

Just a short comment on the syntax checking in indexes.conf:

Beware of typos in

A capitel "S" like "remote.S3.access_key" will be silently ignored and does not create an error message on startup like "storagetype" in the same file would! 

The above examples are correct, just as a heads up. 

0 Karma

Splunk Employee
Splunk Employee

Assuming the configuration for remote store is done as per splunk documentation, splunk provide cli command to verify connectivity.

1) Verify the remote store configuration for indexex.conf using splunk btool command:

$SPLUNK_HOME/bin/splunk cmd btool indexes list | grep -iE '[|homePath|remotePath'
sample output

homePath = $SPLUNK_DB/defaultdb/db
homePath.maxDataSizeMB = 0
remotePath = volume:my_s3_vol/$_index_name

Note : Verify that remotePath is configured.

2)To check connectivity of indexer to remote storage:

./splunk cmd splunkd rfs -- ls --starts-with volume:remote_store

below is sample output listing all buckets in remote store


3)If you wanted to determine a bucket's location on remote S3 storage, then you will have to SSH to once of the indexers which are hosting this bucket and run the following:

./splunk cmd splunkd rfs -- ls --starts-with bucket:_audit~41~EACDAA22-751B-4DE2-A6A9-73B1AADD4AB7

#for full paths run: splunkd rfs -- ls --starts-with volume:my_s3_vol/_audit/db/0c/52/41~EACDAA22-751B-4DE2-A6A9-73B1AADD4AB7/

4)You can also attempt to retrieve buckets date from S3:

./splunk cmd splunkd rfs -- getF volume:splunkcloud_vol/infra_lb/db/4c/29/178~3A0745AC-F5A5-4FF6-B8EB-70BBBD2F7C87/receipt.json /tmp/

5)If you wanted to determine a bucket's location on remote S3 storage, then you will have to SSH to once of the indexers which are hosting this bucket and run the following:

splunk cmd splunkd rfs -- ls bucket:<bid>

splunk cmd splunkd rfs -- ls bucket:infra_lb~178~3A0745AC-F5A5-4FF6-B8EB-70BBBD2F7C87

$ splunk cmd splunkd rfs -- ls bucket:infra_lb~178~3A0745AC-F5A5-4FF6-B8EB-70BBBD2F7C87
#for full paths run: splunkd rfs -- ls --starts-with volume:splunkcloud_vol/infra_lb/db/4c/29/178~3A0745AC-F5A5-4FF6-B8EB-70BBBD2F7C87/

6)To get help:

$splunk cmd splunkd rfs help

Super Champion
0 Karma
Get Updates on the Splunk Community!

Splunk APM & RUM | Upcoming Planned Maintenance

There will be planned maintenance of the streaming infrastructure for Splunk APM and Splunk RUM in the coming ...

Part 2: Diving Deeper With AIOps

Getting the Most Out of Event Correlation and Alert Storm Detection in Splunk IT Service Intelligence   Watch ...

User Groups | Upcoming Events!

If by chance you weren't already aware, the Splunk Community is host to numerous User Groups, organized ...