Knowledge Management
Highlighted

[SmartStore] Can I configure Splunk SmartStore indexer with multiple object stores

Splunk Employee
Splunk Employee

I'm doing a proof of concept of SmartStore with multiple object stores. There appears to be a defect where the remote.s3.accesskey (and maybe remote.s3.secretkey) is not being properly associated with the volume stanza.

Specifically, in my indexes.conf, I have the following:

[volume:remotestore0]
storageType = remote
path = s3://splunk-ss-01-0

remote.s3.accesskey = [REDACTED0]

remote.s3.secretkey = [REDACTED0]

remote.s3.endpoint = http://xx.xx.xx.xxx>

[volume:remotestore1]
storageType = remote
path = s3://splunk-ss-01-1
remote.s3.accesskey = [REDACTED1]
remote.s3.secretkey = [REDACTED1]
remote.s3.endpoint = http://xx.xx.xx.xxx>

What is happening is that when I try to use remotestore1 the access key for remotestore0 is being used. Note that the endpoint and path are properly associated with the volume specification. It is at least the access_key (and maybe the secret key) that is not being properly associated with the volume stanza.

The bug is particularly annoying since doing splunk cmd splunkd rfs -- ls --starts-with volume:remotestore1 will use the correct access_key that is associated with the volume.

Tags (1)
0 Karma
Highlighted

Re: [SmartStore] Can I configure Splunk SmartStore indexer with multiple object stores

Splunk Employee
Splunk Employee

With Splunk Version 7.3.1 and above will let you configure indexes from an indexer to different smartstore objects , for example below I have configure _internal index to use one smartstore and _audit index to use other one.

===========First Smartstore Configuration=======

[volume:mys3vol]
storageType = remote
path = s3://newrbal1
remote.s3.accesskey = AXXKIAIQWJDOATYCYFTTTTTKWZ5A
remote.s3.secret
key = dCCCCCCCCCCN7rMvSN96RSDDDDYqcKeSSSSi3TcD6YQS8J+EzQI5Qm+Ar9
remote.s3.endpoint = https://s3-us-east-2.amazonaws.com
remote.s3.signature_version = v4

===========Second Smartstore Configuration=======

using AWS S3 storage

[volume:awss3vol]
storageType = remote
path = s3://luantest
remote.s3.accesskey = AKIASVRRRRDSSVCAAAANBVKZXK4T
remote.s3.secret
key = JYD7umcpFFFFHKM4/uq7Wi/rfyUUHdcSFFFz3j2N85bg8wK
remote.s3.endpoint = https://s3-us-east-2.amazonaws.com
remote.s3.signature_version = v4

=============Here index internal is configured with smartstore [volume:mys3vol]=====
[
internal]
thawedPath = $SPLUNKDB/internal/thaweddb
remotePath = volume:awss3vol/$indexname
repFactor = auto

=============Here index internal is configured with smartstore [volume:awss3_vol]=====

[audit]
thawedPath = $SPLUNK
DB/audit/thaweddb
remotePath = volume:my
s3vol/$index_name
repFactor = auto

0 Karma
Highlighted

Re: [SmartStore] Can I configure Splunk SmartStore indexer with multiple object stores

Explorer

i recommend to skip 7.3.1 for smart store migration as it is having a serious bug which will freeze buckets while migration and have a high possible chance to loose your data.

0 Karma
Highlighted

Re: [SmartStore] Can I configure Splunk SmartStore indexer with multiple object stores

Explorer

Does Splunk support multiple S3 object stores configured within the same indexer cluster?

I understand the indexes.conf certainly allows this and I can configure each index to point to specific S3 storage but wanted to get clarified if this is indeed supported by Splunk.

0 Karma
Highlighted

Re: [SmartStore] Can I configure Splunk SmartStore indexer with multiple object stores

Explorer

Yes Splunk support's multiple S3 object stores configured within the same indexer cluster

0 Karma
Highlighted

Re: [SmartStore] Can I configure Splunk SmartStore indexer with multiple object stores

Explorer

Thank you.

0 Karma