Splunk Enterprise

S3 indexes.conf issues

anuragschandra
Observer

Hey Guys

We are trying to configure Splunk with S3 and facing issues : 

Have a few questions :

1) what should be under 

Configure the remote volume
We have storageType:remote 

what does [volume:s3] signify? 

2) Do the entries below suffice ?

storageType = remote
path = s3://splunk-smartstore/indexes
remote.s3.supports_versioning = false
remote.s3.endpoint = http://<IP-address>/splunk-smartstore
remote.s3.access_key = <Access_key>
remote.s3.secret_key = <secrey key>

 

We keep seeing the following errors :

/opt/splunk/etc/master-apps/_cluster/local]# /opt/splunk/bin/./splunk cmd splunkd rfs -- ls
error: <remote_id> expected
error: operation failed; check log for details

What log file can help debugging this ?

Labels (2)
0 Karma

isoutamo
SplunkTrust
SplunkTrust
What is your environment: AWS, onprem or mixed or Azure? If onprem, what are your S3 storage, nodes, networks etc?
R. Ismo
0 Karma

anuragschandra
Observer

This is on prem and S3 compatible storage.

Tags (1)
0 Karma

isoutamo
SplunkTrust
SplunkTrust

Ok, what is the storage, it’s peak capacity, network capacity on nodes, middle and storage side?

Also your daily indexing volume and search profile?

0 Karma

inventsekar
Super Champion

Hi @anuragschandra ... 

https://docs.splunk.com/Documentation/Splunk/8.2.1/admin/Indexesconf#indexes.conf.example

### This example demonstrates how to configure a volume that points to
### S3-based remote storage and indexes that use this volume.  The setting
### "storageType=remote" indicates that this is a remote-storage volume.
### The "remotePath" parameter associates the index with that volume
### and configures a top-level location for uploading buckets.

[volume:s3]
storageType = remote
path = s3://remote_volume
remote.s3.bucket_name = example-s3-bucket
remote.s3.access_key = S3_ACCESS_KEY
remote.s3.secret_key = S3_SECRET_KEY

also pls check this 

https://docs.splunk.com/Documentation/Splunk/8.2.1/Indexer/ConfigureremotestoreforSmartStore

also this page has got some good details on indexes.conf for S3:ac

https://blog.arcusdata.io/how-to-set-up-splunk-smart-store-in-aws

 

 

0 Karma

anuragschandra
Observer

Still confused with what remote volume needs to have ?

Can somebody lay out step by step whats needed on the Storage side 

Here is my assumption :

1) S3 bucket 

2) Access ID

3) Secret Key 

 

What does [volume:s3] signify ? is s3 a folder inside the s3 bucket ?

Also , what log file should we look at for failures?

 

0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!