Splunk Enterprise

S3 indexes.conf issues

anuragschandra
Observer

Hey Guys

We are trying to configure Splunk with S3 and facing issues : 

Have a few questions :

1) what should be under 

Configure the remote volume
We have storageType:remote 

what does [volume:s3] signify? 

2) Do the entries below suffice ?

storageType = remote
path = s3://splunk-smartstore/indexes
remote.s3.supports_versioning = false
remote.s3.endpoint = http://<IP-address>/splunk-smartstore
remote.s3.access_key = <Access_key>
remote.s3.secret_key = <secrey key>

 

We keep seeing the following errors :

/opt/splunk/etc/master-apps/_cluster/local]# /opt/splunk/bin/./splunk cmd splunkd rfs -- ls
error: <remote_id> expected
error: operation failed; check log for details

What log file can help debugging this ?

Labels (2)
0 Karma

isoutamo
SplunkTrust
SplunkTrust
What is your environment: AWS, onprem or mixed or Azure? If onprem, what are your S3 storage, nodes, networks etc?
R. Ismo
0 Karma

anuragschandra
Observer

This is on prem and S3 compatible storage.

Tags (1)
0 Karma

isoutamo
SplunkTrust
SplunkTrust

Ok, what is the storage, it’s peak capacity, network capacity on nodes, middle and storage side?

Also your daily indexing volume and search profile?

0 Karma

inventsekar
Ultra Champion

Hi @anuragschandra ... 

https://docs.splunk.com/Documentation/Splunk/8.2.1/admin/Indexesconf#indexes.conf.example

### This example demonstrates how to configure a volume that points to
### S3-based remote storage and indexes that use this volume.  The setting
### "storageType=remote" indicates that this is a remote-storage volume.
### The "remotePath" parameter associates the index with that volume
### and configures a top-level location for uploading buckets.

[volume:s3]
storageType = remote
path = s3://remote_volume
remote.s3.bucket_name = example-s3-bucket
remote.s3.access_key = S3_ACCESS_KEY
remote.s3.secret_key = S3_SECRET_KEY

also pls check this 

https://docs.splunk.com/Documentation/Splunk/8.2.1/Indexer/ConfigureremotestoreforSmartStore

also this page has got some good details on indexes.conf for S3:ac

https://blog.arcusdata.io/how-to-set-up-splunk-smart-store-in-aws

 

 

0 Karma

anuragschandra
Observer

Still confused with what remote volume needs to have ?

Can somebody lay out step by step whats needed on the Storage side 

Here is my assumption :

1) S3 bucket 

2) Access ID

3) Secret Key 

 

What does [volume:s3] signify ? is s3 a folder inside the s3 bucket ?

Also , what log file should we look at for failures?

 

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...