Splunk Enterprise

S3 indexes.conf issues

anuragschandra
Observer

Hey Guys

We are trying to configure Splunk with S3 and facing issues : 

Have a few questions :

1) what should be under 

Configure the remote volume
We have storageType:remote 

what does [volume:s3] signify? 

2) Do the entries below suffice ?

storageType = remote
path = s3://splunk-smartstore/indexes
remote.s3.supports_versioning = false
remote.s3.endpoint = http://<IP-address>/splunk-smartstore
remote.s3.access_key = <Access_key>
remote.s3.secret_key = <secrey key>

 

We keep seeing the following errors :

/opt/splunk/etc/master-apps/_cluster/local]# /opt/splunk/bin/./splunk cmd splunkd rfs -- ls
error: <remote_id> expected
error: operation failed; check log for details

What log file can help debugging this ?

Labels (2)
0 Karma

isoutamo
SplunkTrust
SplunkTrust
What is your environment: AWS, onprem or mixed or Azure? If onprem, what are your S3 storage, nodes, networks etc?
R. Ismo
0 Karma

anuragschandra
Observer

This is on prem and S3 compatible storage.

Tags (1)
0 Karma

isoutamo
SplunkTrust
SplunkTrust

Ok, what is the storage, it’s peak capacity, network capacity on nodes, middle and storage side?

Also your daily indexing volume and search profile?

0 Karma

inventsekar
SplunkTrust
SplunkTrust

Hi @anuragschandra ... 

https://docs.splunk.com/Documentation/Splunk/8.2.1/admin/Indexesconf#indexes.conf.example

### This example demonstrates how to configure a volume that points to
### S3-based remote storage and indexes that use this volume.  The setting
### "storageType=remote" indicates that this is a remote-storage volume.
### The "remotePath" parameter associates the index with that volume
### and configures a top-level location for uploading buckets.

[volume:s3]
storageType = remote
path = s3://remote_volume
remote.s3.bucket_name = example-s3-bucket
remote.s3.access_key = S3_ACCESS_KEY
remote.s3.secret_key = S3_SECRET_KEY

also pls check this 

https://docs.splunk.com/Documentation/Splunk/8.2.1/Indexer/ConfigureremotestoreforSmartStore

also this page has got some good details on indexes.conf for S3:ac

https://blog.arcusdata.io/how-to-set-up-splunk-smart-store-in-aws

 

 

0 Karma

anuragschandra
Observer

Still confused with what remote volume needs to have ?

Can somebody lay out step by step whats needed on the Storage side 

Here is my assumption :

1) S3 bucket 

2) Access ID

3) Secret Key 

 

What does [volume:s3] signify ? is s3 a folder inside the s3 bucket ?

Also , what log file should we look at for failures?

 

0 Karma
Get Updates on the Splunk Community!

How to Monitor Google Kubernetes Engine (GKE)

We’ve looked at how to integrate Kubernetes environments with Splunk Observability Cloud, but what about ...

Index This | How can you make 45 using only 4?

October 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this ...

Splunk Education Goes to Washington | Splunk GovSummit 2024

If you’re in the Washington, D.C. area, this is your opportunity to take your career and Splunk skills to the ...