Deployment Architecture

Splunk smart store S3 config indexes to send data to remote location

Splunk_citizen
Explorer

Hello Splunkers,

i was just doing some POC to send data from our on prem Splunk indexer to AWS s3 bucket as they added new features in 7.2.X

[volume:frozen]
storageType = remote

path = s3://example-s3-bucket/remote_volume

path = s3://xyz-splunk-bb/frozen
remote.s3.access_key = XXX
remote.s3.secret_key = YYYY

But still i did not see any data written on remote location any thoughts ?

woodcock
Esteemed Legend

Leave specific feedback on the docs page. They will get back to you quickly with answers.

0 Karma

p_gurav
Champion

Can you try this :

[volume:s3]
storageType = remote
path = s3://xyz-splunk-bb/frozen
remote.s3.access_key = XXX
remote.s3.secret_key = YYY

[index1]
remotePath = volume:s3/$_index_name
0 Karma

Splunk_citizen
Explorer

got this error Problem parsing indexes.conf: Cannot load IndexConfig: idx=index1 param=homePath Remote volume path specification is only valid for parameter remotePath
Validating databases (splunkd validatedb) failed with code '1'. If you cannot resolve the issue(s) above after consulting documentation, please file a case online at http://www.splunk.com/page/submit_issue

0 Karma

harsmarvania57
Ultra Champion

This was recently discussed on Slack (indexer-clustering channel) and configuration for index should be like this

[index1]
remotePath = volume:s3/$_index_name
homePath = $SPLUNK_DB/$_index_name/db
coldPath = $SPLUNK_DB/$_index_name/colddb
thawedPath = $SPLUNK_DB/$_index_name/thaweddb
0 Karma

cpharvey
Explorer

typo in voLume ?

0 Karma

harsmarvania57
Ultra Champion

Yes, corrected it. Thanks 🙂

0 Karma

Splunk_citizen
Explorer

I have tried below one

[index1]
homePath = volume:frozen/index1/db
coldPath = volume:frozen/index1/colddb
thawedPath = volume:frozen/index1/thaweddb

keep on getting below error
Problem parsing indexes.conf: Cannot load IndexConfig: idx=index1 param=homePath Remote volume path specification is only valid for parameter remotePath
Validating databases (splunkd validatedb) failed with code '1'. If y

0 Karma
Get Updates on the Splunk Community!

Monitoring MariaDB and MySQL

In a previous post, we explored monitoring PostgreSQL and general best practices around which metrics to ...

Financial Services Industry Use Cases, ITSI Best Practices, and More New Articles ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Splunk Federated Analytics for Amazon Security Lake

Thursday, November 21, 2024  |  11AM PT / 2PM ET Register Now Join our session to see the technical ...