topic Re: Splunk smart store S3 bucket config to indexes 7.2.x in All Apps and Add-ons

Splunk smart store S3 bucket config to indexes 7.2.x

Splunk_citizen — Tue, 29 Sep 2020 22:25:39 GMT

Hello Splunkers,

Any one had luck to test for connective from indexer host to send data to your AWS S3 one ? I have tried with below settings to setup frozen volume but its giving error.

[volume:frozen]
storageType = remote

path = s3://example-s3-bucket/remote_volume

path = s3://xyz-splunk-bb/frozen
remote.s3.access_key = XXX
remote.s3.secret_key = YYYY
[default]
remotePath = volume:s3/frozen

Re: Splunk smart store S3 bucket config to indexes 7.2.x

myriadic — Thu, 20 Dec 2018 18:15:04 GMT

you shouldn't have "s3/" in front of "frozen" in volume:s3/frozen

also, remotePath needs an index name

Re: Splunk smart store S3 bucket config to indexes 7.2.x

Splunk_citizen — Tue, 29 Sep 2020 22:26:58 GMT

Thank you but i have tried below ways

[volume:frozen]
storageType = remote

path = s3://example-s3-bucket/remote_volume

path = s3://xyz-splunk-bb/frozen
remote.s3.access_key = XXX
remote.s3.secret_key = YYYY
[default]
remotePath = volume:s3/frozen

[index1]
homePath = volume:frozen/index1/db
coldPath = volume:frozen/index1/colddb
thawedPath = volume:frozen/index1/thaweddb

but no luck ...

Re: Splunk smart store S3 bucket config to indexes 7.2.x

myriadic — Tue, 29 Sep 2020 22:27:05 GMT

if you have "remotePath = volume:s3/frozen" under [default], your "index1" is going to have:

remotePath = volume:s3/frozen

that makes no sense.

replace "remotePath = volume:s3/frozen" with "remotePath = volume:frozen/$_index_name"

Re: Splunk smart store S3 bucket config to indexes 7.2.x

Splunk_citizen — Thu, 20 Dec 2018 19:04:17 GMT

Thanks i have tried all approaches if you have working settings please post it here.

Problem parsing indexes.conf: Cannot load IndexConfig: idx=index1 param=homePath Remote volume path specification is only valid for parameter remotePath

Re: Splunk smart store S3 bucket config to indexes 7.2.x

myriadic — Tue, 29 Sep 2020 22:27:08 GMT

[default]
homePath = $SPLUNK_DB/$_index_name/db
thawedPath = $SPLUNK_DB/$_index_name/thaweddb
coldPath = $SPLUNK_DB/$_index_name/colddb
remotePath = volume:remote_store/$_index_name

[os]
homePath = $SPLUNK_DB/$_index_name/db
thawedPath = $SPLUNK_DB/$_index_name/thaweddb
coldPath = $SPLUNK_DB/$_index_name/colddb
remotePath = volume:remote_store/$_index_name

Re: Splunk smart store S3 bucket config to indexes 7.2.x

Splunk_citizen — Thu, 20 Dec 2018 19:28:50 GMT

Thank you for your quick reply but how data will be written in remote path based on ? what condition ?
I just want to create my new index and all data to be in only S3 bucket hot warm and frozen data or else just want to send frozen data to remote path