All Apps and Add-ons

has anyone successful setup the remotePath option in indexes.conf in Splunk 7.0 to work with indexed data in s3?

Log_wrangler
Builder

I was trying to search copies of indexed data in S3.

Has anyone had luck with this scenario using remotePath??? I know it says not supported but is it functional at this point?

indexes.conf for splunk 7.0
[https://docs.splunk.com/Documentation/Splunk/7.0.1/Admin/Indexesconf]

remotePath =
* Currently not supported. This setting is related to a feature that is
still under development.
* Optional.
* Presence of this parameter means that this index uses remote storage, instead
of the local file system, as the main repository for bucket storage. The
index processor works with a cache manager to fetch buckets locally, as
necessary, for searching and to evict them from local storage as space fills
up and they are no longer needed for searching.
* This setting must be defined in terms of a storageType=remote volume
definition. See the volume section below.
* The path portion that follows the volume reference is relative to the path
specified for the volume. For example, if the path for a volume "v1" is
"s3://bucket/path" and "remotePath" is "volume:v1/idx1", then the fully
qualified path will be "s3://bucket/path/idx1". The rules for resolving the
relative path with the absolute path specified in the volume can vary
depending on the underlying storage type.
* If "remotePath" is specified, the "coldPath" and "thawedPath" attributes are
ignored. However, they still must be specified.

Any advise or lessons learned is appreciated.

Thank you

0 Karma

nickhills
Ultra Champion
remotePath = <root path for remote volume, prefixed by a URI-like scheme>

Currently not supported. This setting is related to a feature that is
still under development.

Even if i knew, I couldn't tell you!
:)

My guess is that its to allow for cloud storage like S3 - hopefully for archive/frozen data - but thats entirely a guess
But i am watching the releases very closely for more info

If my comment helps, please give it a thumbs up!

nickhills
Ultra Champion

Just a follow up comment - I very much doubt that this is going to allow you to store hot/warm or even cold data in s3 - the read performance isn't up to the job, and the write mechanism would be far too clunky for indexers.

If my comment helps, please give it a thumbs up!
0 Karma
Get Updates on the Splunk Community!

Announcing Scheduled Export GA for Dashboard Studio

We're excited to announce the general availability of Scheduled Export for Dashboard Studio. Starting in ...

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics GA in US-AWS!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...