Splunk Enterprise

Splunk Indexes data pointing to different storage

elend
Path Finder

Hi there,

I want to point the secondary storage of my splunk indexer to mix with another storage, like point it to cloud storage?

so it will like this one is the common:

[volume:hot1]
path = /mnt/fast_disk
maxVolumeDataSizeMB = 100000

[volume:s3volume]
storageType = remote
path = s3://<bucketname>/rest/of/path

 

is there a mechanism or reference to did this?

Labels (3)
0 Karma

PrewinThomas
Builder

@elend 

As @richgalloway  mentioned,
This is exactly what SmartStore is designed for.

Hot buckets stay on local disk for fast ingestion and search and warm buckets are offloaded to remote storage (e.g., S3).


#https://docs.splunk.com/Documentation/SVA/current/Architectures/SmartStore


Regards,
Prewin
Splunk Enthusiast | Always happy to help! If this answer helped you, please consider marking it as the solution or giving a Karma. Thanks!

0 Karma

elend
Path Finder

Oh thankyou. so thats just point the bucket like the sample right?

0 Karma

livehybrid
Ultra Champion

Hi @elend 

You configure a volume in your indexes.conf which is your s3 location essentially, and then you can update all or individual indexes to use that volume by setting the remotePath eg 

remotePath = volume:<VOLUME_NAME>/$_index_name

the $_index_name is actually an internal variable so you don’t need to overwrite this.
in addition to the other docs I posted on the previous post it’s worth checking https://docs.splunk.com/Documentation/Splunk/9.4.2/Indexer/SmartStoresecuritystrategies too. 

 

 

:glowing_star: Did this answer help you? If so, please consider:

    • Adding karma to show it was useful
    • Marking it as the solution if it resolved your issue
    • Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing.


 

0 Karma

livehybrid
Ultra Champion

Hi @elend 

Yes, you can configure Splunk (since 7.2 I think) to use mixture of local storage and S3-compliant storage, including the likes of Amazon S3 using Splunk's SmartStore functionality, this essentially uses your local storage for hot buckets and as a local cache for buckets which are also stored in S3. Its more of a complex beast than I can go into here, and there are lots of things to consider - for example this is generally considered a one-way exercise!  

https://docs.splunk.com/Documentation/SVA/current/Architectures/SmartStore gives a good overview of the architecture, benefits and next steps.

Check out https://help.splunk.com/en/splunk-enterprise/administer/manage-indexers-and-indexer-clusters/9.3/man... for more info on setting up smartstore as well as https://help.splunk.com/en/splunk-enterprise/administer/manage-indexers-and-indexer-clusters/9.4/dep... which has some info on setting this up on a single indexer (as a starter, this will depend on your specific environment architecture).

:glowing_star: Did this answer help you? If so, please consider:

  • Adding karma to show it was useful
  • Marking it as the solution if it resolved your issue
  • Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

0 Karma

richgalloway
SplunkTrust
SplunkTrust

You've just stumbled across SmartStore (S2).  S2 keeps hot buckets local and copies warm buckets to S3.  A cache of roughly 30 days of data is retained locally for faster search performance.

To implement S2 correctly, see https://docs.splunk.com/Documentation/Splunk/9.4.2/Indexer/AboutSmartStore

---
If this reply helps you, Karma would be appreciated.
0 Karma
Get Updates on the Splunk Community!

See just what you’ve been missing | Observability tracks at Splunk University

Looking to sharpen your observability skills so you can better understand how to collect and analyze data from ...

Weezer at .conf25? Say it ain’t so!

Hello Splunkers, The countdown to .conf25 is on-and we've just turned up the volume! We're thrilled to ...

How SC4S Makes Suricata Logs Ingestion Simple

Network security monitoring has become increasingly critical for organizations of all sizes. Splunk has ...