Knowledge Management

SmartStore S3 data replication

goldeneye1117
New Member

I have been testing out SmartStore in a test environment. I can not find the setting to control how quickly data ingested into splunk can be replicated to my S3 bucket. What I want is for any data ingested to be replicated to my s3 bucket as quickly as possible, I am looking for the closest to 0 minutes of data loss. Data only seems to replicate when the Splunk server is restarted. I have tested this by setting up another splunk server with the same s3 bucket as my original, and it seems to have only picked up older data when searching. 

 

max_cache_size

 

only controls the size of the local cache which I'm not after

 

hotlist_recency_secs

 

controls how long before hot data could be deleted from cache, not how long before it is replicated to s3

 

frozenTimePeriodInSecs, maxGlobalDataSizeMB,  maxGlobalRawDataSizeMB 

 

controls freezing behavior which is not what I'm looking for.

What setting do I need to configure? Am I missing something within conf files in Splunk or permissions to set in AWS for S3? 

Thank you for the help in advance!

Labels (1)
Tags (1)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

Data is written to SmartStore (S2) as soon as it rolls to warm.  On a test system, it's typical for hot buckets to not roll to warm until the indexers restart.  On a production system, however, that should happen at least once a day.  Hot buckets are never written to S2.  There is no setting to give you instant replication to S2.  In an indexer cluster, hot buckets are replicated to other indexers almost immediately.

---
If this reply helps you, Karma would be appreciated.
0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Meet Splunk Observability Studio: AI-Assisted OpenTelemetry Instrumentation Without ...

Instrumentation is usually the last step or even an afterthought when building out a project. The feature ...

Federated Search for Cisco Security and Analytics Logging (SAL) is now GA on Splunk ...

Federated Search for Cisco  Security Analytics and Logging (SAL) is now generally available as part of the ...

Your Path to AgenticOps: AI Experiences for Every Splunk Practitioner

Your Path to AgenticOps: AI Experiences for Every Splunk Practitioner   Join us for a demo-driven look at how ...