Getting Data In

Index Performance

jmc94
Loves-to-Learn

Hi,

We have an index that is feeding in data from an EKS/K8s infrastructure and getting roughly 4million events / 15 minutes (during peak). The index is doing roughly 80GB/day.

Running queries on the data works great if you search within the current day however running historical searches on the data even using the proper fields specific to what I want to search for takes a very long time and the load on my indexers shoots up very high.

I have not modified any of the index params for this index in indexes.conf. This is a smartstore index and I have roughly 500GB of cache setup for caching locally. If anyone could let me know what tweaks might be best for this it would be greatly appreciated. 

Labels (2)
0 Karma

scelikok
SplunkTrust
SplunkTrust

Hi @jmc94,

Since you have problem only on historical searches, it show that eviction and downloading the buckets from SmartStore takes time. You can check if there is a bandwidth limitation issue between indexers and S3 compatible storage.

Please be sure that your maxDataSize is auto as recommended. If you are using as auto_high_volume it will take much more time on downloading from SmartStore. 

Also if the storage is a kind of Scale-out NAS solution, 6k IOPS shown in tests does not work with the way Splunk uses S3. You can check download actions and durations/sizes from internal logs.

index=_internal component=CacheManager

 

If this reply helps you an upvote and "Accept as Solution" is appreciated.
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @jmc94,

having so many data it's possible to have delays in answers with old data.

The first question is: what performances have you on that storage?

Have you at least 800 IOPS (better 1200) both on the storage for hot and warm data and also for cold data?

Did you tried to accelerate your searches (using Summary indexes or Accelerated DataModels)?

Ciao.

Giuseppe

0 Karma

jmc94
Loves-to-Learn

Yes we have roughly 6k IOPS available on the backend storage, we have 3 indexers currently. We have not tried Summary indexes or Accelerated DataModels as of yet. 

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @jmc94,

If you have a correct infrastructure, having so many events, the only way is accelere your searches.

Ciao.

Giuseppe

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...