Knowledge Management

How does creating a data model affect storage and memory?

packet_hunter
Contributor

I am asking this question as I dig thru the documentation.

Currently I don't have a lot of reserve disk storage or indexing/license reserve in my deployment.

If I start creating and testing different data models, how do I predict how much disk space and indexing the data model will consume?

I will admit I have not taken any of the admin courses yet.

Any advice on this is appreciated.

Thank you

0 Karma
1 Solution

shaskell_splunk
Splunk Employee
Splunk Employee

The answer is that it depends. When a user interacts with an un-accelerated data model it will be accelerated via ad hoc data model acceleration. A summary will be built in the dispatch directory of the search head and will take up some disk space but it will only persist for that session. Once a user navigates away from pivot, it will go away.

If you accelerate the data model, the acceleration will live on the indexers in buckets next to the raw data. How much space it uses will depend what is in the data model and the period of acceleration. I'd suggest building a data model, accelerating it and then inspecting the acceleration to see how much space it is using. There will also be CPU cycles spent to update the acceleration in the background.

Be aware of the caveats around data model acceleration. You can only accelerate a root event object hierarchy.

Here is a good break-down of the differences between ad hoc acceleration and persistent acceleration.

Also be aware that accelerations can take an unlimited amount of disk space so you may want to limit the amount of disk they can take up.

View solution in original post

shaskell_splunk
Splunk Employee
Splunk Employee

The answer is that it depends. When a user interacts with an un-accelerated data model it will be accelerated via ad hoc data model acceleration. A summary will be built in the dispatch directory of the search head and will take up some disk space but it will only persist for that session. Once a user navigates away from pivot, it will go away.

If you accelerate the data model, the acceleration will live on the indexers in buckets next to the raw data. How much space it uses will depend what is in the data model and the period of acceleration. I'd suggest building a data model, accelerating it and then inspecting the acceleration to see how much space it is using. There will also be CPU cycles spent to update the acceleration in the background.

Be aware of the caveats around data model acceleration. You can only accelerate a root event object hierarchy.

Here is a good break-down of the differences between ad hoc acceleration and persistent acceleration.

Also be aware that accelerations can take an unlimited amount of disk space so you may want to limit the amount of disk they can take up.

packet_hunter
Contributor

Thank you for your awesome response, that will get me going.

0 Karma
Get Updates on the Splunk Community!

Index This | A sphere has three, a circle has two, and a point has zero. What is it?

September 2023 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

Build Scalable Security While Moving to Cloud - Guide From Clayton Homes

 Clayton Homes faced the increased challenge of strengthening their security posture as they went through ...

Mission Control | Explore the latest release of Splunk Mission Control (2.3)

We’re happy to announce the release of Mission Control 2.3 which includes several new and exciting features ...