Knowledge Management

How does creating a data model affect storage and memory?

packet_hunter
Contributor

I am asking this question as I dig thru the documentation.

Currently I don't have a lot of reserve disk storage or indexing/license reserve in my deployment.

If I start creating and testing different data models, how do I predict how much disk space and indexing the data model will consume?

I will admit I have not taken any of the admin courses yet.

Any advice on this is appreciated.

Thank you

0 Karma
1 Solution

shaskell_splunk
Splunk Employee
Splunk Employee

The answer is that it depends. When a user interacts with an un-accelerated data model it will be accelerated via ad hoc data model acceleration. A summary will be built in the dispatch directory of the search head and will take up some disk space but it will only persist for that session. Once a user navigates away from pivot, it will go away.

If you accelerate the data model, the acceleration will live on the indexers in buckets next to the raw data. How much space it uses will depend what is in the data model and the period of acceleration. I'd suggest building a data model, accelerating it and then inspecting the acceleration to see how much space it is using. There will also be CPU cycles spent to update the acceleration in the background.

Be aware of the caveats around data model acceleration. You can only accelerate a root event object hierarchy.

Here is a good break-down of the differences between ad hoc acceleration and persistent acceleration.

Also be aware that accelerations can take an unlimited amount of disk space so you may want to limit the amount of disk they can take up.

View solution in original post

shaskell_splunk
Splunk Employee
Splunk Employee

The answer is that it depends. When a user interacts with an un-accelerated data model it will be accelerated via ad hoc data model acceleration. A summary will be built in the dispatch directory of the search head and will take up some disk space but it will only persist for that session. Once a user navigates away from pivot, it will go away.

If you accelerate the data model, the acceleration will live on the indexers in buckets next to the raw data. How much space it uses will depend what is in the data model and the period of acceleration. I'd suggest building a data model, accelerating it and then inspecting the acceleration to see how much space it is using. There will also be CPU cycles spent to update the acceleration in the background.

Be aware of the caveats around data model acceleration. You can only accelerate a root event object hierarchy.

Here is a good break-down of the differences between ad hoc acceleration and persistent acceleration.

Also be aware that accelerations can take an unlimited amount of disk space so you may want to limit the amount of disk they can take up.

View solution in original post

packet_hunter
Contributor

Thank you for your awesome response, that will get me going.

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!