Getting Data In

What are the best way and practices to manage indexes

chlima
Explorer

Hi everyone!

I would like to know what are the best practices to manage the index's size.

I read in this post ( https://www.splunk.com/blog/2011/01/03/managing-index-sizes-in-splunk.html ) that we must control the size using maxWarmDBCount and maxTotalDataSize , which are indexes.conf parameters.

But I know is possible to manage this using other two parameters homePath.maxDataSizeMB and coldPath.maxDataSizeMB, that apears to be easier than first configuration.

What is the best way to do it?

1 Solution

niketn
Legend

One of the best thing to refer would be @sloshburch .conf2017 session on Best Practices for Admins.

http://conf.splunk.com/sessions/2017-sessions.html#search=Best%20Practices%20and%20Better%20Practice...

Also go through couple of Splunk Wiki Topics
https://wiki.splunk.com/Community:More_best_practices_and_processes
https://wiki.splunk.com/Things_I_wish_I_knew_then

____________________________________________
| makeresults | eval message= "Happy Splunking!!!"

View solution in original post

niketn
Legend

One of the best thing to refer would be @sloshburch .conf2017 session on Best Practices for Admins.

http://conf.splunk.com/sessions/2017-sessions.html#search=Best%20Practices%20and%20Better%20Practice...

Also go through couple of Splunk Wiki Topics
https://wiki.splunk.com/Community:More_best_practices_and_processes
https://wiki.splunk.com/Things_I_wish_I_knew_then

____________________________________________
| makeresults | eval message= "Happy Splunking!!!"

sloshburch
Splunk Employee
Splunk Employee

Bingo! I'm a huge fan of defining volumes and the maxVolumeDataSizeMB for that volume. Then also set frozenTimePeriodInSecs per index. This means that (assuming I use the volumes in my index definition) I can limit the time period and size per index, but also prevent the overall filesystem from filling up the data volume per index changes drastically without my realizing it.

gcusello
SplunkTrust
SplunkTrust

Hi chlima,
I think that it mainly depends by your retention requirements: you have to define how long logs must be searchable (maybe there are compliance requirements!) and then you have to configure your storage based on a capacity planning.
If you want it's possible to use the Monitoring Console as input for the Capacity Planning.
Retention can be manager with frozenTimePeriodInSecs option in indexes.conf.

Bye.
Giuseppe

0 Karma

chlima
Explorer

Hi Giuseppe!

Thanks for your response.

I want to maintain my data available for two months and after this roll it to frozen.

My question is about the best way to configure this, since two presented configurations apears to work the same way in my opinion.

0 Karma

gcusello
SplunkTrust
SplunkTrust

I'd use frozenTimePeriodInSecs=5184000.
Bye.
Giuseppe

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...