topic How to change the total disk space use in Splunk (hot, cold data)? in Monitoring Splunk

How to change the total disk space use in Splunk (hot, cold data)?

phamxuantung — Mon, 15 Aug 2022 13:41:32 GMT

Hello,

Our Splunk system just got an increase in size as image below (we have a Master, 1:1 indexes cluster struture)

Meaning we have an increase for hot from 500GB -> 1T and cold from 1.5T -> 3T

I have change the stanza in splunk/etc/master-apps/_cluster/local/indexes.conf (where we put our individual indexes config like maxTotalDataSizeMB, homePath.MaxDataSizeMB, coldPath.MaxDataSizeMB) to match the newly provide disk space. But after I restart services for both our indexers and master, it won;t apply the newly assign disk space but still using old one. I suspect I miss something here.

Can anyone point me to where can I config overall setting? (Because I'm not familial with splunk structure)

Re: How to change the total disk space use in splunk (hot, cold data)

richgalloway — Mon, 15 Aug 2022 12:12:49 GMT

If you're using volumes (which is a good idea) then you'll need to adjust the maxVolumeDataSizeMB setting.

Re: How to change the total disk space use in splunk (hot, cold data)

PickleRick — Mon, 15 Aug 2022 13:13:54 GMT

I don't understand, to be honest. If you only expanded your storage, and raised your limits it doesn't mean that splunk will automatically fill all the storage space.

Firstly, as @richgalloway pointed out - there are volume settings. But even if you're not using volume-based restrictions, storage utilization depends on:

1) ingestion rate (if you're ingesting - for example - 10MB per day, you won't fill a 1TB drive in 10 days)

2) Size-based retention limits

3) Time-based retention limits

You might have your size limits increased but still buckets would get deleted if they got too old.

Re: How to change the total disk space use in Splunk (hot, cold data)?

matt8679 — Mon, 15 Aug 2022 15:06:38 GMT

You need to deploy the cluster bundle to the peers. A restart will not apply the new settings

On the master:

splunk apply cluster-bundle

Re: How to change the total disk space use in splunk (hot, cold data)

phamxuantung — Tue, 16 Aug 2022 08:02:05 GMT

1) We ingest 600GB data on a daily basis, and the old storage is too small for our need.

2) Sized base retention limit. There is an index that I want to extend, the old limit was

maxTotalDataSizeMB = 480000 homePath.maxDataSizeMB = 180000 coldPath.maxDataSizeMB = 300000

We use this config since the start, and now it's not enough anymore so I set it as

maxTotalDataSizeMB = 1300000 homePath.maxDataSizeMB = 500000 coldPath.maxDataSizeMB = 800000

I restart master, index01, index02 but it doesn't apply the new config.

3) We don't have time-based retention limit

And we don't have maxVolumnDataSizeMB in the first place, so why, if I don't add it, the new config won't apply when I change the limit for each index.

So I was wonder if there was a step else where that I missed in this?

Re: How to change the total disk space use in Splunk (hot, cold data)?

phamxuantung — Tue, 16 Aug 2022 08:02:57 GMT

I tried running your command on master, but it show

Re: How to change the total disk space use in Splunk (hot, cold data)?

matt8679 — Tue, 16 Aug 2022 10:06:03 GMT

Try this in the CLI, path to where splunk is installed on the master indexers:

./splunk apply cluster-bundle