primary volume capacity and "df -h" do not agree

[tlmayes]

Primary volume set to 650GB. MC reports that primary volume is 615/650, so all is good... But the volume consumption as reported by "df -h" reports 710GB being used, and nothing is stored in the path but the primary volume (indexes).

What am I missing? What is using the extra 60GB that is not being recognized by Splunk?

[gjanders]

The volume you have mentioned may contain 615GB of indexes, however there are files outside the volume that could be using disk s pace in the same directory, in particular:

1. Indexes which have been removed from the config but remain on the filesystem
2. Data model acceleration summaries
3. Other directories within that subsection of disk which are not indexes or data model summaries

I would normally use a different disk location for indexer data to make this easier.
For the data model summaries run:

splunk btool indexes list --debug

Looks for the tstatsHomePath, I suspect it will be the default volume of _splunk_summaries which has a location on Linux of /opt/splunk/var/lib/splunk/... (or $SPLUNK_DB/...)
Then under that directory per-index you will find the datamodel_summary directories

Since the _splunk_summaries is it's own volume (default volume for data models) it can also use that same section of disk

Also FYI data model summaries are stored on the indexing tier, data models are configured at the search head tier

[tlmayes]

Agree regarding use of a different partition, but is an inherited design so have to deal with what I have (for now). Primary is @ /opt/splunk/var/lib/splunk and secondary separate.

Very good feedback, which investigation shows this is the right track. Checked btool as suggested. Shows "../system/default/indexes.conf tstatsHomePath = volume:_splunk_summaries/$_index_name/datamodel_summary" .

Since there is no setting for volume _splunk_summaries assuming at this point that it is defaulting to the primary volume. Since summary is defaulting to the primary volume is it (_splunk_summary) size controlled by the "maxVolumeDataSizeMB" setting for primary or does it require its own setting?

[gjanders]

According to the indexes.conf file:
tstatsHomePath =
* Required.
* Location where datamodel acceleration TSIDX data for this index should be
stored
* MUST be defined in terms of a volume definition (see volume section below)
* Must restart splunkd after changing this parameter; index reload will not
suffice.
* CAUTION: Path must be writable.
* Defaults to volume:_splunk_summaries/$_index_name/datamodel_summary,
where $_index_name is runtime-expanded to the name of the index

On my system I see:
[volume:_splunk_summaries]
path = $SPLUNK_DB

In my quick test it defaults to 500000MB, and you can change it by explicitly setting the maxTotalDataSizeMB if you wish, however you might want to read Configure size based retention for data models and possibly the conf presentation replication of summary replication data in a cluster, it describes what happens if a data model summary gets trimmed due to size

[deepashri_123]

Hey@tlmayes,

Where are you running df -h exactly? Also does your splunk setup have datamodel accelerated? Can you check if the 60GB is the volume used by datamodel?

[tlmayes]

[volume:primary]
path = /opt/splunk/var/lib/splunk
maxVolumeDataSizeMB = 630001 (is 64,512 bytes)

/opt/splunk/var/lib]# du -cks * | sort -rn|head
661052116 total
661052116 splunk

df -h

Filesystem Size Used Avail Use% Mounted on
............... .... ... ... ... ........
/dev/mapper/vg01-opt_splunk 700G 673G 28G 97% /opt/splunk

To your question about accelerated datamodels so checked indexes.conf on the SH's and sure enough it is set to the same path, but a different value (740G). Since it is on the SH vice indexer, does it really have an effect? Built by others so am having to reverse engineer.

[deepashri_123]

You can try doing du -ch after going to the path opt/splunk/var/lib/splunk . This will give you exact split of what index is taking how much storage and you can further check in which datamodel is taking what storage. May be this would help!!