topic Re: Search head out of disk space because ${SPLUNK_HOME}/var/lib/splunk has HUGE db directories? in All Apps and Add-ons

Search head out of disk space because ${SPLUNK_HOME}/var/lib/splunk has HUGE db directories?

woodcock — Sat, 11 Jan 2014 05:43:45 GMT

I have 2 search heads that are very similar but one has some extra apps installed (such as SoS). The one with more apps is continuously out of disk space and I just found out why. On the search head that is fine, /opt/splunk/var/lib/splunk has 531M used but on the loaded one, it has 35G!!! What is taking up all the space? Many directory pairs like this and .dat. Inside each directory are 3 directories: "colddb", "db", and "thaweddb". The "db" directories are where all the space is consumed. What is creating these and how can I rein it in?

Re: Search head out of disk space because ${SPLUNK_HOME}/var/lib/splunk has HUGE db directories?

Ayn — Sat, 11 Jan 2014 14:42:24 GMT

These are Splunk's indexes. The directory names in var/lib/splunk usually correspond to the index names, so if you want to see what all that data is, just search the corresponding index name in Splunk on your search head.

Re: Search head out of disk space because ${SPLUNK_HOME}/var/lib/splunk has HUGE db directories?

sowings — Mon, 28 Sep 2020 15:38:08 GMT

To add to Ayn's comment, you can run this search on your search head:

| rest /services/data/indexes splunk_server=local | search totalEventCount!=0 | eval cell=tostring(currentDBSizeMB) + " / " + tostring(maxTotalDataSizeMB) + "MB (" + tostring(round(currentDBSizeMB * 100 / maxTotalDataSizeMB)) + "%)" | chart first(cell) over splunk_server by title

Re: Search head out of disk space because ${SPLUNK_HOME}/var/lib/splunk has HUGE db directories?

dwaddle — Sat, 11 Jan 2014 17:56:28 GMT

Given it's a search head, my bet is on summary indexes and/or tsidx files for apps like bluecoat or palo alto..

Re: Search head out of disk space because ${SPLUNK_HOME}/var/lib/splunk has HUGE db directories?

woodcock — Sun, 12 Jan 2014 19:57:31 GMT

We are not running any summary indices but we do have some apps that may have setup some, which was my theory (and why I mentioned SoS app). Is there a way to map these files to the app that created them?

Re: Search head out of disk space because ${SPLUNK_HOME}/var/lib/splunk has HUGE db directories?

sowings — Mon, 13 Jan 2014 13:27:29 GMT

The above search would show summary index usage local to the search head. If you're using report acceleration, you might try | rest /services/admin/summarization splunk_server=local, and pay attention to summary.size. Some apps (like bluecoat or Palo Alto) may call "tscollect" directly to create tsidx name spaces. These are a bit harder to track down (as in, I don't yet have a search for identifying that space). There may also be summary space in use by accelerated data models, but that space would be on the indexers and not on the search head.

Re: Search head out of disk space because ${SPLUNK_HOME}/var/lib/splunk has HUGE db directories?

sowings — Mon, 13 Jan 2014 13:28:27 GMT

And you might consider the Fire Brigade app, appropriate to your Splunk version. In particular the "Indexer Host Overview" page could help explain what's going on with that search head.

Re: Search head out of disk space because ${SPLUNK_HOME}/var/lib/splunk has HUGE db directories?

woodcock — Mon, 28 Sep 2020 15:38:33 GMT

I do not understand what this is telling me:
splunk_server VALUE_audit VALUE_internal summary_forwarders summary_hosts summary_indexers summary_pools summary_sources summary_sourcetypes
splunk-search04 20752 / 500000MB (4%) 5947 / 500000MB (1%) 1316 / 500000MB (0%) 1326 / 500000MB (0%) 129 / 500000MB (0%) 19 / 500000MB (0%) 7551 / 500000MB (2%) 377 / 500000MB (0%)

Re: Search head out of disk space because ${SPLUNK_HOME}/var/lib/splunk has HUGE db directories?

woodcock — Mon, 28 Sep 2020 15:38:36 GMT

Re: Search head out of disk space because ${SPLUNK_HOME}/var/lib/splunk has HUGE db directories?

woodcock — Mon, 28 Sep 2020 15:38:38 GMT

Re: Search head out of disk space because ${SPLUNK_HOME}/var/lib/splunk has HUGE db directories?

nawazns5038 — Wed, 06 Dec 2017 21:45:06 GMT

what does the datamode_summary contain.
how can we move data from one path to another in an indexer cluster.