All Apps and Add-ons

Search head out of disk space because ${SPLUNK_HOME}/var/lib/splunk has HUGE db directories?

woodcock
Esteemed Legend

I have 2 search heads that are very similar but one has some extra apps installed (such as SoS). The one with more apps is continuously out of disk space and I just found out why. On the search head that is fine, /opt/splunk/var/lib/splunk has 531M used but on the loaded one, it has 35G!!! What is taking up all the space? Many directory pairs like this and .dat. Inside each directory are 3 directories: "colddb", "db", and "thaweddb". The "db" directories are where all the space is consumed. What is creating these and how can I rein it in?

Tags (2)
0 Karma
1 Solution

Ayn
Legend

These are Splunk's indexes. The directory names in var/lib/splunk usually correspond to the index names, so if you want to see what all that data is, just search the corresponding index name in Splunk on your search head.

View solution in original post

woodcock
Esteemed Legend

I do not understand what this is telling me:
splunk_server VALUE_audit VALUE_internal summary_forwarders summary_hosts summary_indexers summary_pools summary_sources summary_sourcetypes
splunk-search04 20752 / 500000MB (4%) 5947 / 500000MB (1%) 1316 / 500000MB (0%) 1326 / 500000MB (0%) 129 / 500000MB (0%) 19 / 500000MB (0%) 7551 / 500000MB (2%) 377 / 500000MB (0%)

0 Karma

woodcock
Esteemed Legend

I do not understand what this is telling me:
splunk_server VALUE_audit VALUE_internal summary_forwarders summary_hosts summary_indexers summary_pools summary_sources summary_sourcetypes
splunk-search04 20752 / 500000MB (4%) 5947 / 500000MB (1%) 1316 / 500000MB (0%) 1326 / 500000MB (0%) 129 / 500000MB (0%) 19 / 500000MB (0%) 7551 / 500000MB (2%) 377 / 500000MB (0%)

0 Karma

Ayn
Legend

These are Splunk's indexes. The directory names in var/lib/splunk usually correspond to the index names, so if you want to see what all that data is, just search the corresponding index name in Splunk on your search head.

View solution in original post

nawazns5038
Builder

what does the datamode_summary contain.
how can we move data from one path to another in an indexer cluster.

0 Karma

woodcock
Esteemed Legend

I do not understand what this is telling me:
splunk_server VALUE_audit VALUE_internal summary_forwarders summary_hosts summary_indexers summary_pools summary_sources summary_sourcetypes
splunk-search04 20752 / 500000MB (4%) 5947 / 500000MB (1%) 1316 / 500000MB (0%) 1326 / 500000MB (0%) 129 / 500000MB (0%) 19 / 500000MB (0%) 7551 / 500000MB (2%) 377 / 500000MB (0%)

0 Karma

sowings
Splunk Employee
Splunk Employee

And you might consider the Fire Brigade app, appropriate to your Splunk version. In particular the "Indexer Host Overview" page could help explain what's going on with that search head.

0 Karma

sowings
Splunk Employee
Splunk Employee

The above search would show summary index usage local to the search head. If you're using report acceleration, you might try | rest /services/admin/summarization splunk_server=local, and pay attention to summary.size. Some apps (like bluecoat or Palo Alto) may call "tscollect" directly to create tsidx name spaces. These are a bit harder to track down (as in, I don't yet have a search for identifying that space). There may also be summary space in use by accelerated data models, but that space would be on the indexers and not on the search head.

0 Karma

woodcock
Esteemed Legend

We are not running any summary indices but we do have some apps that may have setup some, which was my theory (and why I mentioned SoS app). Is there a way to map these files to the app that created them?

0 Karma

dwaddle
SplunkTrust
SplunkTrust

Given it's a search head, my bet is on summary indexes and/or tsidx files for apps like bluecoat or palo alto..

0 Karma

sowings
Splunk Employee
Splunk Employee

To add to Ayn's comment, you can run this search on your search head:


| rest /services/data/indexes splunk_server=local
| search totalEventCount!=0
| eval cell=tostring(currentDBSizeMB) + " / " + tostring(maxTotalDataSizeMB) + "MB (" + tostring(round(currentDBSizeMB * 100 / maxTotalDataSizeMB)) + "%)"
| chart first(cell) over splunk_server by title

.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!