- Find Answers
- :
- Splunk Administration
- :
- Admin Other
- :
- Installation
- :
- SummaryDB Rolling over to Frozen
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark Topic
- Subscribe to Topic
- Mute Topic
- Printer Friendly Page
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Our splunk summarydb index is enormous - it has maxed out at the default 500GB and has started moving content into our predefined frozen directory archive. There are over 3 billion events listed.
What is a good size for the summarydb index? Do you ever need to keep the frozen archive? I was afraid of losing the data, but now I'm not so sure we even need it. To my knowledge, we don't even make use of summary index commands (si*). Is it safe to reduce the consumption of 500GB to 200GB or even lower? I don't see the advantage of keeping such a large summary index.
The maximum size of our other indexes (non-summary) are 50GB (they have yet to rollover) - so I'm not sure why the summarydb has such a large footprint when it's supposed to only contain summary information.
This has really only become an issue since we are doing a splunk upgrade to 5.0 and need to backup our index databases per the installation guide steps.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
The idea of a summary index is it contains a subset of your live data to allow for faster searching. This summary data should only be kept as long as it is useful and should not get larger than the original.
As yours is growing out of control, it seems you have some badly set up summarising searches. First look for searches with "index=summary" in them to see what summarised data you are using. Then look for searches with si commands or the "collect" command in them and see if the data they are collecting is useful to you. If not, disable them. You may also find searches that could be merged. Make sure to check any apps you have installed as well.
Deleting summary data can cause inaccurate reports so care should be taken but in most cases, the wanted results can be rebuilt with the backfill script. See http://docs.splunk.com/Documentation/Splunk/5.0/Knowledge/Managesummaryindexgapsandoverlaps for details.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
The idea of a summary index is it contains a subset of your live data to allow for faster searching. This summary data should only be kept as long as it is useful and should not get larger than the original.
As yours is growing out of control, it seems you have some badly set up summarising searches. First look for searches with "index=summary" in them to see what summarised data you are using. Then look for searches with si commands or the "collect" command in them and see if the data they are collecting is useful to you. If not, disable them. You may also find searches that could be merged. Make sure to check any apps you have installed as well.
Deleting summary data can cause inaccurate reports so care should be taken but in most cases, the wanted results can be rebuilt with the backfill script. See http://docs.splunk.com/Documentation/Splunk/5.0/Knowledge/Managesummaryindexgapsandoverlaps for details.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
You rock! This helped us find the source problem. When we queried index=summary, we found the offending application that was filling up our summaryDB. For now we've disabled the savedSearch. It seems there was an issue with the scheduled search not having an earliest time defined so all events were getting put into the summaryDB with each scheduled run. Using the Manager->Searches and reports view, we set App Context to "All" so that we could see where our scheduled searches were and verified that they had a proper start time in their Time Range field. Kudos! +1
Enterprise Security Content Update (ESCU) | New Releases
Why am I not seeing the finding in Splunk Enterprise Security Analyst Queue?
Index This | What are the 12 Days of Splunk-mas?
