Splunk Search

How to search the disk-space consumption of certain logs for the last 60 days and how can I remove them?

New Member

Hi All,

We are running out of drive space. How can I check space consumption of certain logs for last 60 days and how can I remove them?

Tags (3)
0 Karma

Re: How to search the disk-space consumption of certain logs for the last 60 days and how can I remove them?


First, this may or may not apply but if the source data still exists on the Splunk box they can be trimmed or removed possibly. For instance, if you run a syslog server on this machine and it's collecting logs to files, then Splunk is reading those files, you may have forgotten to logrotate those originating files and they could be filling up the drive. As long as the data is in Splunk, the older files should be safe to delete. This is NOT something you do to actual indexes in your s$splunkhome/var/lib/db/splunk folder or wherever you've put them, only something you might be able to do to, say, /var/log/ directories.

For Splunk's indexes, you need to know if you are storing data in separate indexes or if they're all in one. You can likely find this information out from Settings/Indexes. Your indexes will not start with an underscore and the information should be there on oldest event, number of events and the size of the index.

One internal index you can check is _internal, it may have a fairly lengthy retention that you don't need. This is internal logging and if it is, say, 30 GB and holding 6 months, you could perhaps trim it to 5 or 10 GB. You should not go any lower than you need to here, but 30 days or so is usually adequate.

If you have your data split out into multiple separate indexes you can certainly change the sizes there a bit to make some indexes smaller. It'll require a Splunk restart to make any changes take effect, but they don't take long to actually happen once you restart. For each index you can run a search like index=myindexname to see just what's in it and how the data is arranged. That's sometimes useful to double-check and confirm it's got what you think is in it.

If you have all your data in the index "main" then there's little to do that won't affect all the data. The settings control this are on an index by index basis, so they can't pick and choose events. Now, it is possible to "delete" individual events, but that's in quotes because they only disappear from search results and do NOT reduce space consumption so won't work for your needs.

So, if you DO have all your data in index main, I'd start separating new data as it comes in into different indexes based on retention requirements and permissions. If logs X, Y and Z are going to nearly always be related and treated as one and you need to keep a lot of them, they can go into one index with a longer retention set on them. If logs A and B are more ephemeral and won't need different permissions, you could put them into another index with a max size set low enough to only keep a month of them. At least in the future you'll have a better time of this.

Also, then, for now, if you have no way to make your disks bigger and nothing much else to get rid of, I can't think of anything else to do except shrink your index main slightly. Sorry.

View solution in original post

0 Karma