How to send warm data to cold on a separate local partition?

[rodgerkrau]

Local E drive is full and I need to send the "warm" data to "cold" on another partition (D)
I set up coldPath = D:\SplunkColdData in the local indexes.conf file, but it's not working.

[rodgerkrau]

Splunk service wont start after your recommendations. this is what I have

  [_audit]
  maxTotalDataSizeMB = 5000

  [volume:warmVol]
  path=E:\Splunk\warmData
  maxVolumeDataSizeMB = 400000

  [volume:coldvol]
  path=D:\Splunk\coldData
  maxVolumeDataSizeMB = 600000

  [main]
  frozenTimePeriodInSecs = 15552000
  maxTotalDataSizeMB = 5000
  maxDataSize = auto
  coldPath = volume:coldVol/defaultdb
  homePath = volume:warmVol/defaultdb

  [pan_logs]
  coldPath = volume:coldVol/pan_logs
  homePath = volume:warmVol/pan_logs
  homePath.maxDataSizeMB = 50000

  [symantec]
  coldPath = volume:coldVol/symantec
  homePath = volume:warmVol/symantec
  homePath.maxDataSizeMB = 50000

  [winevents]
  coldPath = volume:coldVol/winevents
  homePath = volume:warmVol/winevents
  homePath.maxDataSizeMB = 50000

and I changed the directories to:

D:\Splunk\coldData
E:\Splunk\warmData

[lguinn2]

Okay, now we have to look at the splunk logs to see what we missed. Take a look at about the last 40 lines of splunkd.log

You will find it wherever you placed the Splunk logs. By default that is probably `C:\Program Files\Splunk\var\log\splunk'

This will hopefully tell you what is still wrong...

[rodgerkrau]

08-07-2015 09:39:42.618 -0400 INFO loader - Automatic migration of modular inputs
08-07-2015 09:39:42.798 -0400 ERROR loader - win-service: Error running pre-flight-checks (_pclose returned 10).
08-07-2015 09:39:42.798 -0400 ERROR loader - win-service: Here is the output from running pre-flight-checks:
08-07-2015 09:39:42.798 -0400 ERROR loader - Checking critical directories... Done
08-07-2015 09:39:42.798 -0400 ERROR loader - Checking indexes...
08-07-2015 09:39:42.798 -0400 ERROR loader - Problem parsing indexes.conf: idx=main Configured path 'volume:coldVol/defaultdb' refers to non-existent volume 'coldVol'; 3 volumes in config
08-07-2015 09:39:42.798 -0400 ERROR loader - Validating databases (splunkd validatedb) failed with code '1'. If you cannot resolve the issue(s) above after consulting documentation, please file a case online at http://www.splunk.com/page/submit_issue
08-07-2015 09:39:42.798 -0400 ERROR loader - <<<<< EOF (pre-flight-checks)

[rodgerkrau]

maybe case sensitive? thats the only difference I see

[rodgerkrau]

ok, after changing the case sensitivity.. I am able to start the service and I notice under D:\Splunk\coldData, the indexname folders appeared. So going-forward, the "warm data" will be written to E:\Splunk\coldData and then removed and copied to D:\Splunk\coldData? if so, how often?

[lguinn2]

Splunk is absolutely case-sensitive - except when a user types in the search bar: then it just case-sensitive sometimes.

When editing any .conf file, always respect case and you will do better!

The warm data will be written to the appropriate folder in E:. When the partition is full or the maximum number of warm buckets is reached, the oldest bucket in the warm folder will be moved to 😧

If you are over the size limits now, Splunk will start moving buckets right away until your indexes are with the limits. After that, the Splunk daemon will check regularly. I don't know how often exactly, but it will happen at least every time a new bucket is filled, and probably much more often than that.

[rodgerkrau]

doesnt seem to be working.. here is my indexes.conf

(the site seemed to remove my backslashes for all my paths)

[default]

[_audit]
maxTotalDataSizeMB = 5000

[volume:warmVol]
path=E:\Splunk\_internaldb\warmData
path=E:\Splunk\historydb\warmData
path=E:\Splunk\pan_logs\warmData
path=E:\Splunk\summarydb\warmData
path=E:\Splunk\symantec\warmData
path=E:\Splunk\winevents\warmData
maxVolumeDataSizeMB = 400000

[volume:coldvol]
path=D:\Splunk\_internaldb\coldData
path=D:\Splunk\historydb\coldData
path=D:\Splunk\pan_logs\coldData
path=D:\Splunk\summarydb\coldData
path=D:\Splunk\symantec\coldData
path=D:\Splunk\winevents\coldData
maxVolumeDataSizeMB = 600000

[main]
frozenTimePeriodInSecs = 15552000
maxTotalDataSizeMB = 5000
maxDataSize = auto
coldPath = volume:coldVol/defaultdb
homePath = volume:warmVol/defaultdb

[index1]
coldPath = volume:coldVol/defaultdb
homePath = volume:warmVol/defaultdb
homePath.maxDataSizeMB = 50000

I created D:\Splunk\indexes\coldData (and have separate folders for all indexes)
I created E:\Splunk\indexes\warmData (and have separate folders for all indexes)
I moved the associated db folder from E:\Splunk\var\lib\splunk\indexname\db to E:\Splunk\indexes\warmData\indexname\db

restart splunk..however, the splunkd service will not start
"The Splunkd service started on then stopped. Some services stop automatically if they are not in use by other services or programs."

[lguinn2]

Dang, my answer clearly wasn't very clear... Taking your indexes.conf, you should have something like

[_audit]
 maxTotalDataSizeMB = 5000

 [volume:warmVol]
 path=E:\Splunk\warmData
 maxVolumeDataSizeMB = 400000

 [volume:coldvol]
 path=D:\Splunk\coldData
 maxVolumeDataSizeMB = 600000

 [main]
 frozenTimePeriodInSecs = 15552000
 maxTotalDataSizeMB = 5000
 maxDataSize = auto
 coldPath = volume:coldVol/defaultdb
 homePath = volume:warmVol/defaultdb

 [pan_logs]
 coldPath = volume:coldVol/pan_logs
 homePath = volume:warmVol/pan_logs
 homePath.maxDataSizeMB = 50000

 [symantec]
 coldPath = volume:coldVol/symantec
 homePath = volume:warmVol/symantec
 homePath.maxDataSizeMB = 50000

With stanzas similar to pan_logs and symantec for the rest of the indexes.

Does that make more sense? I tried to tune up the answer to be a little cleaner...

[lguinn2]

I also changed the original answer and my reply - Originally I had used these paths for the volumes

D:\Splunk\indexes\ColdData
E:\Splunk\indexes\warmData

I realized that these names were confusing so I have simplified them to

D:\Splunk\ColdData
E:\Splunk\warmData

[lguinn2]

[Edited for clarity and a nasty typo - sorry]

Answer variation: Assume that you have multiple indexes, and that there are currently no cold buckets in any of them. You want to split the indexes between two disks: D will contain all the cold buckets and E will have the hot/warm buckets. You are willing to allocate 400 GB to indexes on E, while D has 600 GB available for indexes.

The easiest way to manage this is probably by using Splunk volumes. Here is the indexes.conf - I am showing only the relevant settings for two indexes: index1 and main, plus the settings that define the Splunk volumes. You will need to edit indexes.conf for ALL your indexes.

[volume:warmVol]
# the warmVol sets a maximum size for the hot/warm buckets of all indexes combined 
# each index will have its own subdir beneath this path - which is specified in the individual 
# index stanzas 
path=E:\Splunk\warmData
maxVolumeDataSizeMB = 400000

[volume:coldVol]
# similar for the cold buckets of all indexes
path=D:\Splunk\ColdData
maxVolumeDataSizeMB = 600000

[main]
# specifies the subdirectory in the coldVol path for this particular index
# main is weird because its path name doesn't match its index name
# watch out for this with some of the internal indexes as well
coldPath = volume:coldVol/defaultdb
homePath =  volume:warmVol/defaultdb

[index1]
# specifies the subdirectory in the coldVol path for this particular index
coldPath = volume:coldVol/index1
homePath =  volume:warmVol/index1
# the following sets a 50 GB max size for the hot/warm buckets of index1
homePath.maxDataSizeMB = 50000

Note that you will need to create the directories D:\Splunk\ColdData AND E:\Splunk\warmData. You will also need to move the data from the db directories for all indexes into the corresponding locations under E:\Splunk\warmData\indexname\db. Note that there must be a unique subdirectory for each index under the warmData directory. Then restart Splunk.

I know this seems like more trouble, but in the long run it should make it easier to manage your disk space.

[lguinn2]

You can't just changes indexes.conf to move buckets this way. There are steps, depending on what you want to do. First, it is clear that you are out of space on E, but is there anything in the cold directory in E? My answer assumes yes, but just skip step 3 (copy with veryify) if the colddb directory is empty.

1. Stop Splunk
2. Create the directory D:\Splunk\indexname\ColdData
3. Copy (with verify) everything from E:\Program Files\splunk\var\lib\splunk\indexname\colddb to D:\Splunk\indexname\ColdData
4. Backup, then remove E:\Program Files\splunk\var\lib\splunk\indexname\colddb

5. Edit indexes.conf

   coldPath = D:\Splunk\indexname\ColdData
   homePath.maxDataSizeMB = 99999
   The homePath.maxDataSizeMB specifies the maximum amount of disk that you want to use on the E drive for this index. You need to set it!

Finally, start Splunk

IMPORTANT NOTE: Always create a separate directory for each index! That is why you see indexname in all the paths in my example. If you don't do this, you will eventually have a huge tangled mess of buckets!

[rodgerkrau]

the colddb folder is empty for all indexes