How to backup splunk-var data?

[power12]

Hello Splunkers ,

I wrote a python script that explores the splunk-var indexes and calculates their total size, and then asks the user if they’d like to back it up.

After the user indicates which indexes they’d like to back up, it copies all buckets and other metadata in the db path (excluding the hot bucket) to a dir that is specified as a command line arg.

I want to know

• How to actually back up files (is it as simple as copying out the dir and then later copying it in and restarting splunk)
• Best implement bucket policies (maxHotSpanSecs)
• Understand bucket rollover when we have unexpected behavior
  
  What indexes.conf should  I use to have the bucket have one day worth of data

Thanks in Advance

[gcusello]

Hi @power12 ,

answering to your questions:

How to actually back up files (is it as simple as copying out the dir and then later copying it in and restarting splunk)

• back-up of warm and cold buckets can be done also with active Splunk, doesn't need to stop Splunk.

Best implement bucket policies (maxHotSpanSecs)

• it depends on your situaztion, but I usually leave the default values.

Understand bucket rollover when we have unexpected behavior

• why do you speak of unexpected behaviour? if there's something strange, you can see it in the Monitoring console or by messages.

What indexes.conf should  I use to have the bucket have one day worth of data

• why should you have one day worth of data? you don't have any advantage of this and probably some problems, infact there's a Splunk alert that fires when you have too small buckets because this limits performaces; I'd avoid to have too small and too large buckets, for this reason I leave the default values.

Ciao.

Giuseppe

[power12]

@gcusello Thank you for your reply..We have a single instance splunk and 100GB license and on an average we get 10GB of data per day..Even with this..its not good practice to have 1 day worth data?

[gcusello]

Hi @power12,

as also @richgalloway said, it isn't a good idea having one day worth data because, unless you have very large data volumes (as not in your case), in this way you'll have a largen number of very small buckets.

leave the default values!

Ciao.

Giuseppe

[richgalloway]

Splunk has a document that explains how to backup and recover your indexes.   It also explains rollover.  See https://docs.splunk.com/Documentation/Splunk/9.0.3/Indexer/Backupindexeddata

We don't have enough information to answer the second question.  What's best for you may not be best for others.  Also, different indexes within a site may need different settings.

For a bucket to contain at most one day of data, set maxHotSpanSecs to 86400.  Note that a bucket may contain less than a day of data if it filled up early.

[power12]

@richgalloway  Thank you for your reply .  Yes I used the same setting but its chunking before 86400 ..so I checked with btool and saw that the maxDataSize is set to default which is 750 MB..changing that solved the issue

We have a single instance splunk and 100GB license and on an average we get 10GB of data per day..Even with this..is it not good practice to have 1 day worth data ?

[richgalloway]

I'm glad to hear you solved your problem.

I think it is not correct to say that 1 day of data is not good practice.  It can be good practice, depending on your needs.  Many sites use that practice to help ensure their data freezes in a timely fashion.  If a bucket contains multiple days of data then old data in that bucket will remain searchable until the newest event in the bucket expires.  That could violate the site's data retention policy.