Is there a smart way of cleaning event data from an index on a regular basis in a multisite indexer clustering environment?

[rakesh_498115]

Hi All,

I have 4 Indexer Servers, 2 per each Site. Site1 has 2 and Site2 has 2 servers in each. All these 4 servers are in clustering and I have a Master Node to monitor these 4 Indexer peers. Site based clustering is implemented so in any case if any site is down, I can still be able to get the data from the other site.

Now my problem here, I have an index called "my_index", and I need to clean this index on a regular basis without impacting the business.

So I have written a script to each one indexer by other and clean the eventdata of "my_index" and restarted them. By doing so, once the indexer is restarted again, it's getting "my_index" data back, since the replication factor is given as 4 and its getting the data again.

I cannot afford to stop all the 4 indexers at once and clean the eventdata as this will impact the customer till the indexers are up and running. Can you please suggest if there is any SMART WAY of cleaning the indexes on a regular basis without impacting customers?

Thanks,
Rakesh.

[rmorlen_splunk]

This is what I would do:

On Cluster Master:
splunk enable maintenance-mode

Then follow the steps below on each indexer to delete data from an index...

First come up with a search that finds only the data that you want removed. Run it to test that it works.

Then make sure the admin user (or the user that that are using to do the delete) has can_delete role (or add the role).

splunk edit user admin -role admin -role can_delete

Run the search to delete the data (I want to delete data from the main index for the sourcetype syslog for a specific timeframe):

splunk search 'index=main sourcetype=syslog | delete' -earliest_time '-26d@d' -latest_time '-3d@d'

Then do the next indexer.

Once you have run this on all indexers then from the Cluster Master:

splunk rolling-restart cluster-peers
splunk show cluster-status
splunk disable maintenance-mode

[jmallorquin]

Hi,

Have you thought to set a minimum retention of this index to 1mb? and do a cluster-bundle.

Regards

[rakesh_498115]

Hi Jmallorquin, you mean to write a script to set index retention to 1mb and deploy this across all indexes and do a restart and once restarted get back the retention period ??

[jmallorquin]

Hi,

You didn't create this index with a master-app??? Where did you configure this index?

You don't need to create any script, splunk can do a bundle and control the restart all of the peers, just modifique the configure in the master-app and make a bundle.

[rakesh_498115]

hmm..index master is actually having the configuration of indexes replication factor (server.conf) and all the 4 indexers (peers) are having the indexes's configuration ( indexes.conf) .

// server.conf -- at master server
[general]
site=site2

[clustering]
mode=master
multisite=true
available_sites=site1,site2
site_replication_factor = origin:2,total:4
site_search_factor = origin:2,total:4
restart_timeout = 900

// indexes.conf - at peers

[my_index]
coldPath = volume:cold/sphur_lag/colddb
homePath = volume:hotwarm/sphur_lag/db
thawedPath = $SPLUNK_DB/cold/sphur_lag/thaweddb
frozenTimePeriodInSecs = 15638400
repFactor = auto

here my Index master is acting as the Deployment Server as well. Can you please suggest where to change teh config now.??

[jmallorquin]

Hi rakesh_498115

Where exactly did you configure the index in the peers? system/local ?

You should distributed the indexes.conf like a master-app

To distribute configurations across the peer nodes:

1. If distributing any indexes.conf files, configure them so that they support index replication. See "Configure the peer indexes in an indexer cluster".

2. Place the files in the $SPLUNK_HOME/etc/master-apps directory on the master. The set of subdirectories in this location constitute the configuration bundle.

3. Use Splunk Web or the CLI to distribute the configuration bundle to the peer nodes.

For details on these steps, see "Update common peer configurations and apps".

http://docs.splunk.com/Documentation/Splunk/6.1/Indexer/Managecommonconfigurations

[jmallorquin]

Hi,

Thinking... if you configure all the indexes.conf in the peers, the changes will not take effects after you restart, so it will be fine if you configure all the indexes.conf with a small retention and then from the master make a cluster-bundle.

The master will deploy the configuration, in your case nothing... but will control the restart of the indexers.

But you should think in use master-apps to deploy these files in the future.

hope i help you.