topic Re: Configuring Distributed Indexes in Getting Data In

Configuring Distributed Indexes

craigmunro — Thu, 24 Mar 2011 23:24:30 GMT

Hi,

I'm planning to use the deployment server to deploy configs to my indexers and use network storage for the cold DB. I have the following in indexes.conf:

[foo]
homePath = $SPLUNK_DB/foo/db
coldPath = /nfs/foo-splunk/storage/foo/colddb
thawedPath = $SPLUNK_DB/foo/thaweddb
maxDataSize = auto_high_volume
maxWarmDBCount = 7
maxTotalDataSizeMB = 220000
frozenTimePeriodInSecs = 1900800

When I have multiple indexes how is this applied? For example is the maxTotalDataSizeMB per indexer or per index?

Can multiple indexers use the same location for the colddb?

Thanks

Re: Configuring Distributed Indexes

Genti — Thu, 24 Mar 2011 23:32:38 GMT

The above stanza will be applied to the foo index only. no matter how many other indexes you have.
maxTotalDataSizeMB will be per index, ie only for the foo index. (if you are deploying this to more then one indexer, this will be so for all those indexers, but again, only for the foo index.)

You can NOT use the same location for multiple indexers (otherwise you would have clashes between the data). If you want to use the same nfs then you should probably create directories for all your indexers say:

/nfs/indexer1/foo/
/nfs/indexer2/foo/
etc..

Re: Configuring Distributed Indexes

craigmunro — Fri, 25 Mar 2011 17:57:23 GMT

Thanks Genti. I understand that this config only applies to the foo index I'm trying to clarify what happens when this index is distributed across multiple indexers. You say that maxTotalDataSizeMB would apply to all indexers. Does that mean it has the same limit on each indexer or that the limit is across all indexers?

Is the hostname available as a variable in indexes.conf so I can do /nfs/$HOSTNAME/foo/ and deploy the same config to all indexers?

Re: Configuring Distributed Indexes

Genti — Sat, 26 Mar 2011 06:26:31 GMT

you should NOT have all your indexers read and write files to that exact same folder on the NFS. If you set them up in different directories, then maxTotalDataSizeMB will apply to a single indexer/index. Hence if you say 10 Gig, and then each indexer will have 10Gig max for that index. if you have 10 indexers, each of 10 Gig for the particular index, then you need 100Gigs...

Re: Configuring Distributed Indexes

craigmunro — Fri, 01 Apr 2011 20:19:25 GMT

Setting the coldPath as /nfs/$HOSTNAME/foo in indexes.conf works which means I can use the deployment server to configure my indexers.

Re: Configuring Distributed Indexes

lguinn2 — Thu, 03 Nov 2011 16:52:30 GMT

To clarify, let's say I have two indexers. On each of these indexers, I set the size of foo in indexes.conf as

maxTotalDataSizeMB=100000

Each indexer must write to its own directory (typically on local disk for performance reasons). The max size of the index is the SUM of all of the space allocated by all indexers = 200000 MB