Getting Data In

What is the maximum Splunk indexing capacity per second? We need indexing speed of 100 MB/s

Path Finder

As per my requirement, we are required to index data of 100 MB per second. With the default configuration I am able to only @ 2MB/s. What need to be done in order to achieve this?

0 Karma
1 Solution

SplunkTrust
SplunkTrust

In terms of indexing, single reference hardware (or better - multiply RAM by ten!) can easily do 20MB/s or more... you don't want that much data incoming per indexer though, search performance will suffer.

As a rule of thumb, you'll want 75 to 300gb per day per indexer depending on your work load. Assuming 9000GB per day that's 30 to 120 indexers. At this scale, helpful Splunk Answers posts aren't enough - you'll want to talk to Splunk or a local partner, get some sales engineer assistance for initial planning, and professional services to do the architecture and deployment with you.

You'll also want to read the Capacity Planning Manual in Splunk Docs: http://docs.splunk.com/Documentation/Splunk/6.4.2/Capacity/IntroductiontocapacityplanningforSplunkEn...
It has a handy table at http://docs.splunk.com/Documentation/Splunk/6.4.2/Capacity/Summaryofperformancerecommendations that ends at 2-3TB/day with 11 to 21 indexers... scaling that by x3 to x5 you roughly get the indexer counts I mentioned above.

View solution in original post

New Member

@basilarockiaedwin1

I have a problem same as you, but I want send data about 300TB/day.
How did you solve your problem?
thank you.

0 Karma

SplunkTrust
SplunkTrust

A lot of indexers.

Srsly, get professional help on site if you're going to deploy 1000 to 3000 indexers. Splunk and your local Splunk Partner will happily send you capable people, possibly even for free, when you buy such license capacity.

0 Karma

SplunkTrust
SplunkTrust

In terms of indexing, single reference hardware (or better - multiply RAM by ten!) can easily do 20MB/s or more... you don't want that much data incoming per indexer though, search performance will suffer.

As a rule of thumb, you'll want 75 to 300gb per day per indexer depending on your work load. Assuming 9000GB per day that's 30 to 120 indexers. At this scale, helpful Splunk Answers posts aren't enough - you'll want to talk to Splunk or a local partner, get some sales engineer assistance for initial planning, and professional services to do the architecture and deployment with you.

You'll also want to read the Capacity Planning Manual in Splunk Docs: http://docs.splunk.com/Documentation/Splunk/6.4.2/Capacity/IntroductiontocapacityplanningforSplunkEn...
It has a handy table at http://docs.splunk.com/Documentation/Splunk/6.4.2/Capacity/Summaryofperformancerecommendations that ends at 2-3TB/day with 11 to 21 indexers... scaling that by x3 to x5 you roughly get the indexer counts I mentioned above.

View solution in original post

SplunkTrust
SplunkTrust

100 MB/s is about 9 TB/day. There are people ingesting that much (and more), so it can be done.

At this level of ingestion your Splunk rep can assist in lining up some folks to help you plan this installation out. I very much recommend calling them. They can help you plan what sort of indexer cluster you'll need to handle that load.

0 Karma

Path Finder

Thanks for your response.
Will Splunk be able to support real time search using live monitoring for an incoming data @ 100MB /sec

0 Karma

SplunkTrust
SplunkTrust

Again, the folks at Splunk can answer this better than I.

In general the answer may depend on exactly what you mean by "real time" and what sort of hardware you throw at this problem. The license for 9 TB/day will have a not inconsiderable price tag associated with it, so I would hope the budget for hardware would be commensurate. You'll have to work out the details with Splunk.

I also heard through the rumor mill that "real time" system performance limitations may be improved significantly in the very near future (as in the next minor revision or two, for time travelers visiting this from the future, the current version being 6.4.1 with 6.4.2 having just been released). This may or may not affect your use case, but it's worth keeping in mind.

0 Karma

SplunkTrust
SplunkTrust