Splunk Enterprise Security

How many resources would I need to receive 150 GB per day?

christianubeda
Path Finder

Hello team,

I want to build a new SIEM using Splunk.

I hope to receive between 100 and 150 GB of data per day.

How many resources would you need to deploy the server in a virtual machine?

Could anyone give me an estimate? Or where I could access the information to know?

Thanks to everyone

0 Karma
1 Solution

DavidHourani
Super Champion

Hi christianubeda,

Depending on the data mix, the scaling volume can range from 40GB to 100GB per indexer when you have ES in your environment. So in your case for 150GB/day you will need two indexers. That means Indexer cluster. So here's the minimum VM requirement:

  • 1 Cluster Master
  • 2 Indexers
  • 1 Search Head for ES
  • 1 Search Head for other apps
  • 1 Deployment server

Hardware specs can be found here :
http://docs.splunk.com/Documentation/Splunk/7.2.1/Capacity/Referencehardware
For VM make sure your ressources are dedicated as shown here : https://www.splunk.com/pdfs/technical-briefs/splunk-deploying-vmware-tech-brief.pdf
As for people I'd say one admin is still enough. But you'll need someone as a backup I guess in case you're off 😉

Cheers,
David

View solution in original post

DavidHourani
Super Champion

Hi christianubeda,

Depending on the data mix, the scaling volume can range from 40GB to 100GB per indexer when you have ES in your environment. So in your case for 150GB/day you will need two indexers. That means Indexer cluster. So here's the minimum VM requirement:

  • 1 Cluster Master
  • 2 Indexers
  • 1 Search Head for ES
  • 1 Search Head for other apps
  • 1 Deployment server

Hardware specs can be found here :
http://docs.splunk.com/Documentation/Splunk/7.2.1/Capacity/Referencehardware
For VM make sure your ressources are dedicated as shown here : https://www.splunk.com/pdfs/technical-briefs/splunk-deploying-vmware-tech-brief.pdf
As for people I'd say one admin is still enough. But you'll need someone as a backup I guess in case you're off 😉

Cheers,
David

jmeyers_splunk
Splunk Employee
Splunk Employee

I realize this is a bit of a late comment, but I wanted to share a point of clarity around distributed search vs. index replication / indexer clustering.

Having more than one indexer means does not automatically mean indexer clustering. It does mean distributed search - which enables horizontal scaling of indexers. from: About Distributed Search

Horizontal scaling for enhanced performance. Distributed search facilitates horizontal scaling by providing a way to distribute the indexing and searching loads across multiple Splunk Enterprise instances, making it possible to index and search large quantities of data.

Indexer clustering is a complementary concept which then adds replication for indexed data. from About Indexer Clusters

Indexer clusters are groups of Splunk Enterprise indexers configured to replicate each others' data, so that the system keeps multiple copies of all data. This process is known as index replication. By maintaining multiple, identical copies of Splunk Enterprise data, clusters prevent data loss while promoting data availability for searching.

0 Karma

christianubeda
Path Finder

Hi David!

For now I will install a single indexer. My actual data volume that I will have imminently will be 10GB. DoYou think that with 16 cores and 32Gb of Ram will be enough?.

Thank you!

DavidHourani
Super Champion

yeah! that will do !

You can also consider setting up a cluster master and join that one indexer as a peer. That will help you later on since when you get to 150GB/day you will have to get another indexer and you don't want to have to go through the trouble of migrating from stand alone to clustered environment.

Richfez
SplunkTrust
SplunkTrust

I heartily agree with @DavidHourani and also recommend you create a Cluster Master and use the one indexer you have as a single-member cluster. It will really make transitioning to a two-member cluster easier later.

mdessus_splunk
Splunk Employee
Splunk Employee

Are you talking about human or hardware resources ?
For hardware, have a look here: http://docs.splunk.com/Documentation/ES/5.2.0/Install/DeploymentPlanning

By the way, I suggest you to run the indexers in physical servers instead of VM.

christianubeda
Path Finder

Hi again,

I just reall all.

I have taken into account the tips of the manual and will try to apply them.

I think I will finally build a structure with a search head and multiple Indexers as for the resources we will start with 16 cores and 32 GB.

If we need more we will add more.

What do you think? It will manage 10GB per day and 100 in the future?

Thanks!

0 Karma

christianubeda
Path Finder

Hi,

I was talking about hardware. But since you have brought up the subject, how many human resources do I need? We currently receive 2GB and I can manage it but we are going to change it soon, so it will be necessary to expand everything.

Thanks!

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...