topic Re: Deployment Requirements for following data usage in Installation

Deployment Requirements for following data usage

harshal_chakran — Tue, 01 Apr 2014 12:47:30 GMT

Hi Team,

I have been evaluating splunk since last two months, I have Core i3 Processor & 4 GB ram as PC configuration. I have developed a solution using splunk and python sdk and at present it takes approximately - 8 Min to give me the result

Input is 12 files(.CSV files) of total 2MB (This is for testing purpose only, generally each file will be of approx 300mb)

In the above life cycle of app, I follow the following steps
1. I have a paramter file and some 100s of xml file which is parsed in python and a Reference file is generated
2. I get the raw data which is csv file in a folder, from which my python script upload in splunk
3. I run few queries which are automated ones and dynamic based on the parameter file and xml files of step 1, and using outputcsv command i get corresponding intermediate files (for 12 input files -> 12 Intermediate files)
4. Then this intermediate files are again uploaded on splunk
5. 12*3 = 36 Queries are fired on each of the files and total to 2 new files are generated
6. Now this newly generated files are used as input for all the graphs n tables and charts on the dashboard.

The above whole procedure takes approx 8 Min, i Wanted to know what will be the requirement of the server if following is my data size.

If we have the requirement to process 5 GB data per day and maintain historical data of about 3 months, What would be the Server Configuration needed:
1. RAM Size
2. CPU Power
3. Hard Disk Space
4. Any other specific requirement is terms of software & hardware
5. Which splunk license do i have to buy? I mean which will be best for this scenario and what will be the approximate cost.

Thanks!

Re: Deployment Requirements for following data usage

sunrise — Tue, 01 Apr 2014 13:48:47 GMT

You should refer to recommened hardware and capacity planing in manual.
System requirements - http://docs.splunk.com/Documentation/Splunk/latest/Installation/Systemrequirements#Recommended_and_minimum_hardware_capacity

Hardware capacity planning for your Splunk Enterprise deployment - http://docs.splunk.com/Documentation/Splunk/latest/Installation/CapacityplanningforalargerSplunkdeployment

And in addition you should pay attention following.
1. IOPS is the king in indexing
2. Search needs CPU Cores and memory.
3. Data should be compressed about 50 percent in Splunk

How Splunk licensing works -
http://docs.splunk.com/Documentation/Splunk/latest/Admin/HowSplunklicensingworks

Re: Deployment Requirements for following data usage

harshal_chakran — Wed, 02 Apr 2014 05:08:12 GMT

I understood the theoretical part and i had already read this but how do i estimate the hardware requirement.

You do not have any calculator in which i input my data size and i get recommended hardware output?

If I have the requirement to process 5 GB data per day and maintain historical data of about 3 months,

What would be the Server Configuration needed:

Is this enough for my requirement

2x six-core, 2+ GHz CPU, 12 GB RAM, RAID 0 or 1+0, with a 64 bit OS installed

& what if this 5GB changes to 5TB per day, what hardware changes/upgrades will be required??