Deployment Architecture

Multi indexer / mixed hardware question

jamiejames45
Engager

I am in the process of migrating(in production!) from a Two machine implementation ( one search head, and one search head /indexer) to a properly distributed setup. I placed a Heavy Forwarder in-line yesterday. It seems to be humming along smoothly. the load on my indexer has gone down quite a bit as it is no longer having to parse the incoming data.

Heres what I am trying to do ,

log-001(SH/IN) -> log-001(IN)

log-002(SH) -> log-002(SH/IN)

log-003(HF) -> log-003(HF)

Specs:

log-001

8 Cores @ 3.0GHz

8G RAM

800 G storage ( broken up into 250 hot and 550 warm/cold)

log-002

24 Cores @ 2.8Ghz

32G RAM

150G storage (RAID 10)

Indexing approx 100G a day, soon to be 300 😕

My question is, whats the best way to re-allocate the resources I have to get the best performance out of Splunk.
I am guessing I will need more storage on log-002, as well as a procedure to "split" the indexes that currently live on 001.

If I don't add more storage to 002, what will be the impact as that drive isnt able to index an equal share of the data?
Am I coorect in thinking that moving the search functionality totally off of log-001 will leave me with a more-suited-to-the-task indexer?

aaaaaand, I thank ya's

J

TLDR;
Splunk grew faster than the HW, attempting to migrate from overloaded box to distrubted enviro. tips? gotchas?

0 Karma

MarioM
Motivator

Actually you should dedicate your search head thus 1SH + 2 Indexers and if you going to grow to 300GB 4 indexers should be your aim.

But it all depends on what kind of disks you have,how many searches,how many concurrent users,what used cases,...

You should get in contact with Splunk Professional Services to get proper sizing as they have experience and knowledge.