I've consulted at F100's for 15 years (9 of which were at Cisco Systems) so I am intimately familiar with these scenarios.
redundancy and HA:
Ok, so use 4x$2k computers - joking a bit there, but sure, you can do it just fine at a fraction of the cost.
Just because you have not seen this done doesn't mean it can't be.
The world wasn't always round, ya know?
... View more
At 500GB per day, that works out to a little under 25k events per second.
So why does it take 7 (quite expensive) servers with all of these resources to do what could be done on a $2k PC?
I'll give you a hint: You're doing it wrong 😉
Here's what I used (an older pc):
It has 32GB ram and Ubuntu 16 with a Samsung EVO NVME being used as a cache drive using lvmcache.
I'm able to ingest at around 40k eps and searches take a few seconds.
If you do insist on spending a whole bunch of money for something you don't need, be sure your disks are set up correctly both in raid (10 is best, but 6 is fine). Be sure you have the correct strip size in both the raid bios/config and when you partition the disks in Linux. As well as the mount options you use in /etc/fstab - and be VERY sure you partition on the right sector boundaries (4k by default) - not doing so will drastically reduce performance.
Be sure your OS disks are not the same as your data disks. You will thank me later 😉
I spent a few hours and went from around 300MB/s Read, 76MB/s Write to 1.3GB/s read and 800MB/s write.
Happy to share my notes if you want to PM me.
... View more