Hi All,
So I'm in the process of fixing some huge delays in my data ingestion (some data takes up to 2 hours+) to be searchable . I've adjusted the maxkbps = 4096 in my forwarders limits.conf
I can see Splunkd it's now transferring 9,000,000 bytes per second (9MB's) and has drastically reduced the lag in data ingestion. Splunkd is now easily the highest network traffic generating process on the server. With that said in my internal logs shows it's still reaching the max 4096 limit and throttles the flow of logs to the indexer.
There's about 8 webservers that will generate this amount of traffic. There's 2 indexers, where the forwarders data is divided up by sourcetype via tcp routing
With a 10gb network (forwarders and indexers) shouldn't I be able to double this max to 8192 and not worry about the network choking? So theoretically if each server is maxing out that would be a constant flow of 18MB/s. 18 * 8 = 144MB/S
a 10gbps network can handle 1250MB/s do you think this is feasible to do? Would the bottle neck be my indexers ability to write the data fast enough?
Originally parsing queue was an issue however I've upped it.
Parsing queue = 30MB in my forwarders server.conf and doesn't seem to be a issue anymore.
Someone who's ingesting a LOT of data please chime in with your setup/ numbers.
