How to optimize a heavy forwarder sending HEC data to index cluster?

[robertlynch2020]

Hi

I am running a heavy forwarder with HEC and it is sending data to 3 indexers. 
I am starting to read about ways to optimise this configuration, but I am not sure if I have all the settings.

[tcpout]
defaultGroup = default-autolb-group
[tcpout-server://hp923srv:9997]
[tcpout-server://hp924srv:9997]
[tcpout:default-autolb-group]
disabled = false
server = hp923srv:9997,hp924srv:9997,hp925srv:9997
[tcpout-server://hp925srv:9997]

Or if someone has a few settings that they know work. All machines are on 56 threads with HT on - so I have lots of CPU free.

1st - How to I monitor the history of the data coming in from the HF->indexers
2nd - Can you share some settings for the heavy forwarder and the indexers please to get the data into Splunk the fastest


This is what I have read so far, but I am  not 100% sure about some of them any advice would be great

parallelIngestionPipelines = X (This is to be set on the HF and the Indexer, i think)

dedicatedIoThreads=Y (To be set on the HF)

Thanks in advance

Robert 

[PickleRick]

The first and most important question is if you're having performance problems that you want to solve or just fiddle with things just for the sake of tuning itself. Also remember that as a rule of thumb Splunk scales horizontally better than by adding cores to a single machine.

[robertlynch2020]

hi 

After reading this paper it is saying that out of the box this is not optimized, so I just want to optimize it.

I don't have any specif error message at the moment, but i am also not sure if it is working as fast as it can

https://www.outcoldsolutions.com/blog/2021-04-21-configuring-hec-for-performance/

[PickleRick · Accepted solution]

I think you're overthinking it a bit. That document is also a bit flawed. Not even regarding technicalities, because I didn't read it thoughrouly but design-wise. You don't architect an environment with a single undexer and expect it to handle several terabytes of data per day. Remember that indexing data is not everything that indexers do. Sure, given enough cpu's and iops you can increase number of ingestion pipelines (although the docs say that the increase in performance drops over the value of 2 - I haven't tested it personally) but you'll end up with a huge server which might even manage to index huge amount of data but you won't be able to search it because you have only one cpu per search available and no search peers to distribute the load across.

So I believe you're looking in the wrong place at the wrong moment - the famous quote popularized by Donald Knuth says that premature optimization is the root of all evil. And I believe this is your case. You're focusing (and using your time worrying about) single element's efficiency when it's more reasonable to verify whether your overall architecture is solid. Especially that you don't have any problems with your setup.

Sure, the parameters are tunable and are there for a reason but unless you have a real need for that - leave them be. Optimizers fot the sake of optimizing itself end up running Gentoo and recompiling software whole weekend so that your machine boots two seconds faster.

OK, You can increase the number of ingestion pipelines if you have CPUs to spare - that's a low hanging fruit worth reaching for. But beyond that... I'd rather have more smaller forwarders than a single big one (it also removes the SPOF you have with single HF).

[robertlynch2020]

Hi 

Thanks for the good replay.

I think you are right I will wait for things to go wrong before I try to add on optimizations.

We are about to move to 8.1.9, however, I think there is a bug with the "of ingestion pipelines" not working for a few versions. I think it might only work in 8.2.4 or something like that.

Thanks again

Rob