Getting Data In

How to optimize a heavy forwarder sending HEC data to index cluster?

robertlynch2020
Motivator

Hi

I am running a heavy forwarder with HEC and it is sending data to 3 indexers. 
I am starting to read about ways to optimise this configuration, but I am not sure if I have all the settings.

 

 

[tcpout]
defaultGroup = default-autolb-group
[tcpout-server://hp923srv:9997]
[tcpout-server://hp924srv:9997]
[tcpout:default-autolb-group]
disabled = false
server = hp923srv:9997,hp924srv:9997,hp925srv:9997
[tcpout-server://hp925srv:9997]

 

 

 

Or if someone has a few settings that they know work. All machines are on 56 threads with HT on - so I have lots of CPU free.

1st - How to I monitor the history of the data coming in from the HF->indexers
2nd - Can you share some settings for the heavy forwarder and the indexers please to get the data into Splunk the fastest


This is what I have read so far, but I am  not 100% sure about some of them any advice would be great

parallelIngestionPipelines = X (This is to be set on the HF and the Indexer, i think)

dedicatedIoThreads=Y (To be set on the HF)

Thanks in advance

Robert 

 

Labels (1)
Tags (2)
0 Karma
1 Solution

PickleRick
SplunkTrust
SplunkTrust

I think you're overthinking it a bit. That document is also a bit flawed. Not even regarding technicalities, because I didn't read it thoughrouly but design-wise. You don't architect an environment with a single undexer and expect it to handle several terabytes of data per day. Remember that indexing data is not everything that indexers do. Sure, given enough cpu's and iops you can increase number of ingestion pipelines (although the docs say that the increase in performance drops over the value of 2 - I haven't tested it personally) but you'll end up with a huge server which might even manage to index huge amount of data but you won't be able to search it because you have only one cpu per search available and no search peers to distribute the load across.

So I believe you're looking in the wrong place at the wrong moment - the famous quote popularized by Donald Knuth says that premature optimization is the root of all evil. And I believe this is your case. You're focusing (and using your time worrying about) single element's efficiency when it's more reasonable to verify whether your overall architecture is solid. Especially that you don't have any problems with your setup.

Sure, the parameters are tunable and are there for a reason but unless you have a real need for that - leave them be. Optimizers fot the sake of optimizing itself end up running Gentoo and recompiling software whole weekend so that your machine boots two seconds faster.

OK, You can increase the number of ingestion pipelines if you have CPUs to spare - that's a low hanging fruit worth reaching for. But beyond that... I'd rather have more smaller forwarders than a single big one (it also removes the SPOF you have with single HF).

View solution in original post

0 Karma

PickleRick
SplunkTrust
SplunkTrust

The first and most important question is if you're having performance problems that you want to solve or just fiddle with things just for the sake of tuning itself. Also remember that as a rule of thumb Splunk scales horizontally better than by adding cores to a single machine.

0 Karma

robertlynch2020
Motivator

hi 

 

After reading this paper it is saying that out of the box this is not optimized, so I just want to optimize it.

I don't have any specif error message at the moment, but i am also not sure if it is working as fast as it can

https://www.outcoldsolutions.com/blog/2021-04-21-configuring-hec-for-performance/

0 Karma

PickleRick
SplunkTrust
SplunkTrust

I think you're overthinking it a bit. That document is also a bit flawed. Not even regarding technicalities, because I didn't read it thoughrouly but design-wise. You don't architect an environment with a single undexer and expect it to handle several terabytes of data per day. Remember that indexing data is not everything that indexers do. Sure, given enough cpu's and iops you can increase number of ingestion pipelines (although the docs say that the increase in performance drops over the value of 2 - I haven't tested it personally) but you'll end up with a huge server which might even manage to index huge amount of data but you won't be able to search it because you have only one cpu per search available and no search peers to distribute the load across.

So I believe you're looking in the wrong place at the wrong moment - the famous quote popularized by Donald Knuth says that premature optimization is the root of all evil. And I believe this is your case. You're focusing (and using your time worrying about) single element's efficiency when it's more reasonable to verify whether your overall architecture is solid. Especially that you don't have any problems with your setup.

Sure, the parameters are tunable and are there for a reason but unless you have a real need for that - leave them be. Optimizers fot the sake of optimizing itself end up running Gentoo and recompiling software whole weekend so that your machine boots two seconds faster.

OK, You can increase the number of ingestion pipelines if you have CPUs to spare - that's a low hanging fruit worth reaching for. But beyond that... I'd rather have more smaller forwarders than a single big one (it also removes the SPOF you have with single HF).

0 Karma

robertlynch2020
Motivator

Hi 

Thanks for the good replay.

I think you are right I will wait for things to go wrong before I try to add on optimizations.

We are about to move to 8.1.9, however, I think there is a bug with the "of ingestion pipelines" not working for a few versions. I think it might only work in 8.2.4 or something like that.

Thanks again

Rob

Tags (1)
0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...