Deployment Architecture

expand heavy forwarder

raindrop18
Communicator

My logs volume increased and notice time out on my heavy forwarder, which is best way to add capacity increase the size of the server (more powerful server) or add multiple server. on the other word escaling horizontally vs vertically.

Tags (1)
0 Karma
1 Solution

Javip
Path Finder

ummm...

Have a look to your HF queues using DMC, perhaps HF is unable to handle all this incomming data from your 3k UFs.
Have a look to limits.conf file also.

Let me know if you find the problem there finally.

J.

View solution in original post

0 Karma

Javip
Path Finder

ummm...

Have a look to your HF queues using DMC, perhaps HF is unable to handle all this incomming data from your 3k UFs.
Have a look to limits.conf file also.

Let me know if you find the problem there finally.

J.

0 Karma

raindrop18
Communicator

Ok. I will and let you know but the bottom line is HF can't expand horizontally as Indexer?

0 Karma

Javip
Path Finder

yes, if necessary, you can have 2 HFs and you can send them balanced data from your UFs

0 Karma

Javip
Path Finder

Hi,

do you have CPU, memory, filesystem free space stats for this HF?
Do you use it only to pass data from your UFs to your IXs or you do more tasks there (i.e. Db connect, ...)?

If you provide all that info (or even more details) we can suggest you better solutions 😉

Regards,
J.

0 Karma

raindrop18
Communicator

thanks J. the current server not indicate any cpu or memory issue, the metrics for that is low only network input out put is high as high traffic coming to the HF, I am using the HF as HEC also. most of the time out is on the logs pushed over HEC.

0 Karma

p_gurav
Champion

Are you using heavy forward for filtering data or as center server to pass data?

0 Karma

raindrop18
Communicator

mainly to pass data not filtering.

0 Karma

p_gurav
Champion

Also is there any error in _internal logs for heavy forwarder? Is there any firewall in between?

0 Karma

raindrop18
Communicator

No firewall, I don't see any error. but on UF I see traffic is on waiting state. BTW I have close to 3k UF forward the logs to The HF. and HEC.

0 Karma
Get Updates on the Splunk Community!

This Week's Community Digest - Splunk Community Happenings [9.26.22]

Get the latest news and updates from the Splunk Community here! Upcoming User Group Events! 👏 Check ...

BSides Splunk 2022 - The Call for Papers is now Open!

TLDR; Main Site: https://bsidessplunk.com CFP Site: https://bsidessplunk.com/cfp CFP Opens: December 15th, ...

Sending Metrics to Splunk Enterprise With the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...