Splunk Forward Internal Logs

[ws]

Hi,

I recently took over a Splunk environment and would like to clarify the following.

My objective is to forward all Splunk instance logs to the indexer through the Heavy Forwarder (HF). However, I noticed that the current setup uses indexerDiscovery, which sends data directly to the indexer.

When I reconfigured the Splunk instances to use server = HeavyForwarder_IP:9997 instead, I encountered a "tcpout has been blocked" error on the HF when attempting to connect to the indexer. The logs indicate that the connection to the indexer is established, but immediately followed by the message "tcpout has been blocked."

I also discovered that the indexer is configured with [splunktcp-ssl://9997], whereas the HF is using [splunktcp://9997] without any SSL settings.

Could this be the root cause? Do I need to configure the HF with sslCertPath, sslRootCAPath, sslPassword, and sslVerifyServerCert in order for it to connect properly?

[PrewinThomas]

@ws 

Looks like your highlighted root cause is correct. You need to configure the HF to send data over SSL.
Try to configure HF over SSL and check
eg:

[tcpout]
defaultGroup = indexer_group

[tcpout:indexer_group]
server = indexer_IP:9997
sslCertPath = $SPLUNK_HOME/etc/auth/mycerts/hf_cert.pem
sslRootCAPath = $SPLUNK_HOME/etc/auth/mycerts/ca.pem
sslPassword = your_password
sslVerifyServerCert = true

Regards,
Prewin
If this answer helped you, please consider marking it as the solution or giving a Karma. Thanks!

[ws]

Hi @PrewinThomas,

Thanks for clarifying. I understand that this issue could be due to SSL needing to be configured. I’ll be testing the following steps to see if any further errors occur.

Fingers crossed, hopefully, this resolves the issue. If not, I’ll revisit this post and follow up.

Thanks again!

[isoutamo]

You should configure also your HF to listen with TLS/SSL. Without that UF's cannot send events to HF which then could forward those into Indexers.
Have you some real reasons why you want to use HF between source systems and indexers? The best practices is sent events directly into indexers!

[ws]

Hi @isoutamo 

Currently all mine UF is able to send to the HF without the SSL.

As of now, the issue mainly would be between the HF and Indexer while forwarding Splunk instance logs with the flow of HF > Indexer.

I'm not really too sure what is the reason behind as when I took over, I was told that this is suppose to be the setup whereby all logs including Splunk instance should go thru the HF.

[isoutamo]

I just realized that you are trying to send e.g. SH logs to indexers via HF, if I understand correctly?
Why there is a this kind on order to add unnecessary complexity into your environment?

Normal way is send all Splunk infra instances (like SH, LM, MC, CM, DS, Deployer etc.) log directly into indexers. This is actually 1st time when I even heard this kind of additional step/requirement! With this setup you will be probably get more issues than what this try to solve (I couldn't even guess what the problem is what you are trying to solve).

But as said the issue is that indexers only listen to TLS enabled ports. This means that also those other Splunk Servers are trying to send TLS enabled streams. When you try to sent TLS enabled stream to HF (which just listen plain TCP), it didn't work.

My proposal is that forget (at least SH, MC, LM, CM) to use HF between those and Indexers. If you have e.g. DS, HFs etc. outside of your main servers then those could be configured to use HF as IHF. 

Then just ensure that those are using plain tcp instead of tcp with ssl in outputs. conf.