I've an enviroment like this:
1 Search Head Cluester ( 3 servers ) ;
1 Indexers Cluster ( 4 server );
1 Deployment Server;
1 Cluster Master;
1 Heavy Forwarder;
and N universal forwarder.
On each server, of the indexer cluster, I've opened the 9997 port to receive and ingest data.
I configured the Heavy forwarder to send data to the IDX cluster through an'APP with an outputs.conf like this:
[tcpout] defaultGroup = default-autolb-group [tcpout:default-autolb-group] indexerDiscovery = idxc1 useACK = true autoLBVolume = 65536 [indexer_discovery:idxc1] master_uri = https://[cm_ip]:8090 pass4SymmKey = **********
This output is not SSL and all the data which use this output are ingested from IDX cluster.
Now I've to send from the same Heavy Forwarder other data but in SSL. So after creating the certificates for Indexers and for the forwarders, after copying the RootCA on each server, I've created an app on the forwarder with an inputs.conf like this:
[tcp-ssl://6514] disabled = false sourcetype = syslog index = ads [SSL] serverCert = $SPLUNK_HOME/etc/certs/hforwarder.pem sslPassword = *********** requireClientCert = false
and outputs.conf like this :
[tcpout] defaultGroup=splunkindexer-ssl [tcpout:splunkindexer-ssl] autoLBFrequency = 30 compressed = false server = idx1:9996,idx2:9996,idx3:9996,idx4:9996 clientCert = $SPLUNK_HOME/etc/certs/splunk.pem useSSL=true sslPassword = ***********
On the each indexers node I've create an app to open the 9996 port on SSL, with an inputs.conf like this:
[tcp-ssl://9996] disabled = false [SSL] serverCert = $SPLUNK_HOME/etc/certs/splunk.pem sslPassword = ********** requireClientCert = false
After deploying this configuration the first flow through the 9997 doesn't receive any data and not all data are forwarded through 9996.
With tcpdump I see the right flow from the source to Heavy Forwarder (6514) , I see some data receveived from the indexer cluster on 9996 but no data where indexed on ads index and on other index.
So now I ask myself what should be wrong and if a forwarder can support two outputs.conf so configured.
Any help is appreciated and welcome
Why are you sending part of data over SSL and other data on non-SSL from same Heavy Forwarders to same set of Indexers ? If there is requirement to send some data over SSL then I'll send all data over SSL.
Probably the easiest way is to configure the 9997 port on indexer in SSL, and all the comunication will go over SSL
But If I want to configure the other port ?
Does the Heavy Forwarder support a configuration like the one I've posted in my question ?
in the last days I've succeded in establishing the right comunication between HF and IDX. So now when i bring up the port 9996 all the other forwarding still work fine.
Only the one which runs on 6514 (hf) ---> 9996 (idx_cl ) give me those messagges in metrics.log:
01-27-2020 14:41:26.012 +0100 INFO Metrics - group=tcpin_connections, a.b.c.d:40386:6514, connectionType=rawSSL, sourcePort=40386, sourceHost=a.b.c.d, sourceIp=a.b.c.d, destPort=6514, kb=0, _tcp_Bps=0, _tcp_KBps=0, _tcp_avg_thruput=0, _tcp_Kprocessed=0, _tcp_eps=0, _process_time_ms=0, evt_misc_kBps=0, evt_raw_kBps=0, evt_fields_kBps=0, evt_fn_kBps=0, evt_fv_kBps=0, evt_fn_str_kBps=0, evt_fn_meta_dyn_kBps=0, evt_fn_meta_predef_kBps=0, evt_fn_meta_str_kBps=0, evt_fv_num_kBps=0, evt_fv_str_kBps=0, evt_fv_predef_kBps=0, evt_fv_offlen_kBps=0, evt_fv_fp_kBps=0
I've tried even this way, but after the deployment, was alway the same.
In all the NO-SSL app's input.conf I put
_TCP_ROUTING = default-autolb-group
and in the SSL one :
but it is always the same.