topic Re: How to send the logs from Universal forwaders to Heavy forwaders ? in Getting Data In

How to send the logs from Universal forwaders to Heavy forwaders ?

nagesh — Wed, 06 Dec 2023 09:36:10 GMT

I am trying to send the data from client machine (UF) installed and Heavy forwarder installed on other machine. But i am getting the below error.

12-06-2023 10:01:22.626 +0100 INFO ClientSessionsManager [3779231 TcpChannelThread] - Adding client: ip=10.112.73.20 uts=windows-x64 id=86E862DA-2CDC-4B21-9E37-45DFF4C5EFBE name=86E862DA-2CDC-4B21-9E37-45DFF4C5EFBE

12-06-2023 10:01:22.626 +0100 INFO ClientSessionsManager [3779231 TcpChannelThread] - ip=10.112.73.20 name=86E862DA-2CDC-4B21-9E37-45DFF4C5EFBE New record for sc=100_IngestAction_AutoGenerated app=splunk_ingest_actions: action=Phonehome result=Ok checksum=0

12-06-2023 10:01:24.551 +0100 INFO AutoLoadBalancedConnectionStrategy [3778953 TcpOutEloop] - Removing quarantine from idx=3.234.1.140:9997 connid=0

12-06-2023 10:01:24.551 +0100 INFO AutoLoadBalancedConnectionStrategy [3778953 TcpOutEloop] - Removing quarantine from idx=54.85.90.105:9997 connid=0

12-06-2023 10:01:24.784 +0100 ERROR TcpOutputFd [3778953 TcpOutEloop] - Read error. Connection reset by peer

12-06-2023 10:01:25.028 +0100 ERROR TcpOutputFd [3778953 TcpOutEloop] - Read error. Connection reset by peer

12-06-2023 10:01:28.082 +0100 WARN TcpOutputProc [3779070 indexerPipe_1] - The TCP output processor has paused the data flow. Forwarding to host_dest=inputs10.align.splunkcloud.com inside output group default-autolb-group from host_src=prdpl2splunk02.aligntech.com has been blocked for blocked_seconds=60. This can stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.

Re: How to send the logs from Universal forwaders to Heavy forwaders ?

PickleRick — Wed, 06 Dec 2023 10:03:45 GMT

OK.

1. What is your setup? You seem to be trying to send the data to Cloud, right?

2. This is a log from where? UF or HF? Because it's trying to send to cloud directly. So if it's the UF's log, your output is not properly configured. If it's a HF's log, then you don't have your network port open on the firewall.

3. What's the whole point of pushing the data from UF via HF? Remember than UF sends data cooked but HF sends the data parsed which means roughly 6x the bandwidth (and you don't get to parse the data on the indexers so some parts of your configuration might not work the way you expect).

Re: How to send the logs from Universal forwaders to Heavy forwaders ?

gcusello — Wed, 06 Dec 2023 10:04:14 GMT

Hi @nagesh ,

it seems that there's a block in connections between UF and HF.

At first:

did you enabled receiving on HF?
did you enabled forwardring to the HF on the UF?

Then, check the connection using telnet on the port you're using (default 9997).

If it's all ok, yiou should have, in your Splunk (not on the HF), the Splunk internal logs from that UF:

index=_internal host=<your_UF_hostname>

Ciao.

Giuseppe

Re: How to send the logs from Universal forwaders to Heavy forwaders ?

nagesh — Wed, 06 Dec 2023 10:18:59 GMT

yes , we have the connectivity.

splunk.exe cmd btool outputs list
UF node:
[tcpout-server://UFnode:9997]
[tcpout:default-autolb-group]
server =UFnode:9997
HF:

Not getting the logs in splunk while using the index="_internal" host=""

Re: How to send the logs from Universal forwaders to Heavy forwaders ?

nagesh — Wed, 06 Dec 2023 10:35:52 GMT

Yes, I am trying to send the data to splunk cloud.

The log file i am trying to receive from UF.

[root@HFNode bin]# telnet inputs2.align.<<stack>>.com 9997

Trying 54.159.30.2...

Connected to inputs2.<<stack>>.splunkcloud.com.

Escape character is '^]'.

^C^C^CConnection closed by foreign host.
Connected successfully.

Re: How to send the logs from Universal forwaders to Heavy forwaders ?

PickleRick — Wed, 06 Dec 2023 11:05:00 GMT

OK. So you have your UF pointed at the Cloud inputs, not at your HF. You should set your output to your HF.

Re: How to send the logs from Universal forwaders to Heavy forwaders ?

nagesh — Wed, 06 Dec 2023 12:22:34 GMT

Yes, I have created output.conf file and added the required info.

It is placed under etc/system/local/ folder.

tcpout]

defaultGroup = default-autolb-group

indexAndForward = 0

negotiateProtocolLevel = 0

sslCommonNameToCheck = *.<<stack>>.splunkcloud.com

sslVerifyServerCert = true

useClientSSLCompression = true

[tcpout-server://inputs1.<<stack>>.splunkcloud.com:9997]

[tcpout-server://inputs2.<<stack>>.splunkcloud.com:9997]

[tcpout-server://inputs14.align.splunkcloud.com:9997]

[tcpout:default-autolb-group]

disabled = false

server = 54.85.90.105:9997, inputs2.<<stack>>.splunkcloud.com:9997, inputs3.<<stack>>.splunkcloud.com:9997, .....
inputs15.<<stack>>.splunkcloud.com:9997

[tcpout-server://inputs15.<<stack>>.splunkcloud.com:9997]

sslCommonNameToCheck = *.<<stack>>.splunkcloud.com

sslVerifyServerCert = false

sslVerifyServerName = false

useClientSSLCompression = true

autoLBFrequency = 120

[tcpout:scs]

disabled=1

server = stack.forwarders.scs.splunk.com:9997

compressed = true

Re: How to send the logs from Universal forwaders to Heavy forwaders ?

nagesh — Wed, 06 Dec 2023 14:56:38 GMT

Yes, I have already created output.conf file and added the required info.

It is placed under the etc/system/local/ folder.

[tcpout] defaultGroup = default-autolb-group indexAndForward = 0 negotiateProtocolLevel = 0 sslCommonNameToCheck = *.<<stack>>.splunkcloud.com sslVerifyServerCert = true useClientSSLCompression = true [tcpout-server://inputs1.<<stack>>.splunkcloud.com:9997] [tcpout-server://inputs2.<<stack>>.splunkcloud.com:9997] [tcpout-server://inputs14.align.splunkcloud.com:9997] [tcpout:default-autolb-group] disabled = false server = 54.85.90.105:9997, inputs2.<<stack>>.splunkcloud.com:9997, inputs3.<<stack>>.splunkcloud.com:9997, ..... inputs15.<<stack>>.splunkcloud.com:9997 [tcpout-server://inputs15.<<stack>>.splunkcloud.com:9997] sslCommonNameToCheck = *.<<stack>>.splunkcloud.com sslVerifyServerCert = false sslVerifyServerName = false useClientSSLCompression = true autoLBFrequency = 120 [tcpout:scs] disabled=1 server = stack.forwarders.scs.splunk.com:9997 compressed = true

Re: How to send the logs from Universal forwaders to Heavy forwaders ?

nkamma — Thu, 07 Dec 2023 05:39:25 GMT

Can you provide me ant suggestions to resolve this issue?

Re: How to send the logs from Universal forwaders to Heavy forwaders ?

PickleRick — Thu, 07 Dec 2023 07:36:49 GMT

OK. Because you're posting those config snippets a bit chaotically.

Please do a

splunk btool inputs list splunktcp

and

splunk btool outputs list splunktcp

On both of your components.

And while posting the results here please use either code block (the </> sign on top of the editing window here on the Answers forum) or the "preformatted" paragraph style. Makes it way easier to read.

Re: How to send the logs from Universal forwaders to Heavy forwaders ?

nkamma — Thu, 07 Dec 2023 12:12:09 GMT

From UF installed:- [splunktcp] _rcvbuf = 1572864 acceptFrom = * connection_host = ip evt_dc_name = evt_dns_name = evt_resolve_ad_obj = 0 host = prdpl2bcl1101 index = default logRetireOldS2S = true logRetireOldS2SMaxCache = 10000 logRetireOldS2SRepeatFrequency = 1d route = has_key:tautology:parsingQueue;absent_key:tautology:parsingQueue Splunkcloud inputs machine: [root@servername bin]# ./splunk btool inputs list splunktcp [splunktcp] _rcvbuf = 1572864 acceptFrom = * connection_host = ip host = servername.aligntech.com index = default logRetireOldS2S = true logRetireOldS2SMaxCache = 10000 logRetireOldS2SRepeatFrequency = 1d route = has_key:_replicationBucketUUID:replicationQueue;has_key:_dstrx:typingQueue;has_key:_linebreaker:rulesetQueue;absent_key:_linebreaker:parsingQueue [splunktcp://9997] _rcvbuf = 1572864 connection_host = ip host = servername.aligntech.com index = default

Re: How to send the logs from Universal forwaders to Heavy forwaders ?

PickleRick — Thu, 07 Dec 2023 18:03:25 GMT

OK. So these are your inputs.

And your outputs?

Re: How to send the logs from Universal forwaders to Heavy forwaders ?

nkamma — Fri, 08 Dec 2023 08:46:45 GMT

Please below. [root@prdpl2splunk02 bin]# ./splunk btool outputs list [rfs] batchSizeThresholdKB = 131072 batchTimeout = 30 compression = zstd compressionLevel = 3 dropEventsOnUploadError = false format = json format.json.index_time_fields = true format.ndjson.index_time_fields = true partitionBy = legacy [syslog] maxEventSize = 1024 priority = <13> type = udp [tcpout] ackTimeoutOnShutdown = 30 autoLBFrequency = 30 autoLBVolume = 0 blockOnCloning = true blockWarnThreshold = 100 cipherSuite = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:AES256-GCM-SHA384:AES128-GCM-SHA256:AES128-SHA256:ECDH-ECDSA-AES256-GCM-SHA384:ECDH-ECDSA-AES128-GCM-SHA256:ECDH-ECDSA-AES256-SHA384:ECDH-ECDSA-AES128-SHA256 compressed = false connectionTTL = 0 connectionTimeout = 20 defaultGroup = default-autolb-group disabled = false dropClonedEventsOnQueueFull = 5 dropEventsOnQueueFull = -1 ecdhCurves = prime256v1, secp384r1, secp521r1 enableOldS2SProtocol = false forceTimebasedAutoLB = false forwardedindex.0.whitelist = .* forwardedindex.1.blacklist = _.* forwardedindex.2.whitelist = (_audit|_introspection|_telemetry) forwardedindex.filter.disable = false heartbeatFrequency = 30 indexAndForward = 0 maxConnectionsPerIndexer = 2 maxFailuresPerInterval = 2 maxQueueSize = 500KB negotiateProtocolLevel = 0 readTimeout = 300 secsInFailureInterval = 1 sendCookedData = true sslCommonNameToCheck = *.align.splunkcloud.com sslQuietShutdown = false sslVerifyServerCert = true sslVersions = tls1.2 tcpSendBufSz = 0 useACK = false useClientSSLCompression = true writeTimeout = 300 [tcpout-server://inputs1.stack.splunkcloud.com:9997] [tcpout-server://inputs15.stack.splunkcloud.com:9997] autoLBFrequency = 120 sslCommonNameToCheck = *.stack.splunkcloud.com sslVerifyServerCert = false sslVerifyServerName = false useClientSSLCompression = true [tcpout-server://inputs2.stack.splunkcloud.com:9997] [tcpout-server://inputs3.stack.splunkcloud.com:9997] [tcpout-server://inputs4.stack.splunkcloud.com:9997] [tcpout-server://inputs5.stack.splunkcloud.com:9997] [tcpout-server://inputs6.stack.splunkcloud.com:9997] [tcpout-server://inputs7.stack.splunkcloud.com:9997] [tcpout-server://inputs8.stack.splunkcloud.com:9997] [tcpout-server://inputs9.stack.splunkcloud.com:9997] [tcpout:default-autolb-group] disabled = false server = 54.85.90.105:9997, inputs2.stack.splunkcloud.com:9997, inputs3.stack.splunkcloud.com:9997,...... inputs15.stack.splunkcloud.com:9997 [tcpout:scs] compressed = true disabled = 1 server = stack.forwarders.scs.splunk.com:9997 UF Output: [rfs] batchSizeThresholdKB = 131072 batchTimeout = 30 compression = zstd compressionLevel = 3 dropEventsOnUploadError = false format = json format.json.index_time_fields = true format.ndjson.index_time_fields = true partitionBy = legacy [syslog] maxEventSize = 1024 priority = <13> type = udp [tcpout] ackTimeoutOnShutdown = 30 autoLBFrequency = 30 autoLBVolume = 0 blockOnCloning = true blockWarnThreshold = 100 cipherSuite = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:AES256-GCM-SHA384:AES128-GCM-SHA256:AES128-SHA256:ECDH-ECDSA-AES256-GCM-SHA384:ECDH-ECDSA-AES128-GCM-SHA256:ECDH-ECDSA-AES256-SHA384:ECDH-ECDSA-AES128-SHA256 compressed = false connectionTTL = 0 connectionTimeout = 20 defaultGroup = default-autolb-group disabled = false dropClonedEventsOnQueueFull = 5 dropEventsOnQueueFull = -1 ecdhCurves = prime256v1, secp384r1, secp521r1 enableOldS2SProtocol = false forceTimebasedAutoLB = false forwardedindex.0.whitelist = .* forwardedindex.1.blacklist = _.* forwardedindex.2.whitelist = (_audit|_introspection|_internal|_telemetry|_configtracker) forwardedindex.filter.disable = false heartbeatFrequency = 30 indexAndForward = false maxConnectionsPerIndexer = 2 maxFailuresPerInterval = 2 maxQueueSize = auto readTimeout = 300 secsInFailureInterval = 1 sendCookedData = true sslQuietShutdown = false sslVersions = tls1.2 tcpSendBufSz = 0 useACK = false useClientSSLCompression = true writeTimeout = 300 [tcpout-server://prdpl2splunk02.domainame.com:9997] [tcpout:default-autolb-group] server = prdpl2splunk02.domainame.com:9997

Re: How to send the logs from Universal forwaders to Heavy forwaders ?

nagesh — Mon, 11 Dec 2023 03:28:59 GMT

Any suggestions?

Re: How to send the logs from Universal forwaders to Heavy forwaders ?

soniya-01 — Mon, 11 Dec 2023 06:16:11 GMT

Sending logs from Universal Forwarders to Heavy Forwarders is like passing along important messages from one person to another in a relay. Here's a simple way to understand it:

Imagine Passing Notes:
- Think of Universal Forwarders as individuals who have notes (logs) with important information. Heavy Forwarders are the ones ready to collect and manage these notes.
Universal Forwarders (Note Holders):
- Universal Forwarders are like people holding notes (logs) and standing in a line. They generate logs from different sources on a computer.
Heavy Forwarders (Note Collectors):
- Heavy Forwarders are the ones waiting at the end of the line to collect these notes (logs) from the Universal Forwarders.
Setting Up the Relay:
- You set up a system where each person (Universal Forwarder) in the line passes their note (log) to the next person (Heavy Forwarder) until it reaches the end.
Configuring Universal Forwarders:
- On each computer with a Universal Forwarder, you configure it to know where the next person (Heavy Forwarder) is in line. This is like telling each note holder where to pass their note.
Logs Move Down the Line:
- As logs are generated, they move down the line from Universal Forwarder to Universal Forwarder until they reach the Heavy Forwarder.
Heavy Forwarder Collects and Manages:
- The Heavy Forwarder collects all the notes (logs) from different Universal Forwarders. It's like the person at the end of the line collecting all the notes to manage and make sense of them.
Centralized Log Management:
- Now, all the important information is centralized on the Heavy Forwarder, making it easier to analyze and keep track of everything in one place.

In technical terms, configuring Universal Forwarders to send logs to Heavy Forwarders involves setting up these systems to efficiently collect and manage logs from different sources across a network. It's like orchestrating a relay of information to ensure that important data reaches its destination for centralized management and analysis.

Re: How to send the logs from Universal forwaders to Heavy forwaders ?

PickleRick — Mon, 11 Dec 2023 20:32:49 GMT

At first glance it looks relatively OK. You have your inputs matching your outputs.

Check your splunkd.log on the sending UF and the receiving HF. There should be hints as to the reason for lack of connectivity. If nothing else helps - try to tcpdump the traffic and see what's going on there.

EDIT: OK, your initial post says that you get "Connection reset by peer" but it's a bit unclear which side this is from.