topic Re: HF not receiving logs from UF in Getting Data In

HF not receiving logs from UF

anil8 — Sun, 23 Jan 2022 12:43:02 GMT

Hi Splunkers,

We have configured 3 new heavy forwarder in our splunk enterprise where 2 HF was already working.

Now we want traffic route from universal forwarder to all the 5 HF but we are receiving traffic from only old 2 HF but not from 3 newly introduced HF.

telnet from UF to HF is working fine and input and output are configured properly.

Can any one suggest solution for this.

Thanks.

Re: HF not receiving logs from UF

anil8 — Sun, 23 Jan 2022 12:48:51 GMT

Just missed to mentioned, I can see internal log from HF(new) to indexer. so, there is connectivity b/w HF to indexer.

Re: HF not receiving logs from UF

SinghK — Sun, 23 Jan 2022 12:55:10 GMT

2things to check for starters

Is there input configured on hf to receive logs from uf

And from uf are you getting anything on hf try netstat -aon |grep port on which you are forwarding.

Re: HF not receiving logs from UF

SanjayReddy — Sun, 23 Jan 2022 12:58:02 GMT

Hello @anil8

Did you restarted UF after updating the new HF list in outputs.conf of UF? .

if Yes can you please run following command from Splunk bin directory on UF to check forwarders list

./splunk list forward-server

o/p
Active forwards:

Configured but inactive forwards:

2. from Internal logs on UF did you see nay WARN or ERROR messages for those new HFs configured

Re: HF not receiving logs from UF

isoutamo — Mon, 24 Jan 2022 07:20:17 GMT

Have you updated your outputs.conf on UFs to use also these 3 new HFs and restart those after that?

Re: HF not receiving logs from UF

anil8 — Mon, 24 Jan 2022 07:23:34 GMT

yes, We have updated outputs.conf for for 3 new HF

Re: HF not receiving logs from UF

anil8 — Thu, 27 Jan 2022 09:57:45 GMT

Hi @SanjayReddy ,

Thanks for your reply.

I am getting below error in Splunkd log in in HF.

01-27-2022 09:41:18.678 +0000 ERROR TcpInputProc [17904 FwdDataReceiverThread] - Message rejected. Received unexpected message of size=369295616 bytes from src=XXXX(UF) in streaming mode. Maximum message size allowed=67108864. (::) Possible invalid source sending data to splunktcp port or valid source sending unsupported payload.

and below one is from UF:

01-27-2022 09:52:47.004 +0000 WARN TcpOutputFd - Connect to XXXX:9997(HF failed. Connection refused
01-27-2022 09:52:47.004 +0000 ERROR TcpOutputFd - Connection to host=XXXX:9997(HF) failed

Re: HF not receiving logs from UF

SinghK — Thu, 27 Jan 2022 15:03:19 GMT

can you paste the inputs.conf from hf after removing sensitive data.

Re: HF not receiving logs from UF

anil8 — Thu, 27 Jan 2022 15:20:48 GMT

Hi @SinghK ,

Thanks for your response.

Please find the below inputs.conf in system/default and system/local.

Please free to ask any other details.

[default]$ SplunkHome/etc/system/default

# Version 8.2.0
# DO NOT EDIT THIS FILE!
# Changes to default files will be lost on update and are difficult to
# manage and support.
#
# Please make any changes to system defaults by overriding them in
# apps or $SPLUNK_HOME/etc/system/local
# (See "Configuration file precedence" in the web documentation).
#
# To override a specific setting, copy the name of the stanza and
# setting to the file where you wish to override it.
#
# This file contains possible attributes and values you can use to
# configure inputs, distributed inputs and file system monitoring.

[default]
index = default
_rcvbuf = 1572864
host = $decideOnStartup

[blacklist:$SPLUNK_HOME/etc/auth]

[blacklist:$SPLUNK_HOME/etc/passwd]

[monitor://$SPLUNK_HOME/var/log/splunk]
index = _internal

[monitor://$SPLUNK_HOME/var/log/watchdog/watchdog.log*]
index = _internal

[monitor://$SPLUNK_HOME/var/log/splunk/license_usage_summary.log]
index = _telemetry

[monitor://$SPLUNK_HOME/var/log/splunk/splunk_instrumentation_cloud.log*]
index = _telemetry
sourcetype = splunk_cloud_telemetry

[monitor://$SPLUNK_HOME/etc/splunk.version]
_TCP_ROUTING = *
index = _internal
sourcetype=splunk_version

[batch://$SPLUNK_HOME/var/run/splunk/search_telemetry/*search_telemetry.json]
move_policy = sinkhole
index = _introspection
sourcetype = search_telemetry
crcSalt = <SOURCE>
log_on_completion = 0

[batch://$SPLUNK_HOME/var/spool/splunk]
move_policy = sinkhole
crcSalt = <SOURCE>

[batch://$SPLUNK_HOME/var/spool/splunk/tracker.log*]
index = _internal
sourcetype = splunkd_latency_tracker
move_policy = sinkhole

[batch://$SPLUNK_HOME/var/spool/splunk/...stash_new]
queue = stashparsing
sourcetype = stash_new
move_policy = sinkhole
crcSalt = <SOURCE>
time_before_close = 0

[batch://$SPLUNK_HOME/var/spool/splunk/...stash_hec]
sourcetype = stash_hec
move_policy = sinkhole
crcSalt = <SOURCE>

[fschange:$SPLUNK_HOME/etc]
disabled = false
#poll every 10 minutes
pollPeriod = 600
#generate audit events into the audit index, instead of fschange events
signedaudit=true
recurse=true
followLinks=false
hashMaxSize=-1
fullEvent=false
sendEventMaxSize=-1
filesPerDelay = 10
delayInMills = 100

[udp]
connection_host=ip

[tcp]
acceptFrom=*
connection_host=dns

[splunktcp]
route=has_key:_replicationBucketUUID:replicationQueue;has_key:_dstrx:typingQueue;has_key:_linebreaker:indexQueue;absent_key:_linebreaker:parsingQueue
acceptFrom=*
connection_host=ip

[script]
interval = 60.0
start_by_shell = true

[SSL]
# SSL settings
# The following provides modern TLS configuration that guarantees forward-
# secrecy and efficiency. This configuration drops support for old Splunk
# versions (Splunk 5.x and earlier).
# To add support for Splunk 5.x set sslVersions to tls and add this to the
# end of cipherSuite:
# DHE-RSA-AES256-SHA:AES256-SHA:DHE-RSA-AES128-SHA:AES128-SHA
# and this, in case Diffie Hellman is not configured:
# AES256-SHA:AES128-SHA

sslVersions = tls1.2
cipherSuite = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
ecdhCurves = prime256v1, secp384r1, secp521r1

allowSslRenegotiation = true
sslQuietShutdown = false

$ /opt/ee_splunk/splunk/etc/system/local

[monitor:///tmp/test]
disabled = 0
index = test

Re: HF not receiving logs from UF

SinghK — Thu, 27 Jan 2022 15:29:46 GMT

Ah there is no input defined to receive logs from UF.

[splunktcp://<port you are connecting with hf from uf .] e.g. [splunktcp://:9997]

index= <your index>

disabled =0

and add that line to at bottom of file or in local directory create a inpust.conf and put the info there and restart splunk service.

Re: HF not receiving logs from UF

anil8 — Thu, 27 Jan 2022 16:48:40 GMT

Hi @SinghK ,

I added the below in local inputs.conf but still i am getting the same error. 9997 is the port on which UF is sending the logs

[splunktcp://:9997]
index = test
disabled = 0

Re: HF not receiving logs from UF

SinghK — Thu, 27 Jan 2022 16:53:42 GMT

Can you please list the output of this command

Splunk_home/bin/splunk btool inputs list --debug

Re: HF not receiving logs from UF

anil8 — Fri, 28 Jan 2022 09:06:20 GMT

Hi @SinghK ,

Please find the output of Splunk_home/bin/splunk btool inputs list --debug below

Kindly let me know if any other information required.

[splunk@XXXX bin]$ ./splunk btool inputs list --debug
SPLUNK_HOME/apps/splunk_UF_HF_custom_ssl/local/inputs.conf [SSL]
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/system/default/inputs.conf allowSslRenegotiation = true
SPLUNK_HOME/system/default/inputs.conf cipherSuite = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
SPLUNK_HOME/system/default/inputs.conf ecdhCurves = prime256v1, secp384r1, secp521r1
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/system/default/inputs.conf index = default
SPLUNK_HOME/apps/splunk_UF_HF_custom_ssl/local/inputs.conf requireClientCert = false
SPLUNK_HOME/apps/splunk_UF_HF_custom_ssl/local/inputs.conf serverCert = SPLUNK_HOME/apps/splunk_UF_HF_certificates/auth/HFServerCertificate.pem
SPLUNK_HOME/apps/splunk_UF_HF_custom_ssl/local/inputs.conf sslPassword = $7$3z35Ihr9fi8UGaqQIwH/hRBJRYUhe6Icor3Ajha+rVovWHLQplcosqLi
SPLUNK_HOME/system/default/inputs.conf sslQuietShutdown = false
SPLUNK_HOME/system/default/inputs.conf sslVersions = tls1.2
SPLUNK_HOME/system/default/inputs.conf [batch:///opt/ee_splunk/splunk/var/run/splunk/search_telemetry/*search_telemetry.json]
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/system/default/inputs.conf crcSalt = <SOURCE>
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/system/default/inputs.conf index = _introspection
SPLUNK_HOME/system/default/inputs.conf log_on_completion = 0
SPLUNK_HOME/system/default/inputs.conf move_policy = sinkhole
SPLUNK_HOME/system/default/inputs.conf sourcetype = search_telemetry
SPLUNK_HOME/system/default/inputs.conf [batch:///opt/ee_splunk/splunk/var/spool/splunk]
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/system/default/inputs.conf crcSalt = <SOURCE>
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/system/default/inputs.conf index = default
SPLUNK_HOME/system/default/inputs.conf move_policy = sinkhole
SPLUNK_HOME/system/default/inputs.conf [batch:///opt/ee_splunk/splunk/var/spool/splunk/...stash_hec]
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/system/default/inputs.conf crcSalt = <SOURCE>
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/system/default/inputs.conf index = default
SPLUNK_HOME/system/default/inputs.conf move_policy = sinkhole
SPLUNK_HOME/system/default/inputs.conf sourcetype = stash_hec
SPLUNK_HOME/system/default/inputs.conf [batch:///opt/ee_splunk/splunk/var/spool/splunk/...stash_new]
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/system/default/inputs.conf crcSalt = <SOURCE>
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/system/default/inputs.conf index = default
SPLUNK_HOME/system/default/inputs.conf move_policy = sinkhole
SPLUNK_HOME/system/default/inputs.conf queue = stashparsing
SPLUNK_HOME/system/default/inputs.conf sourcetype = stash_new
SPLUNK_HOME/system/default/inputs.conf time_before_close = 0
SPLUNK_HOME/system/default/inputs.conf [batch:///opt/ee_splunk/splunk/var/spool/splunk/tracker.log*]
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/system/default/inputs.conf index = _internal
SPLUNK_HOME/system/default/inputs.conf move_policy = sinkhole
SPLUNK_HOME/system/default/inputs.conf sourcetype = splunkd_latency_tracker
SPLUNK_HOME/system/default/inputs.conf [blacklist:SPLUNK_HOME/auth]
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/system/default/inputs.conf index = default
SPLUNK_HOME/system/default/inputs.conf [blacklist:SPLUNK_HOME/passwd]
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/system/default/inputs.conf index = default
SPLUNK_HOME/system/default/inputs.conf [fschange:SPLUNK_HOME]
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/system/default/inputs.conf delayInMills = 100
SPLUNK_HOME/system/default/inputs.conf disabled = false
SPLUNK_HOME/system/default/inputs.conf filesPerDelay = 10
SPLUNK_HOME/system/default/inputs.conf followLinks = false
SPLUNK_HOME/system/default/inputs.conf fullEvent = false
SPLUNK_HOME/system/default/inputs.conf hashMaxSize = -1
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/system/default/inputs.conf index = default
SPLUNK_HOME/system/default/inputs.conf pollPeriod = 600
SPLUNK_HOME/system/default/inputs.conf recurse = true
SPLUNK_HOME/system/default/inputs.conf sendEventMaxSize = -1
SPLUNK_HOME/system/default/inputs.conf signedaudit = true
SPLUNK_HOME/apps/splunk_httpinput/default/inputs.conf [http]
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/apps/splunk_httpinput/default/inputs.conf ackIdleCleanup = true
SPLUNK_HOME/apps/splunk_httpinput/default/inputs.conf allowSslCompression = true
SPLUNK_HOME/apps/splunk_httpinput/default/inputs.conf allowSslRenegotiation = true
SPLUNK_HOME/apps/splunk_httpinput/default/inputs.conf dedicatedIoThreads = 2
SPLUNK_HOME/apps/splunk_httpinput/default/inputs.conf disabled = 1
SPLUNK_HOME/apps/splunk_httpinput/default/inputs.conf enableSSL = 1
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/system/default/inputs.conf index = default
SPLUNK_HOME/apps/splunk_httpinput/default/inputs.conf maxSockets = 0
SPLUNK_HOME/apps/splunk_httpinput/default/inputs.conf maxThreads = 0
SPLUNK_HOME/apps/splunk_httpinput/default/inputs.conf port = 8088
SPLUNK_HOME/apps/splunk_httpinput/default/inputs.conf sslVersions = *,-ssl2
SPLUNK_HOME/apps/splunk_httpinput/default/inputs.conf useDeploymentServer = 0
SPLUNK_HOME/system/default/inputs.conf [monitor://SPLUNK_HOME/splunk.version]
SPLUNK_HOME/system/default/inputs.conf _TCP_ROUTING = *
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/system/default/inputs.conf index = _internal
SPLUNK_HOME/system/default/inputs.conf sourcetype = splunk_version
SPLUNK_HOME/apps/introspection_generator_addon/default/inputs.conf [monitor:///opt/ee_splunk/splunk/var/log/introspection]
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/apps/introspection_generator_addon/default/inputs.conf index = _introspection
SPLUNK_HOME/system/default/inputs.conf [monitor:///opt/ee_splunk/splunk/var/log/splunk]
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/system/default/inputs.conf index = _internal
SPLUNK_HOME/system/default/inputs.conf [monitor:///opt/ee_splunk/splunk/var/log/splunk/license_usage_summary.log]
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/system/default/inputs.conf index = _telemetry
SPLUNK_HOME/system/default/inputs.conf [monitor:///opt/ee_splunk/splunk/var/log/splunk/splunk_instrumentation_cloud.log*]
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/system/default/inputs.conf index = _telemetry
SPLUNK_HOME/system/default/inputs.conf sourcetype = splunk_cloud_telemetry
SPLUNK_HOME/system/default/inputs.conf [monitor:///opt/ee_splunk/splunk/var/log/watchdog/watchdog.log*]
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/system/default/inputs.conf index = _internal
SPLUNK_HOME/system/default/inputs.conf [script]
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/system/default/inputs.conf index = default
SPLUNK_HOME/system/default/inputs.conf interval = 60.0
SPLUNK_HOME/system/default/inputs.conf start_by_shell = true
SPLUNK_HOME/apps/introspection_generator_addon/default/inputs.conf [script://SPLUNK_HOME/apps/introspection_generator_addon/bin/collector.path]
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/system/default/inputs.conf index = default
SPLUNK_HOME/apps/introspection_generator_addon/default/inputs.conf interval = 0
SPLUNK_HOME/apps/introspection_generator_addon/default/inputs.conf sourcetype = splunk_resource_usage__internal
SPLUNK_HOME/apps/python_upgrade_readiness_app/default/inputs.conf [script://SPLUNK_HOME/apps/python_upgrade_readiness_app/bin/pura_get_all_apps.py]
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/apps/python_upgrade_readiness_app/default/inputs.conf disabled = 0
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/system/default/inputs.conf index = default
SPLUNK_HOME/apps/python_upgrade_readiness_app/default/inputs.conf interval = 00 23 */1 * *
SPLUNK_HOME/apps/python_upgrade_readiness_app/default/inputs.conf passAuth = admin
SPLUNK_HOME/apps/python_upgrade_readiness_app/default/inputs.conf python.version = python3
SPLUNK_HOME/apps/python_upgrade_readiness_app/default/inputs.conf sourcetype = script
SPLUNK_HOME/apps/python_upgrade_readiness_app/default/inputs.conf [script://SPLUNK_HOME/apps/python_upgrade_readiness_app/bin/pura_scan_apps.py]
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/apps/python_upgrade_readiness_app/default/inputs.conf disabled = 0
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/system/default/inputs.conf index = default
SPLUNK_HOME/apps/python_upgrade_readiness_app/default/inputs.conf interval = 00 1 */1 * *
SPLUNK_HOME/apps/python_upgrade_readiness_app/default/inputs.conf passAuth = admin
SPLUNK_HOME/apps/python_upgrade_readiness_app/default/inputs.conf python.version = python3
SPLUNK_HOME/apps/python_upgrade_readiness_app/default/inputs.conf sourcetype = script
SPLUNK_HOME/apps/python_upgrade_readiness_app/default/inputs.conf [script://SPLUNK_HOME/apps/python_upgrade_readiness_app/bin/pura_send_email.py]
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/apps/python_upgrade_readiness_app/default/inputs.conf disabled = 0
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/system/default/inputs.conf index = default
SPLUNK_HOME/apps/python_upgrade_readiness_app/default/inputs.conf interval = 0 6 * * 1
SPLUNK_HOME/apps/python_upgrade_readiness_app/default/inputs.conf passAuth = admin
SPLUNK_HOME/apps/python_upgrade_readiness_app/default/inputs.conf python.version = python3
SPLUNK_HOME/apps/python_upgrade_readiness_app/default/inputs.conf sourcetype = script
SPLUNK_HOME/apps/splunk-dashboard-studio/default/inputs.conf [script://SPLUNK_HOME/apps/splunk-dashboard-studio/bin/save_image_and_icon_on_install.py]
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/system/default/inputs.conf index = default
SPLUNK_HOME/apps/splunk-dashboard-studio/default/inputs.conf interval = -1
SPLUNK_HOME/apps/splunk-dashboard-studio/default/inputs.conf passAuth = splunk-system-user
SPLUNK_HOME/apps/splunk-dashboard-studio/default/inputs.conf python.version = python3
SPLUNK_HOME/apps/splunk-dashboard-studio/default/inputs.conf run_only_one = false
SPLUNK_HOME/apps/splunk_instrumentation/default/inputs.conf [script://SPLUNK_HOME/apps/splunk_instrumentation/bin/instrumentation.py]
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/apps/splunk_instrumentation/default/inputs.conf disabled = false
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/apps/splunk_instrumentation/default/inputs.conf index = _telemetry
SPLUNK_HOME/apps/splunk_instrumentation/default/inputs.conf interval = 0 * * * *
SPLUNK_HOME/apps/splunk_instrumentation/default/inputs.conf passAuth = splunk-system-user
SPLUNK_HOME/apps/splunk_instrumentation/default/inputs.conf python.version = python3
SPLUNK_HOME/apps/splunk_instrumentation/default/inputs.conf source = instrumentation_scripted_input
SPLUNK_HOME/apps/splunk_instrumentation/default/inputs.conf sourcetype = splunk_telemetry_log
SPLUNK_HOME/apps/splunk_instrumentation/default/inputs.conf [script://SPLUNK_HOME/apps/splunk_instrumentation/bin/on_splunk_start.py]
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/apps/splunk_instrumentation/default/inputs.conf disabled = false
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/system/default/inputs.conf index = default
SPLUNK_HOME/apps/splunk_instrumentation/default/inputs.conf interval = -1
SPLUNK_HOME/apps/splunk_instrumentation/default/inputs.conf passAuth = splunk-system-user
SPLUNK_HOME/apps/splunk_instrumentation/default/inputs.conf python.version = python3
SPLUNK_HOME/apps/splunk_instrumentation/default/inputs.conf [script://SPLUNK_HOME/apps/splunk_instrumentation/bin/schedule_delete.py]
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/apps/splunk_instrumentation/default/inputs.conf disabled = false
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/system/default/inputs.conf index = default
SPLUNK_HOME/apps/splunk_instrumentation/default/inputs.conf interval = 0 0 * * *
SPLUNK_HOME/apps/splunk_instrumentation/default/inputs.conf passAuth = splunk-system-user
SPLUNK_HOME/apps/splunk_instrumentation/default/inputs.conf python.version = python3
SPLUNK_HOME/apps/splunk_monitoring_console/default/inputs.conf [script://SPLUNK_HOME/apps/splunk_monitoring_console/bin/dmc_config.py]
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/system/default/inputs.conf index = default
SPLUNK_HOME/apps/splunk_monitoring_console/default/inputs.conf interval = -1
SPLUNK_HOME/apps/splunk_monitoring_console/default/inputs.conf passAuth = splunk-system-user
SPLUNK_HOME/apps/splunk_monitoring_console/default/inputs.conf python.version = python3
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf [secure_gateway_modular_input://default]
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf disabled = 1

Re: HF not receiving logs from UF

anil8 — Fri, 28 Jan 2022 09:05:17 GMT

SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf [secure_gateway_modular_input://default]
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf disabled = 1
host = $decideOnStartup
index = default
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf interval = 1
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf python.version = python3
SPLUNK_HOME/system/default/inputs.conf [splunktcp]
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/system/default/inputs.conf acceptFrom = *
SPLUNK_HOME/system/default/inputs.conf connection_host = ip
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/system/default/inputs.conf index = default
SPLUNK_HOME/system/default/inputs.conf route = has_key:_replicationBucketUUID:replicationQueue;has_key:_dstrx:typingQueue;has_key:_linebreaker:indexQueue;absent_key:_linebreaker:parsingQueue
SPLUNK_HOME/apps/splunk_UF_HF_custom_ssl/local/inputs.conf [splunktcp-ssl://42000]
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/system/default/inputs.conf index = default
SPLUNK_HOME/apps/splunk_UF_HF_custom_ssl/local/inputs.conf [splunktcp-ssl://9997]
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/system/default/inputs.conf index = default
SPLUNK_HOME/apps/launcher/local/inputs.conf [splunktcp://42000]
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/apps/launcher/local/inputs.conf connection_host = ip
SPLUNK_HOME/apps/launcher/local/inputs.conf disabled = 0
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/system/default/inputs.conf index = default
SPLUNK_HOME/apps/launcher/local/inputs.conf [splunktcp://9997]
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/apps/launcher/local/inputs.conf connection_host = ip
SPLUNK_HOME/apps/launcher/local/inputs.conf disabled = 0
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/system/default/inputs.conf index = default
SPLUNK_HOME/system/local/inputs.conf [splunktcp://:9997]
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/system/local/inputs.conf disabled = 0
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/system/local/inputs.conf index = test
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf [ssg_alerts_ttl_modular_input://default]
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf disabled = 1
host = $decideOnStartup
index = default
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf interval = 3600
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf python.version = python3
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf ttl_days = 1
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf [ssg_delete_tokens_modular_input://default]
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf disabled = 1
host = $decideOnStartup
index = default
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf interval = 7200
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf python.version = python3
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf [ssg_device_role_modular_input://default]
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf disabled = 1
host = $decideOnStartup
index = default
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf interval = 300
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf python.version = python3
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf [ssg_enable_modular_input://default]
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf disabled = 0
host = $decideOnStartup
index = default
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf interval = 60
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf python.version = python3
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf [ssg_metrics_modular_input://default]
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf disabled = 1
host = $decideOnStartup
index = default
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf interval = 43200
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf python.version = python3
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf [ssg_registered_users_list_modular_input://default]
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf disabled = 1
host = $decideOnStartup
index = default
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf interval = 86400
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf python.version = python3
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf [ssg_subscription_clean_up_modular_input://default]
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf cleanup_threshold_seconds = 120
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf disabled = 1
host = $decideOnStartup
index = default
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf interval = 120
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf python.version = python3
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf [ssg_subscription_modular_input://default]
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf disabled = 1
host = $decideOnStartup
index = default
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf interval = 0
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf maximum_iteration_time_warn_threshold_seconds = 300
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf minimum_iteration_time_seconds = 5
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf python.version = python3
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf subscription_processor_parallelism = N_CPU
SPLUNK_HOME/system/default/inputs.conf [tcp]
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/system/default/inputs.conf acceptFrom = *
SPLUNK_HOME/system/default/inputs.conf connection_host = dns
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/system/default/inputs.conf index = default
SPLUNK_HOME/system/default/inputs.conf [udp]
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/system/default/inputs.conf connection_host = ip
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/system/default/inputs.conf index = default
SPLUNK_HOME/apps/btec_t_input_apigw_op/default/inputs.conf [udp://10714]
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/apps/btec_t_input_apigw_op/default/inputs.conf acceptFrom = 10.50.146.148,10.50.146.149,10.50.146.150,10.50.146.151,10.36.85.77,10.36.85.78,10.36.85.79,10.36.85.80
SPLUNK_HOME/apps/btec_t_input_apigw_op/default/inputs.conf disabled = false
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/apps/btec_t_input_apigw_op/default/inputs.conf index = rt_apigw
SPLUNK_HOME/apps/btec_t_input_apigw_op/default/inputs.conf sourcetype = syslog
SPLUNK_HOME/apps/btec_p_input_excalibur/default/inputs.conf [udp://10715]
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/apps/btec_p_input_excalibur/default/inputs.conf acceptFrom = 10.45.26.137,10.45.26.138,10.45.26.139,10.45.26.140,10.45.9.15,10.45.52.227,10.45.52.228
SPLUNK_HOME/apps/btec_p_input_excalibur/default/inputs.conf disabled = false
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/apps/btec_p_input_excalibur/default/inputs.conf index = rt_excalibur
SPLUNK_HOME/apps/btec_p_input_excalibur/default/inputs.conf sourcetype = excalibur_LB

Re: HF not receiving logs from UF

isoutamo — Fri, 28 Jan 2022 09:16:50 GMT

It seems that you have tried configure both splunktcp and splunktcp-ssl on port 9997. Based on your app names etc. I suppose that splunktcp:9997 is winning listener. Then you probably try to send events from UF by splunktcp-ssl (port 9997 or 42000) and as 9997 is working with SSL it didn't accept your connection.

Please check from UF side which outputs.conf is in use and in which protocol and port it try to use.

Re: HF not receiving logs from UF

SinghK — Fri, 28 Jan 2022 09:17:18 GMT

you have duplicate inputs configured on same port

SPLUNK_HOME/apps/splunk_UF_HF_custom_ssl/local/inputs.conf [splunktcp-ssl://42000]
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/system/default/inputs.conf index = default
SPLUNK_HOME/apps/splunk_UF_HF_custom_ssl/local/inputs.conf [splunktcp-ssl://9997]
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/system/default/inputs.conf index = default
SPLUNK_HOME/apps/launcher/local/inputs.conf [splunktcp://42000]
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/apps/launcher/local/inputs.conf connection_host = ip
SPLUNK_HOME/apps/launcher/local/inputs.conf disabled = 0
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/system/default/inputs.conf index = default
SPLUNK_HOME/apps/launcher/local/inputs.conf [splunktcp://9997]
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/apps/launcher/local/inputs.conf connection_host = ip
SPLUNK_HOME/apps/launcher/local/inputs.conf disabled = 0
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/system/default/inputs.conf index = default
SPLUNK_HOME/system/local/inputs.conf [splunktcp://:9997]
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/system/local/inputs.conf disabled = 0
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/system/local/inputs.conf index = test
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf

do you have ssl enabled in your environment ? as i can see you have some ssl inputs configured.

if the answer to this is yes have created ssl certs for new hf's ?

Re: HF not receiving logs from UF

anil8 — Fri, 28 Jan 2022 09:42:55 GMT

Yes...I have created the SSL certificate.

Re: HF not receiving logs from UF

SinghK — Fri, 28 Jan 2022 09:53:01 GMT

Please paste the outputs.conf from uf

Re: HF not receiving logs from UF

anil8 — Fri, 28 Jan 2022 10:02:44 GMT

Hi @SinghK ,

Please find below output.conf from UF

[tcpout]
defaultGroup = splunkssl
useACK = true

[tcpout:splunkssl]
server = XXXX.XX.XX.com:9997,XXXX.XX.XX.com:9997,XXXX.XX.XX.com:9997,XXXX.XX.XX.com:9997,XXXX.XX.XX.com:9997
sslCertPath = /opt/product/splunk/splunkforwarder_ee/etc/apps/ee_splunk_forwarder_certs/auth/Forwarder.pem
sslPassword = ee_splunk_iif
sslRootCAPath = /opt/product/splunk/splunkforwarder_ee/etc/apps/ee_splunk_forwarder_certs/auth/UFHFCACertificate.pem
sslVerifyServerCert = false
useSSL=true

Re: HF not receiving logs from UF

anil8 — Fri, 28 Jan 2022 10:12:23 GMT

Hi @isoutamo ,

Can you please advice what change should be done in output.conf in UF.

[tcpout]
defaultGroup = splunkssl
useACK = true