Getting Data In
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Ask a Question

HF not receiving logs from UF

anil8
Loves-to-Learn Everything
‎01-23-2022 04:43 AM

Hi Splunkers,

We have configured 3 new heavy forwarder in our splunk enterprise where 2 HF was already working.

Now we want traffic route from universal forwarder to  all the 5 HF but we are receiving traffic from only old 2 HF but not from 3 newly introduced HF.

telnet from UF to HF is working fine and input and output are configured properly.

Can any one suggest solution for this.

 Thanks.

0 Karma
Reply

pdudhaiya
Splunk Employee
Splunk Employee
‎01-31-2022 07:13 AM

@anil8  Did you try setting some other port apart from 9997 as the receiving port on the new 3 HFs ?

Does that work. Once you know the answer to that, you can have some more clarity for debugging this.

0 Karma
Reply

isoutamo
SplunkTrust
SplunkTrust
‎01-23-2022 11:20 PM

Have you updated your outputs.conf on UFs to use also these 3 new HFs and restart those after that?

0 Karma
Reply

anil8
Loves-to-Learn Everything
‎01-23-2022 11:23 PM

yes, We have updated outputs.conf for for 3 new HF

0 Karma
Reply

anil8
Loves-to-Learn Everything
‎01-23-2022 04:48 AM

Just missed to mentioned, I can see internal log from HF(new) to indexer. so, there is connectivity b/w HF to indexer.

0 Karma
Reply

SanjayReddy
SplunkTrust
SplunkTrust
‎01-23-2022 04:56 AM



Hello  @anil8 

Did you restarted UF  after updating the new HF list in outputs.conf  of UF? .

if Yes can you please run following command from Splunk bin directory on UF to check forwarders list 

./splunk list forward-server

o/p
Active forwards:

Configured but inactive forwards:


2. from Internal logs on UF did  you see nay WARN or ERROR messages for those new HFs configured 

0 Karma
Reply

anil8
Loves-to-Learn Everything
‎01-27-2022 01:57 AM

Hi @SanjayReddy ,

 

Thanks for your reply.

I am getting below error in Splunkd log in in HF.

01-27-2022 09:41:18.678 +0000 ERROR TcpInputProc [17904 FwdDataReceiverThread] - Message rejected. Received unexpected message of size=369295616 bytes from src=XXXX(UF) in streaming mode. Maximum message size allowed=67108864. (::) Possible invalid source sending data to splunktcp port or valid source sending unsupported payload.

 

and below one is from UF:

01-27-2022 09:52:47.004 +0000 WARN TcpOutputFd - Connect to XXXX:9997(HF failed. Connection refused
01-27-2022 09:52:47.004 +0000 ERROR TcpOutputFd - Connection to host=XXXX:9997(HF) failed

 

0 Karma
Reply

SinghK
Builder
‎01-27-2022 07:03 AM

can you paste the inputs.conf from hf after removing sensitive data.

0 Karma
Reply

anil8
Loves-to-Learn Everything
‎01-27-2022 07:20 AM

Hi @SinghK ,

Thanks for your response.

Please find the below inputs.conf in system/default and system/local.

Please free to ask any other details.

[default]$ SplunkHome/etc/system/default


# Version 8.2.0
# DO NOT EDIT THIS FILE!
# Changes to default files will be lost on update and are difficult to
# manage and support.
#
# Please make any changes to system defaults by overriding them in
# apps or $SPLUNK_HOME/etc/system/local
# (See "Configuration file precedence" in the web documentation).
#
# To override a specific setting, copy the name of the stanza and
# setting to the file where you wish to override it.
#
# This file contains possible attributes and values you can use to
# configure inputs, distributed inputs and file system monitoring.


[default]
index = default
_rcvbuf = 1572864
host = $decideOnStartup

[blacklist:$SPLUNK_HOME/etc/auth]

[blacklist:$SPLUNK_HOME/etc/passwd]

[monitor://$SPLUNK_HOME/var/log/splunk]
index = _internal

[monitor://$SPLUNK_HOME/var/log/watchdog/watchdog.log*]
index = _internal

[monitor://$SPLUNK_HOME/var/log/splunk/license_usage_summary.log]
index = _telemetry

[monitor://$SPLUNK_HOME/var/log/splunk/splunk_instrumentation_cloud.log*]
index = _telemetry
sourcetype = splunk_cloud_telemetry

[monitor://$SPLUNK_HOME/etc/splunk.version]
_TCP_ROUTING = *
index = _internal
sourcetype=splunk_version

[batch://$SPLUNK_HOME/var/run/splunk/search_telemetry/*search_telemetry.json]
move_policy = sinkhole
index = _introspection
sourcetype = search_telemetry
crcSalt = <SOURCE>
log_on_completion = 0

[batch://$SPLUNK_HOME/var/spool/splunk]
move_policy = sinkhole
crcSalt = <SOURCE>

[batch://$SPLUNK_HOME/var/spool/splunk/tracker.log*]
index = _internal
sourcetype = splunkd_latency_tracker
move_policy = sinkhole

[batch://$SPLUNK_HOME/var/spool/splunk/...stash_new]
queue = stashparsing
sourcetype = stash_new
move_policy = sinkhole
crcSalt = <SOURCE>
time_before_close = 0

[batch://$SPLUNK_HOME/var/spool/splunk/...stash_hec]
sourcetype = stash_hec
move_policy = sinkhole
crcSalt = <SOURCE>

[fschange:$SPLUNK_HOME/etc]
disabled = false
#poll every 10 minutes
pollPeriod = 600
#generate audit events into the audit index, instead of fschange events
signedaudit=true
recurse=true
followLinks=false
hashMaxSize=-1
fullEvent=false
sendEventMaxSize=-1
filesPerDelay = 10
delayInMills = 100

[udp]
connection_host=ip

[tcp]
acceptFrom=*
connection_host=dns

[splunktcp]
route=has_key:_replicationBucketUUID:replicationQueue;has_key:_dstrx:typingQueue;has_key:_linebreaker:indexQueue;absent_key:_linebreaker:parsingQueue
acceptFrom=*
connection_host=ip

[script]
interval = 60.0
start_by_shell = true

[SSL]
# SSL settings
# The following provides modern TLS configuration that guarantees forward-
# secrecy and efficiency. This configuration drops support for old Splunk
# versions (Splunk 5.x and earlier).
# To add support for Splunk 5.x set sslVersions to tls and add this to the
# end of cipherSuite:
# DHE-RSA-AES256-SHA:AES256-SHA:DHE-RSA-AES128-SHA:AES128-SHA
# and this, in case Diffie Hellman is not configured:
# AES256-SHA:AES128-SHA

sslVersions = tls1.2
cipherSuite = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
ecdhCurves = prime256v1, secp384r1, secp521r1

allowSslRenegotiation = true
sslQuietShutdown = false

 

 

$ /opt/ee_splunk/splunk/etc/system/local

[monitor:///tmp/test]
disabled = 0
index = test

 

 

 

0 Karma
Reply

SinghK
Builder
‎01-27-2022 07:29 AM

Ah there is no input defined to receive logs from UF.

[splunktcp://<port you are connecting with hf from uf .] e.g. [splunktcp://:9997]

index= <your index>

disabled =0 

 

and add that line to at bottom of file or in local directory create a inpust.conf and put the info there and restart splunk service.

0 Karma
Reply

anil8
Loves-to-Learn Everything
‎01-27-2022 08:48 AM

Hi @SinghK ,

I added the below in local inputs.conf but still i am getting the same error. 9997 is the port on which UF is sending the logs

[splunktcp://:9997]
index = test
disabled = 0

Tags (1)
0 Karma
Reply

SinghK
Builder
‎01-27-2022 08:53 AM

Can you please list the output of this command 

Splunk_home/bin/splunk btool inputs list --debug

0 Karma
Reply

anil8
Loves-to-Learn Everything
‎01-28-2022 01:03 AM

Hi @SinghK ,

Please find the output of Splunk_home/bin/splunk btool inputs list --debug below

Kindly let me know if any other information required.

[splunk@XXXX bin]$ ./splunk btool inputs list --debug
SPLUNK_HOME/apps/splunk_UF_HF_custom_ssl/local/inputs.conf [SSL]
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/system/default/inputs.conf allowSslRenegotiation = true
SPLUNK_HOME/system/default/inputs.conf cipherSuite = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
SPLUNK_HOME/system/default/inputs.conf ecdhCurves = prime256v1, secp384r1, secp521r1
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/system/default/inputs.conf index = default
SPLUNK_HOME/apps/splunk_UF_HF_custom_ssl/local/inputs.conf requireClientCert = false
SPLUNK_HOME/apps/splunk_UF_HF_custom_ssl/local/inputs.conf serverCert = SPLUNK_HOME/apps/splunk_UF_HF_certificates/auth/HFServerCertificate.pem
SPLUNK_HOME/apps/splunk_UF_HF_custom_ssl/local/inputs.conf sslPassword = $7$3z35Ihr9fi8UGaqQIwH/hRBJRYUhe6Icor3Ajha+rVovWHLQplcosqLi
SPLUNK_HOME/system/default/inputs.conf sslQuietShutdown = false
SPLUNK_HOME/system/default/inputs.conf sslVersions = tls1.2
SPLUNK_HOME/system/default/inputs.conf [batch:///opt/ee_splunk/splunk/var/run/splunk/search_telemetry/*search_telemetry.json]
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/system/default/inputs.conf crcSalt = <SOURCE>
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/system/default/inputs.conf index = _introspection
SPLUNK_HOME/system/default/inputs.conf log_on_completion = 0
SPLUNK_HOME/system/default/inputs.conf move_policy = sinkhole
SPLUNK_HOME/system/default/inputs.conf sourcetype = search_telemetry
SPLUNK_HOME/system/default/inputs.conf [batch:///opt/ee_splunk/splunk/var/spool/splunk]
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/system/default/inputs.conf crcSalt = <SOURCE>
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/system/default/inputs.conf index = default
SPLUNK_HOME/system/default/inputs.conf move_policy = sinkhole
SPLUNK_HOME/system/default/inputs.conf [batch:///opt/ee_splunk/splunk/var/spool/splunk/...stash_hec]
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/system/default/inputs.conf crcSalt = <SOURCE>
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/system/default/inputs.conf index = default
SPLUNK_HOME/system/default/inputs.conf move_policy = sinkhole
SPLUNK_HOME/system/default/inputs.conf sourcetype = stash_hec
SPLUNK_HOME/system/default/inputs.conf [batch:///opt/ee_splunk/splunk/var/spool/splunk/...stash_new]
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/system/default/inputs.conf crcSalt = <SOURCE>
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/system/default/inputs.conf index = default
SPLUNK_HOME/system/default/inputs.conf move_policy = sinkhole
SPLUNK_HOME/system/default/inputs.conf queue = stashparsing
SPLUNK_HOME/system/default/inputs.conf sourcetype = stash_new
SPLUNK_HOME/system/default/inputs.conf time_before_close = 0
SPLUNK_HOME/system/default/inputs.conf [batch:///opt/ee_splunk/splunk/var/spool/splunk/tracker.log*]
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/system/default/inputs.conf index = _internal
SPLUNK_HOME/system/default/inputs.conf move_policy = sinkhole
SPLUNK_HOME/system/default/inputs.conf sourcetype = splunkd_latency_tracker
SPLUNK_HOME/system/default/inputs.conf [blacklist:SPLUNK_HOME/auth]
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/system/default/inputs.conf index = default
SPLUNK_HOME/system/default/inputs.conf [blacklist:SPLUNK_HOME/passwd]
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/system/default/inputs.conf index = default
SPLUNK_HOME/system/default/inputs.conf [fschange:SPLUNK_HOME]
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/system/default/inputs.conf delayInMills = 100
SPLUNK_HOME/system/default/inputs.conf disabled = false
SPLUNK_HOME/system/default/inputs.conf filesPerDelay = 10
SPLUNK_HOME/system/default/inputs.conf followLinks = false
SPLUNK_HOME/system/default/inputs.conf fullEvent = false
SPLUNK_HOME/system/default/inputs.conf hashMaxSize = -1
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/system/default/inputs.conf index = default
SPLUNK_HOME/system/default/inputs.conf pollPeriod = 600
SPLUNK_HOME/system/default/inputs.conf recurse = true
SPLUNK_HOME/system/default/inputs.conf sendEventMaxSize = -1
SPLUNK_HOME/system/default/inputs.conf signedaudit = true
SPLUNK_HOME/apps/splunk_httpinput/default/inputs.conf [http]
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/apps/splunk_httpinput/default/inputs.conf ackIdleCleanup = true
SPLUNK_HOME/apps/splunk_httpinput/default/inputs.conf allowSslCompression = true
SPLUNK_HOME/apps/splunk_httpinput/default/inputs.conf allowSslRenegotiation = true
SPLUNK_HOME/apps/splunk_httpinput/default/inputs.conf dedicatedIoThreads = 2
SPLUNK_HOME/apps/splunk_httpinput/default/inputs.conf disabled = 1
SPLUNK_HOME/apps/splunk_httpinput/default/inputs.conf enableSSL = 1
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/system/default/inputs.conf index = default
SPLUNK_HOME/apps/splunk_httpinput/default/inputs.conf maxSockets = 0
SPLUNK_HOME/apps/splunk_httpinput/default/inputs.conf maxThreads = 0
SPLUNK_HOME/apps/splunk_httpinput/default/inputs.conf port = 8088
SPLUNK_HOME/apps/splunk_httpinput/default/inputs.conf sslVersions = *,-ssl2
SPLUNK_HOME/apps/splunk_httpinput/default/inputs.conf useDeploymentServer = 0
SPLUNK_HOME/system/default/inputs.conf [monitor://SPLUNK_HOME/splunk.version]
SPLUNK_HOME/system/default/inputs.conf _TCP_ROUTING = *
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/system/default/inputs.conf index = _internal
SPLUNK_HOME/system/default/inputs.conf sourcetype = splunk_version
SPLUNK_HOME/apps/introspection_generator_addon/default/inputs.conf [monitor:///opt/ee_splunk/splunk/var/log/introspection]
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/apps/introspection_generator_addon/default/inputs.conf index = _introspection
SPLUNK_HOME/system/default/inputs.conf [monitor:///opt/ee_splunk/splunk/var/log/splunk]
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/system/default/inputs.conf index = _internal
SPLUNK_HOME/system/default/inputs.conf [monitor:///opt/ee_splunk/splunk/var/log/splunk/license_usage_summary.log]
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/system/default/inputs.conf index = _telemetry
SPLUNK_HOME/system/default/inputs.conf [monitor:///opt/ee_splunk/splunk/var/log/splunk/splunk_instrumentation_cloud.log*]
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/system/default/inputs.conf index = _telemetry
SPLUNK_HOME/system/default/inputs.conf sourcetype = splunk_cloud_telemetry
SPLUNK_HOME/system/default/inputs.conf [monitor:///opt/ee_splunk/splunk/var/log/watchdog/watchdog.log*]
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/system/default/inputs.conf index = _internal
SPLUNK_HOME/system/default/inputs.conf [script]
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/system/default/inputs.conf index = default
SPLUNK_HOME/system/default/inputs.conf interval = 60.0
SPLUNK_HOME/system/default/inputs.conf start_by_shell = true
SPLUNK_HOME/apps/introspection_generator_addon/default/inputs.conf [script://SPLUNK_HOME/apps/introspection_generator_addon/bin/collector.path]
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/system/default/inputs.conf index = default
SPLUNK_HOME/apps/introspection_generator_addon/default/inputs.conf interval = 0
SPLUNK_HOME/apps/introspection_generator_addon/default/inputs.conf sourcetype = splunk_resource_usage__internal
SPLUNK_HOME/apps/python_upgrade_readiness_app/default/inputs.conf [script://SPLUNK_HOME/apps/python_upgrade_readiness_app/bin/pura_get_all_apps.py]
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/apps/python_upgrade_readiness_app/default/inputs.conf disabled = 0
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/system/default/inputs.conf index = default
SPLUNK_HOME/apps/python_upgrade_readiness_app/default/inputs.conf interval = 00 23 */1 * *
SPLUNK_HOME/apps/python_upgrade_readiness_app/default/inputs.conf passAuth = admin
SPLUNK_HOME/apps/python_upgrade_readiness_app/default/inputs.conf python.version = python3
SPLUNK_HOME/apps/python_upgrade_readiness_app/default/inputs.conf sourcetype = script
SPLUNK_HOME/apps/python_upgrade_readiness_app/default/inputs.conf [script://SPLUNK_HOME/apps/python_upgrade_readiness_app/bin/pura_scan_apps.py]
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/apps/python_upgrade_readiness_app/default/inputs.conf disabled = 0
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/system/default/inputs.conf index = default
SPLUNK_HOME/apps/python_upgrade_readiness_app/default/inputs.conf interval = 00 1 */1 * *
SPLUNK_HOME/apps/python_upgrade_readiness_app/default/inputs.conf passAuth = admin
SPLUNK_HOME/apps/python_upgrade_readiness_app/default/inputs.conf python.version = python3
SPLUNK_HOME/apps/python_upgrade_readiness_app/default/inputs.conf sourcetype = script
SPLUNK_HOME/apps/python_upgrade_readiness_app/default/inputs.conf [script://SPLUNK_HOME/apps/python_upgrade_readiness_app/bin/pura_send_email.py]
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/apps/python_upgrade_readiness_app/default/inputs.conf disabled = 0
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/system/default/inputs.conf index = default
SPLUNK_HOME/apps/python_upgrade_readiness_app/default/inputs.conf interval = 0 6 * * 1
SPLUNK_HOME/apps/python_upgrade_readiness_app/default/inputs.conf passAuth = admin
SPLUNK_HOME/apps/python_upgrade_readiness_app/default/inputs.conf python.version = python3
SPLUNK_HOME/apps/python_upgrade_readiness_app/default/inputs.conf sourcetype = script
SPLUNK_HOME/apps/splunk-dashboard-studio/default/inputs.conf [script://SPLUNK_HOME/apps/splunk-dashboard-studio/bin/save_image_and_icon_on_install.py]
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/system/default/inputs.conf index = default
SPLUNK_HOME/apps/splunk-dashboard-studio/default/inputs.conf interval = -1
SPLUNK_HOME/apps/splunk-dashboard-studio/default/inputs.conf passAuth = splunk-system-user
SPLUNK_HOME/apps/splunk-dashboard-studio/default/inputs.conf python.version = python3
SPLUNK_HOME/apps/splunk-dashboard-studio/default/inputs.conf run_only_one = false
SPLUNK_HOME/apps/splunk_instrumentation/default/inputs.conf [script://SPLUNK_HOME/apps/splunk_instrumentation/bin/instrumentation.py]
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/apps/splunk_instrumentation/default/inputs.conf disabled = false
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/apps/splunk_instrumentation/default/inputs.conf index = _telemetry
SPLUNK_HOME/apps/splunk_instrumentation/default/inputs.conf interval = 0 * * * *
SPLUNK_HOME/apps/splunk_instrumentation/default/inputs.conf passAuth = splunk-system-user
SPLUNK_HOME/apps/splunk_instrumentation/default/inputs.conf python.version = python3
SPLUNK_HOME/apps/splunk_instrumentation/default/inputs.conf source = instrumentation_scripted_input
SPLUNK_HOME/apps/splunk_instrumentation/default/inputs.conf sourcetype = splunk_telemetry_log
SPLUNK_HOME/apps/splunk_instrumentation/default/inputs.conf [script://SPLUNK_HOME/apps/splunk_instrumentation/bin/on_splunk_start.py]
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/apps/splunk_instrumentation/default/inputs.conf disabled = false
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/system/default/inputs.conf index = default
SPLUNK_HOME/apps/splunk_instrumentation/default/inputs.conf interval = -1
SPLUNK_HOME/apps/splunk_instrumentation/default/inputs.conf passAuth = splunk-system-user
SPLUNK_HOME/apps/splunk_instrumentation/default/inputs.conf python.version = python3
SPLUNK_HOME/apps/splunk_instrumentation/default/inputs.conf [script://SPLUNK_HOME/apps/splunk_instrumentation/bin/schedule_delete.py]
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/apps/splunk_instrumentation/default/inputs.conf disabled = false
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/system/default/inputs.conf index = default
SPLUNK_HOME/apps/splunk_instrumentation/default/inputs.conf interval = 0 0 * * *
SPLUNK_HOME/apps/splunk_instrumentation/default/inputs.conf passAuth = splunk-system-user
SPLUNK_HOME/apps/splunk_instrumentation/default/inputs.conf python.version = python3
SPLUNK_HOME/apps/splunk_monitoring_console/default/inputs.conf [script://SPLUNK_HOME/apps/splunk_monitoring_console/bin/dmc_config.py]
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/system/default/inputs.conf index = default
SPLUNK_HOME/apps/splunk_monitoring_console/default/inputs.conf interval = -1
SPLUNK_HOME/apps/splunk_monitoring_console/default/inputs.conf passAuth = splunk-system-user
SPLUNK_HOME/apps/splunk_monitoring_console/default/inputs.conf python.version = python3
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf [secure_gateway_modular_input://default]
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf disabled = 1

0 Karma
Reply

anil8
Loves-to-Learn Everything
‎01-28-2022 01:05 AM

SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf [secure_gateway_modular_input://default]
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf disabled = 1
host = $decideOnStartup
index = default
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf interval = 1
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf python.version = python3
SPLUNK_HOME/system/default/inputs.conf [splunktcp]
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/system/default/inputs.conf acceptFrom = *
SPLUNK_HOME/system/default/inputs.conf connection_host = ip
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/system/default/inputs.conf index = default
SPLUNK_HOME/system/default/inputs.conf route = has_key:_replicationBucketUUID:replicationQueue;has_key:_dstrx:typingQueue;has_key:_linebreaker:indexQueue;absent_key:_linebreaker:parsingQueue
SPLUNK_HOME/apps/splunk_UF_HF_custom_ssl/local/inputs.conf [splunktcp-ssl://42000]
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/system/default/inputs.conf index = default
SPLUNK_HOME/apps/splunk_UF_HF_custom_ssl/local/inputs.conf [splunktcp-ssl://9997]
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/system/default/inputs.conf index = default
SPLUNK_HOME/apps/launcher/local/inputs.conf [splunktcp://42000]
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/apps/launcher/local/inputs.conf connection_host = ip
SPLUNK_HOME/apps/launcher/local/inputs.conf disabled = 0
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/system/default/inputs.conf index = default
SPLUNK_HOME/apps/launcher/local/inputs.conf [splunktcp://9997]
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/apps/launcher/local/inputs.conf connection_host = ip
SPLUNK_HOME/apps/launcher/local/inputs.conf disabled = 0
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/system/default/inputs.conf index = default
SPLUNK_HOME/system/local/inputs.conf [splunktcp://:9997]
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/system/local/inputs.conf disabled = 0
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/system/local/inputs.conf index = test
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf [ssg_alerts_ttl_modular_input://default]
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf disabled = 1
host = $decideOnStartup
index = default
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf interval = 3600
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf python.version = python3
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf ttl_days = 1
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf [ssg_delete_tokens_modular_input://default]
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf disabled = 1
host = $decideOnStartup
index = default
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf interval = 7200
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf python.version = python3
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf [ssg_device_role_modular_input://default]
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf disabled = 1
host = $decideOnStartup
index = default
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf interval = 300
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf python.version = python3
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf [ssg_enable_modular_input://default]
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf disabled = 0
host = $decideOnStartup
index = default
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf interval = 60
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf python.version = python3
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf [ssg_metrics_modular_input://default]
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf disabled = 1
host = $decideOnStartup
index = default
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf interval = 43200
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf python.version = python3
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf [ssg_registered_users_list_modular_input://default]
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf disabled = 1
host = $decideOnStartup
index = default
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf interval = 86400
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf python.version = python3
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf [ssg_subscription_clean_up_modular_input://default]
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf cleanup_threshold_seconds = 120
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf disabled = 1
host = $decideOnStartup
index = default
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf interval = 120
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf python.version = python3
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf [ssg_subscription_modular_input://default]
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf disabled = 1
host = $decideOnStartup
index = default
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf interval = 0
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf maximum_iteration_time_warn_threshold_seconds = 300
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf minimum_iteration_time_seconds = 5
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf python.version = python3
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf subscription_processor_parallelism = N_CPU
SPLUNK_HOME/system/default/inputs.conf [tcp]
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/system/default/inputs.conf acceptFrom = *
SPLUNK_HOME/system/default/inputs.conf connection_host = dns
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/system/default/inputs.conf index = default
SPLUNK_HOME/system/default/inputs.conf [udp]
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/system/default/inputs.conf connection_host = ip
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/system/default/inputs.conf index = default
SPLUNK_HOME/apps/btec_t_input_apigw_op/default/inputs.conf [udp://10714]
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/apps/btec_t_input_apigw_op/default/inputs.conf acceptFrom = 10.50.146.148,10.50.146.149,10.50.146.150,10.50.146.151,10.36.85.77,10.36.85.78,10.36.85.79,10.36.85.80
SPLUNK_HOME/apps/btec_t_input_apigw_op/default/inputs.conf disabled = false
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/apps/btec_t_input_apigw_op/default/inputs.conf index = rt_apigw
SPLUNK_HOME/apps/btec_t_input_apigw_op/default/inputs.conf sourcetype = syslog
SPLUNK_HOME/apps/btec_p_input_excalibur/default/inputs.conf [udp://10715]
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/apps/btec_p_input_excalibur/default/inputs.conf acceptFrom = 10.45.26.137,10.45.26.138,10.45.26.139,10.45.26.140,10.45.9.15,10.45.52.227,10.45.52.228
SPLUNK_HOME/apps/btec_p_input_excalibur/default/inputs.conf disabled = false
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/apps/btec_p_input_excalibur/default/inputs.conf index = rt_excalibur
SPLUNK_HOME/apps/btec_p_input_excalibur/default/inputs.conf sourcetype = excalibur_LB

0 Karma
Reply

SinghK
Builder
‎01-28-2022 01:17 AM

you have duplicate inputs configured on same port

SPLUNK_HOME/apps/splunk_UF_HF_custom_ssl/local/inputs.conf [splunktcp-ssl://42000]
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/system/default/inputs.conf index = default
SPLUNK_HOME/apps/splunk_UF_HF_custom_ssl/local/inputs.conf [splunktcp-ssl://9997]
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/system/default/inputs.conf index = default
SPLUNK_HOME/apps/launcher/local/inputs.conf [splunktcp://42000]
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/apps/launcher/local/inputs.conf connection_host = ip
SPLUNK_HOME/apps/launcher/local/inputs.conf disabled = 0
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/system/default/inputs.conf index = default
SPLUNK_HOME/apps/launcher/local/inputs.conf [splunktcp://9997]
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/apps/launcher/local/inputs.conf connection_host = ip
SPLUNK_HOME/apps/launcher/local/inputs.conf disabled = 0
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/system/default/inputs.conf index = default
SPLUNK_HOME/system/local/inputs.conf [splunktcp://:9997]
SPLUNK_HOME/system/default/inputs.conf _rcvbuf = 1572864
SPLUNK_HOME/system/local/inputs.conf disabled = 0
SPLUNK_HOME/system/default/inputs.conf host = $decideOnStartup
SPLUNK_HOME/system/local/inputs.conf index = test
SPLUNK_HOME/apps/splunk_secure_gateway/default/inputs.conf

do you have ssl enabled in your environment ? as i can see you have some ssl inputs configured.

if the answer to this is yes have created ssl certs for new hf's ? 

Tags (1)
0 Karma
Reply

anil8
Loves-to-Learn Everything
‎01-28-2022 01:42 AM

Yes...I have created the SSL certificate.

0 Karma
Reply

SinghK
Builder
‎01-28-2022 01:53 AM

Please paste the outputs.conf from uf

0 Karma
Reply

anil8
Loves-to-Learn Everything
‎01-28-2022 02:02 AM

Hi @SinghK ,

 

Please find below output.conf from UF

[tcpout]
defaultGroup = splunkssl
useACK = true

 

[tcpout:splunkssl]
server = XXXX.XX.XX.com:9997,XXXX.XX.XX.com:9997,XXXX.XX.XX.com:9997,XXXX.XX.XX.com:9997,XXXX.XX.XX.com:9997
sslCertPath = /opt/product/splunk/splunkforwarder_ee/etc/apps/ee_splunk_forwarder_certs/auth/Forwarder.pem
sslPassword = ee_splunk_iif
sslRootCAPath = /opt/product/splunk/splunkforwarder_ee/etc/apps/ee_splunk_forwarder_certs/auth/UFHFCACertificate.pem
sslVerifyServerCert = false
useSSL=true

0 Karma
Reply

isoutamo
SplunkTrust
SplunkTrust
‎01-28-2022 01:16 AM

It seems that you have tried configure both splunktcp and splunktcp-ssl on port 9997. Based on your app names etc. I suppose that splunktcp:9997 is winning listener. Then you probably try to send events from UF by splunktcp-ssl (port 9997 or 42000) and as 9997 is working with SSL it didn't accept your connection.

Please check from UF side which outputs.conf is in use and in which protocol and port it try to use.

0 Karma
Reply

anil8
Loves-to-Learn Everything
‎01-28-2022 02:12 AM

Hi @isoutamo ,

 

Can you please advice what change should be done in output.conf in UF.

[tcpout]
defaultGroup = splunkssl
useACK = true

 

[tcpout:splunkssl]
server = XXXX.XX.XX.com:9997,XXXX.XX.XX.com:9997,XXXX.XX.XX.com:9997,XXXX.XX.XX.com:9997,XXXX.XX.XX.com:9997
sslCertPath = /opt/product/splunk/splunkforwarder_ee/etc/apps/ee_splunk_forwarder_certs/auth/Forwarder.pem
sslPassword = ee_splunk_iif
sslRootCAPath = /opt/product/splunk/splunkforwarder_ee/etc/apps/ee_splunk_forwarder_certs/auth/UFHFCACertificate.pem
sslVerifyServerCert = false
useSSL=true

0 Karma
Reply

SinghK
Builder
‎01-28-2022 03:54 AM

[tcpout]
defaultGroup=splunkssl

[tcpout:ssl]
autoLBFrequency=40
server=<your servers>
useACK=true
indexandforward=false

 

Thats all you need.

documentation on outputs.conf

https://docs.splunk.com/Documentation/Splunk/8.2.4/Admin/Outputsconf

0 Karma
Reply
Get Updates on the Splunk Community!

Developer Spotlight with Paul Stout

Welcome to our very first developer spotlight release series where we'll feature some awesome Splunk ...

State of Splunk Careers 2024: Maximizing Career Outcomes and the Continued Value of ...

For the past four years, Splunk has partnered with Enterprise Strategy Group to conduct a survey that gauges ...

Data-Driven Success: Splunk & Financial Services

Splunk streamlines the process of extracting insights from large volumes of data. In this fast-paced world, ...