Logs WIndows to Splunk Cloud

[adrifesa95]

Hello,

I have a problem because I can't see the windows logs in splunk cloud.

My architecture is as follows: UF->HF->Splunk cloud

I get the logs on the HF because I see them by doing packet inspection with tcpdump. So I have 9997 open, but these are not being forwarded to the cloud.

These are my inputs.conf

/opt/splunk/etc/apps/Splunk_TA_windows/local/

###### OS Logs ######
[WinEventLog://Application]
disabled = 0
index=mx_windows
start_from = oldest
current_only = 0
checkpointInterval = 5
renderXml=true

[WinEventLog://Security]
disabled = 0
index=mx_windows
start_from = oldest
current_only = 0
evt_resolve_ad_obj = 1
checkpointInterval = 5
blacklist1 = EventCode="4662" Message="Object Type:(?!\s*groupPolicyContainer)"
blacklist2 = EventCode="566" Message="Object Type:(?!\s*groupPolicyContainer)"
renderXml=true

[WinEventLog://System]
disabled = 0
index=mx_windows
start_from = oldest
current_only = 0
checkpointInterval = 5
renderXml=true

###### Forwarded WinEventLogs (WEF) ######
[WinEventLog://ForwardedEvents]
disabled = 0
start_from = oldest
current_only = 0
checkpointInterval = 5
## The addon supports only XML format for the collection of WinEventLogs using WEF, hence do not change the below renderXml parameter to false.
renderXml=true
host=WinEventLogForwardHost
index=mx_windows





/opt/splunk/etc/system/local/inputs.conf

[splunktcp://9997]
index=mx_windows
disabled = 0

[WinEventLog://ForwardedEvents]
index=mx_windows
disabled = 0

[adrifesa95]

I have tried what @gcusello said, but it did not work.

Now I suspect that maybe they are not sending anything to the HF, because checking the connections with tcpdump to port 9997, I have seen that they are only with the splunk tenant. Can they use the same port to receive and make connections to the indexers (9997) and to receive logs from the UF (9997)? Do you recommend any other test?

[gcusello]

Hi @adrifesa95 ,

it shouldn't be a probem: on your HF, you can receive logs on port 9997 and send logs to Splunk Cloud on 9997 port.

Check if from the UFs you can reach the HF (using e.g. telnet).

Ciao.

Giuseppe

[deepakc]

You Windows Hosts should have an output.conf that sends to the HF only, if this is how you want your data flow architecture. (You don’t need the 100_tenant_splunkcloud installed on the Windows UF's Unless you want to send from them directly to Splunk cloud, this is also a viable solution.  

I'm starting to think you may have the 100_tenant_splunkcloud and configured outputs to the HF on the Windows hosts, you need to have one or the other for this setup.

Run the btool outputs command on the Windows UF let’s see what that shows?

/opt/splunkforwarder/bin/splunk btool outputs list –debug

[deepakc]

I should have said change to windows path as the command I gave is for Linux 

[adrifesa95]

/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf [MonitorNoHandle://$WINDIR\System32\Dns\dns.log]
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf disabled = 1
host = $decideOnStartup
index = default
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf sourcetype = MSAD:NT6:DNS
/opt/splunk/etc/system/default/inputs.conf [SSL]
/opt/splunk/etc/system/default/inputs.conf _rcvbuf = 1572864
/opt/splunk/etc/system/default/inputs.conf allowSslRenegotiation = true
/opt/splunk/etc/system/default/inputs.conf certLogMaxCacheEntries = 10000
/opt/splunk/etc/system/default/inputs.conf certLogRepeatFrequency = 1d
/opt/splunk/etc/system/default/inputs.conf cipherSuite = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
/opt/splunk/etc/system/default/inputs.conf ecdhCurves = prime256v1, secp384r1, secp521r1
/opt/splunk/etc/system/default/inputs.conf host = $decideOnStartup
/opt/splunk/etc/system/default/inputs.conf index = default
/opt/splunk/etc/system/default/inputs.conf logCertificateData = true
/opt/splunk/etc/system/default/inputs.conf sslQuietShutdown = false
/opt/splunk/etc/system/default/inputs.conf sslVersions = tls1.2
/opt/splunk/etc/apps/Splunk_TA_windows/local/inputs.conf [WinEventLog://Application]
/opt/splunk/etc/apps/Splunk_TA_windows/local/inputs.conf checkpointInterval = 5
/opt/splunk/etc/apps/Splunk_TA_windows/local/inputs.conf current_only = 0
/opt/splunk/etc/apps/Splunk_TA_windows/local/inputs.conf disabled = 0
host = $decideOnStartup
/opt/splunk/etc/apps/Splunk_TA_windows/local/inputs.conf index = mx_windows
/opt/splunk/etc/apps/Splunk_TA_windows/local/inputs.conf renderXml = true
/opt/splunk/etc/apps/Splunk_TA_windows/local/inputs.conf start_from = oldest
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf [WinEventLog://DFS Replication]
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf disabled = 1
host = $decideOnStartup
index = default
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf renderXml = true
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf [WinEventLog://DNS Server]
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf disabled = 1
host = $decideOnStartup
index = default
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf renderXml = true
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf [WinEventLog://Directory Service]
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf disabled = 1
host = $decideOnStartup
index = default
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf renderXml = true
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf [WinEventLog://File Replication Service]
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf disabled = 1
host = $decideOnStartup
index = default
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf renderXml = true
/opt/splunk/etc/apps/Splunk_TA_windows/local/inputs.conf [WinEventLog://ForwardedEvents]
/opt/splunk/etc/apps/Splunk_TA_windows/local/inputs.conf checkpointInterval = 5
/opt/splunk/etc/apps/Splunk_TA_windows/local/inputs.conf current_only = 0
/opt/splunk/etc/apps/Splunk_TA_windows/local/inputs.conf disabled = 0
/opt/splunk/etc/apps/Splunk_TA_windows/local/inputs.conf host = WinEventLogForwardHost
/opt/splunk/etc/apps/Splunk_TA_windows/local/inputs.conf index = mx_windows
/opt/splunk/etc/apps/Splunk_TA_windows/local/inputs.conf renderXml = true
/opt/splunk/etc/apps/Splunk_TA_windows/local/inputs.conf start_from = oldest
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf [WinEventLog://Key Management Service]
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf disabled = 1
host = $decideOnStartup
index = default
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf renderXml = true
/opt/splunk/etc/apps/Splunk_TA_windows/local/inputs.conf [WinEventLog://Security]
/opt/splunk/etc/apps/Splunk_TA_windows/local/inputs.conf blacklist1 = EventCode="4662" Message="Object Type:(?!\s*groupPolicyContainer)"
/opt/splunk/etc/apps/Splunk_TA_windows/local/inputs.conf blacklist2 = EventCode="566" Message="Object Type:(?!\s*groupPolicyContainer)"
/opt/splunk/etc/apps/Splunk_TA_windows/local/inputs.conf checkpointInterval = 5
/opt/splunk/etc/apps/Splunk_TA_windows/local/inputs.conf current_only = 0
/opt/splunk/etc/apps/Splunk_TA_windows/local/inputs.conf disabled = 0
/opt/splunk/etc/apps/Splunk_TA_windows/local/inputs.conf evt_resolve_ad_obj = 1
host = $decideOnStartup
/opt/splunk/etc/apps/Splunk_TA_windows/local/inputs.conf index = mx_windows
/opt/splunk/etc/apps/Splunk_TA_windows/local/inputs.conf renderXml = true
/opt/splunk/etc/apps/Splunk_TA_windows/local/inputs.conf start_from = oldest
/opt/splunk/etc/apps/Splunk_TA_windows/local/inputs.conf [WinEventLog://System]
/opt/splunk/etc/apps/Splunk_TA_windows/local/inputs.conf checkpointInterval = 5
/opt/splunk/etc/apps/Splunk_TA_windows/local/inputs.conf current_only = 0
/opt/splunk/etc/apps/Splunk_TA_windows/local/inputs.conf disabled = 0
host = $decideOnStartup
/opt/splunk/etc/apps/Splunk_TA_windows/local/inputs.conf index = mx_windows
/opt/splunk/etc/apps/Splunk_TA_windows/local/inputs.conf renderXml = true
/opt/splunk/etc/apps/Splunk_TA_windows/local/inputs.conf start_from = oldest
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf [WinHostMon://Computer]
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf disabled = 1
host = $decideOnStartup
index = default
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf interval = 600
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf type = Computer
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf [WinHostMon://Disk]
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf disabled = 1
host = $decideOnStartup
index = default
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf interval = 600
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf type = Disk
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf [WinHostMon://Driver]
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf disabled = 1
host = $decideOnStartup
index = default
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf interval = 600
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf type = Driver
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf [WinHostMon://NetworkAdapter]
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf disabled = 1
host = $decideOnStartup
index = default
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf interval = 600
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf type = NetworkAdapter
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf [WinHostMon://OperatingSystem]
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf disabled = 1
host = $decideOnStartup
index = default
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf interval = 600
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf type = OperatingSystem
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf [WinHostMon://Process]
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf disabled = 1
host = $decideOnStartup
index = default
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf interval = 600
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf type = Process
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf [WinHostMon://Processor]
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf disabled = 1
host = $decideOnStartup
index = default
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf interval = 600
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf type = Processor
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf [WinHostMon://Roles]
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf disabled = 1
host = $decideOnStartup
index = default
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf interval = 600
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf type = Roles
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf [WinHostMon://Service]
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf disabled = 1
host = $decideOnStartup
index = default
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf interval = 600
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf type = Service
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf [WinNetMon://inbound]
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf direction = inbound
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf disabled = 1
host = $decideOnStartup
index = default
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf [WinNetMon://outbound]
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf direction = outbound
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf disabled = 1
host = $decideOnStartup
index = default
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf [WinPrintMon://driver]
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf baseline = 1
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf disabled = 1
host = $decideOnStartup
index = default
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf interval = 600
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf type = driver
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf [WinPrintMon://port]
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf baseline = 1
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf disabled = 1
host = $decideOnStartup
index = default
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf interval = 600
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf type = port
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf [WinPrintMon://printer]
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf baseline = 1
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf disabled = 1
host = $decideOnStartup
index = default
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf interval = 600
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf type = printer
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf [WinRegMon://default]
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf disabled = 1
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf hive = .*
host = $decideOnStartup
index = default
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf proc = .*
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf type = rename|set|delete|create
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf [WinRegMon://hkcu_run]
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf disabled = 1
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf hive = \\REGISTRY\\USER\\.*\\Software\\Microsoft\\Windows\\CurrentVersion\\Run\\.*
host = $decideOnStartup
index = default
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf proc = .*
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf type = set|create|delete|rename
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf [WinRegMon://hklm_run]
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf disabled = 1
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf hive = \\REGISTRY\\MACHINE\\SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Run\\.*
host = $decideOnStartup
index = default
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf proc = .*
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf type = set|create|delete|rename
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf [admon://default]
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf disabled = 1
host = $decideOnStartup
index = default
/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf monitorSubtree = 1
/opt/splunk/etc/system/default/inputs.conf [batch:///opt/splunk/var/run/splunk/search_telemetry/*search_telemetry.json]
/opt/splunk/etc/system/default/inputs.conf _rcvbuf = 1572864
/opt/splunk/etc/system/default/inputs.conf crcSalt = <SOURCE>
/opt/splunk/etc/system/default/inputs.conf host = $decideOnStartup
/opt/splunk/etc/system/default/inputs.conf index = _introspection
/opt/splunk/etc/system/default/inputs.conf log_on_completion = 0
/opt/splunk/etc/system/default/inputs.conf move_policy = sinkhole
/opt/splunk/etc/system/default/inputs.conf sourcetype = search_telemetry
/opt/splunk/etc/system/default/inputs.conf [batch:///opt/splunk/var/spool/splunk]
/opt/splunk/etc/system/default/inputs.conf _rcvbuf = 1572864
/opt/splunk/etc/system/default/inputs.conf crcSalt = <SOURCE>
/opt/splunk/etc/system/default/inputs.conf host = $decideOnStartup
/opt/splunk/etc/system/default/inputs.conf index = default
/opt/splunk/etc/system/default/inputs.conf move_policy = sinkhole
/opt/splunk/etc/system/default/inputs.conf [batch:///opt/splunk/var/spool/splunk/...stash_hec]
/opt/splunk/etc/system/default/inputs.conf _rcvbuf = 1572864
/opt/splunk/etc/system/default/inputs.conf crcSalt = <SOURCE>
/opt/splunk/etc/system/default/inputs.conf host = $decideOnStartup
/opt/splunk/etc/system/default/inputs.conf index = default
/opt/splunk/etc/system/default/inputs.conf move_policy = sinkhole
/opt/splunk/etc/system/default/inputs.conf sourcetype = stash_hec
/opt/splunk/etc/system/default/inputs.conf [batch:///opt/splunk/var/spool/splunk/...stash_new]
/opt/splunk/etc/system/default/inputs.conf _rcvbuf = 1572864

and many lines below

/opt/splunk/etc/apps/launcher/local/inputs.conf [splunktcp://9997]

[adrifesa95]

/opt/splunk/etc/system/default/outputs.conf [rfs]
/opt/splunk/etc/system/default/outputs.conf batchSizeThresholdKB = 131072
/opt/splunk/etc/system/default/outputs.conf batchTimeout = 30
/opt/splunk/etc/system/default/outputs.conf compression = zstd
/opt/splunk/etc/system/default/outputs.conf compressionLevel = 3
/opt/splunk/etc/system/default/outputs.conf dropEventsOnUploadError = false
/opt/splunk/etc/system/default/outputs.conf format = json
/opt/splunk/etc/system/default/outputs.conf format.json.index_time_fields = true
/opt/splunk/etc/system/default/outputs.conf format.ndjson.index_time_fields = true
/opt/splunk/etc/system/default/outputs.conf partitionBy = legacy
/opt/splunk/etc/system/default/outputs.conf [syslog]
/opt/splunk/etc/system/default/outputs.conf maxEventSize = 1024
/opt/splunk/etc/system/default/outputs.conf priority = <13>
/opt/splunk/etc/system/default/outputs.conf type = udp
/opt/splunk/etc/apps/100_tenant_splunkcloud/local/outputs.conf [tcpout]
/opt/splunk/etc/system/default/outputs.conf ackTimeoutOnShutdown = 30
/opt/splunk/etc/system/default/outputs.conf autoLBFrequency = 30
/opt/splunk/etc/apps/100_tenant_splunkcloud/local/outputs.conf autoLBFrequencyIntervalOnGroupFailure = -1
/opt/splunk/etc/system/default/outputs.conf autoLBVolume = 0
/opt/splunk/etc/system/default/outputs.conf blockOnCloning = true
/opt/splunk/etc/system/default/outputs.conf blockWarnThreshold = 100
/opt/splunk/etc/apps/100_tenant_splunkcloud/local/outputs.conf channelReapInterval = 60000
/opt/splunk/etc/apps/100_tenant_splunkcloud/local/outputs.conf channelReapLowater = 10
/opt/splunk/etc/apps/100_tenant_splunkcloud/local/outputs.conf channelTTL = 300000
/opt/splunk/etc/system/default/outputs.conf cipherSuite = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:AES256-GCM-SHA384:AES128-GCM-SHA256:AES128-SHA256:ECDH-ECDSA-AES256-GCM-SHA384:ECDH-ECDSA-AES128-GCM-SHA256:ECDH-ECDSA-AES256-SHA384:ECDH-ECDSA-AES128-SHA256
/opt/splunk/etc/system/default/outputs.conf compressed = false
/opt/splunk/etc/system/default/outputs.conf connectionTTL = 0
/opt/splunk/etc/system/default/outputs.conf connectionTimeout = 20
/opt/splunk/etc/apps/100_tenant_splunkcloud/local/outputs.conf connectionsPerTarget = 0
/opt/splunk/etc/system/local/outputs.conf defaultGroup = splunkcloud_20231028_9aaa4b04216cd9a0a4dc1eb274307fd1
/opt/splunk/etc/system/default/outputs.conf disabled = false
/opt/splunk/etc/apps/100_tenant_splunkcloud/local/outputs.conf dnsResolutionInterval = 300
/opt/splunk/etc/system/default/outputs.conf dropClonedEventsOnQueueFull = 5
/opt/splunk/etc/system/default/outputs.conf dropEventsOnQueueFull = -1
/opt/splunk/etc/system/default/outputs.conf ecdhCurves = prime256v1, secp384r1, secp521r1
/opt/splunk/etc/system/default/outputs.conf enableOldS2SProtocol = false
/opt/splunk/etc/system/default/outputs.conf forceTimebasedAutoLB = false
/opt/splunk/etc/apps/SplunkForwarder/default/outputs.conf forwardedindex.0.whitelist = .*
/opt/splunk/etc/apps/SplunkForwarder/default/outputs.conf forwardedindex.1.blacklist = _.*
/opt/splunk/etc/apps/SplunkDeploymentServerConfig/default/outputs.conf forwardedindex.2.whitelist = (_audit|_internal|_introspection|_telemetry|_metrics|_metrics_rollup|_configtracker|_dsclient|_dsphonehome|_dsappevent)
/opt/splunk/etc/apps/SplunkForwarder/default/outputs.conf forwardedindex.filter.disable = false
/opt/splunk/etc/system/default/outputs.conf heartbeatFrequency = 30
/opt/splunk/etc/system/local/outputs.conf indexAndForward = 1
/opt/splunk/etc/system/default/outputs.conf maxConnectionsPerIndexer = 2
/opt/splunk/etc/system/default/outputs.conf maxFailuresPerInterval = 2
/opt/splunk/etc/apps/SplunkForwarder/default/outputs.conf maxQueueSize = 500KB
/opt/splunk/etc/apps/100_tenant_splunkcloud/local/outputs.conf negotiateNewProtocol = true
/opt/splunk/etc/apps/100_tenant_splunkcloud/local/outputs.conf polling_interval = 5
/opt/splunk/etc/system/default/outputs.conf readTimeout = 300
/opt/splunk/etc/system/default/outputs.conf secsInFailureInterval = 1
/opt/splunk/etc/system/default/outputs.conf sendCookedData = true
/opt/splunk/etc/apps/100_tenant_splunkcloud/local/outputs.conf socksResolveDNS = false
/opt/splunk/etc/apps/100_tenant_splunkcloud/local/outputs.conf sslPassword =
/opt/splunk/etc/system/default/outputs.conf sslQuietShutdown = false
/opt/splunk/etc/system/default/outputs.conf sslVersions = tls1.2
/opt/splunk/etc/system/default/outputs.conf tcpSendBufSz = 0
/opt/splunk/etc/system/local/outputs.conf useACK = true
/opt/splunk/etc/system/default/outputs.conf useClientSSLCompression = true
/opt/splunk/etc/system/default/outputs.conf writeTimeout = 300
/opt/splunk/etc/system/local/outputs.conf [tcpout:scs]
/opt/splunk/etc/system/local/outputs.conf autoLBFrequency = 120
/opt/splunk/etc/system/local/outputs.conf clientCert = $SPLUNK_HOME/etc/apps/100_tenant_splunkcloud/default/tenant_server.pem
/opt/splunk/etc/system/local/outputs.conf compressed = true
/opt/splunk/etc/system/local/outputs.conf disabled = 1
/opt/splunk/etc/system/local/outputs.conf server = tenant.forwarders.scs.splunk.com:9997
/opt/splunk/etc/system/local/outputs.conf sslAltNameToCheck = *.forwarders.scs.splunk.com
/opt/splunk/etc/system/local/outputs.conf sslVerifyServerCert = true
/opt/splunk/etc/system/local/outputs.conf useClientSSLCompression = false
/opt/splunk/etc/system/local/outputs.conf [tcpout:splunkcloud_]
/opt/splunk/etc/system/local/outputs.conf autoLBFrequency = 120
/opt/splunk/etc/system/local/outputs.conf clientCert = $SPLUNK_HOME/etc/apps/100_tenant_splunkcloud/default/tenant_server.pem
/opt/splunk/etc/system/local/outputs.conf compressed = false
/opt/splunk/etc/system/local/outputs.conf server = inputs1.tenant.splunkcloud.com:9997, inputs2.tenant.splunkcloud.com:9997, inputs3.tenant.splunkcloud.com:9997, inputs4.tenant.splunkcloud.com:9997, inputs5.tenant.splunkcloud.com:9997, inputs6.tenant.splunkcloud.com:9997, inputs7.tenant.splunkcloud.com:9997, inputs8.tenant.splunkcloud.com:9997, inputs9.tenant.splunkcloud.com:9997, inputs10.tenant.splunkcloud.com:9997, inputs11.tenant.splunkcloud.com:9997, inputs12.tenant.splunkcloud.com:9997, inputs13.tenant.splunkcloud.com:9997, inputs14.tenant.splunkcloud.com:9997, inputs15.tenant.splunkcloud.com:9997
/opt/splunk/etc/system/local/outputs.conf sslCommonNameToCheck = *.tenant.splunkcloud.com
/opt/splunk/etc/system/local/outputs.conf sslVerifyServerCert = true
/opt/splunk/etc/system/local/outputs.conf sslVerifyServerName = true
/opt/splunk/etc/system/local/outputs.conf useClientSSLCompression = true

[deepakc]

Well, it can be several things, network/config:

You have shown the inputs but what about the outputs?

Obviously, you will have a better understanding of your network / access / data flow details, but here's a number of area's for you to check  and investigate.

1. Have you installed the Splunk Cloud UF App Package onto the HF (splunkclouduf.spl This contains the outputs.conf  / TLS config, you download this from your Splunk cloud stack).
2. Have you allowed the HF for outbound connectivity to Splunk Cloud (Firewall changes) ?
3. After you download and install the Splunk Cloud UF App Package onto the HF, can you see the HF's _internal logs in Splunk cloud?
4. In Splunk cloud there is allow IP whitelisting feature, have you configured this for the HF to allow data to be sent to Splunk cloud?

[adrifesa95]

Hello,

I answer to both of you, I leave you my outputs.conf that as you say I downloaded it from the cloud and it points to the indexers.

[root@host ~]# cat /opt/splunk/etc/system/local/outputs.conf
[tcpout]
defaultGroup = splunkcloud_20231028_9aaa4b04216cd9a0a4dc1eb274307fd1
useACK = true
indexAndForward = 0

[tcpout:splunkcloud_20231028_9aaa4b04216cd9a0a4dc1eb274307fd1]
server = inputs1.tenant.splunkcloud.com:9997, inputs2.tenant.splunkcloud.com:9997, inputs3.tenant.splunkcloud.com:9997, inputs4.tenant.splunkcloud.com:9997, inputs5.tenant.splunkcloud.com:9997, inputs6.tenant.splunkcloud.com:9997, inputs7.tenant.splunkcloud.com:9997, inputs8.tenant.splunkcloud.com:9997, inputs9.tenant.splunkcloud.com:9997, inputs10.tenant.splunkcloud.com:9997, inputs11.tenant.splunkcloud.com:9997, inputs12.tenant.splunkcloud.com:9997, inputs13.tenant.splunkcloud.com:9997, inputs14.tenant.splunkcloud.com:9997, inputs15.tenant.splunkcloud.com:9997

But this is a problem with this source, because I have other sources that go through that HF and arrive correctly to the cloud. I have already tested that port 9997 is up, but I must be missing something else. I have created the index mx_windows on both cloud and HF. any more ideas?

[deepakc]

Sounds like order of precedence issue- These two will help in figuring out what is take the priority setting: (Some config is taking place before the other) but go by what @gcusello  is saying. 

Inputs config

/opt/splunk/bin/splunk btool inputs list --debug

 outputs config

/opt/splunk/bin/splunk btool outputs list --debug

[gcusello]

Hi @adrifesa95 ,

if your HF is forwarding other logs, te connection is ok.

so, try to remove the second stana in the inputs.conf of the HF leaving only:

[splunktcp://9997]
disabled = 0

Ciao.

Giuseppe

[gcusello]

Hi @adrifesa95,

are you receiving Splunk internal logs from the HF and UFs in Splunk Cloud?

how did you configure the outputs.conf on the HF?

and on the UFs?

Ciao.

Giuseppe