How to achieve access outbound via proxy?

[andrewparkes]

Hi,

We have deployed splunk on-prem components, heavy forwarder, syslog-ng and deployment server. 

Configured it correctly, we think, we can install the universal forwarder on an endpoint and see the endpoint in the on-prem console.

Now to get his information flowing out to splunk cloud, we have to go via a proxy server as it is the only way out of the environment. We've configured the servers, and proxy, to allow it to communicate with the yum repositories, so we know it can get out and connect. The issue we are now having is we don't seem able to get the data flowing out to the cloud, I've followed the below article, correctly I believe:

https://docs.splunk.com/Documentation/Splunk/9.0.2/Forwarding/ConfigureaforwardertouseaSOCKSproxy

This config should be on the heavy forwarder, is that correct?

The proxy server is configured correctly we also believe, but that could also be wrong. Anyone have any ideas/pointers/advice on why this doesn't work and how we resolve it?

[andrewparkes]

Yes, granted its a bit vague, still trying to understand splunk and its components. Yes, aware its about SOCKS proxy, we have configured the service in the proxy server as per the symantec article and documentation. The use of the HTTP was to verify we could actually get this talking externally to something. We know its a different port, we have also configured the SOCKS requirements as per symantec and the article referenced above.

Not ruling out misconfiguration to be honest, its becoming a bit of a minefield! If i log into the onprem deployment server, i can see the endpoints we have installed the UF on. So we are getting from the endpoint to the onprem servers fine. I dont see any further errors in the deployment server. I do see them in the heavy forwarder console, as such:

[PickleRick]

Your warnings tell you that indeed the forwarding is not happening. To find out why you need to look into /opt/splunk/var/log/splunk/splunkd.log

The Deployment Server<->Deployment Clients (UFs) communication is something completely different than forwarding events to outputs so in case of diagnosing the outputs it won't help.

Of course you should also check logs of your proxy for any connection attempts from your Splunk components.

[PickleRick]

You are very vague about how your environment is configured an what is actually happening.

You say your proxy is configured correctly but we have no way to know that or even why you believe that. Did you verify it in any way?

We don't have your forwarder's configuration and we don't know what's reported in logs.

So short of responding "something's wrong, most probably misconfigured", we're not able to say anything.

Most importantly - you are aware that the article talks about SOCKS proxy, not a HTTP proxy, right?

[andrewparkes]

This is what i'm seeing in the splunkd log. any ideas how to resolve it would be appreciated. 

[PickleRick]

This log shows that the server doesn't know the destination server - can't resolve its DNS name.

Do

splunk btool ouputs list

replace hostnames/IPs with some placeholders if that's sensitive data for you and post it here.

[andrewparkes]

Output as below:

[root@ts-test-splnkhvf local]# /opt/splunk/bin/splunk btool outputs list
[rfs]
batchSizeThresholdKB = 2048
batchTimeout = 30
compression = zstd
compressionLevel = 3
dropEventsOnUploadError = false
partitionBy = legacy
[syslog]
maxEventSize = 1024
priority = <13>
type = udp
[tcpout]
ackTimeoutOnShutdown = 30
autoLBFrequency = 30
autoLBFrequencyIntervalOnGroupFailure = -1
autoLBVolume = 0
blockOnCloning = true
blockWarnThreshold = 100
channelReapInterval = 60000
channelReapLowater = 10
channelTTL = 300000
cipherSuite = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:AES256-GCM-SHA384:AES128-GCM-SHA256:AES128-SHA256:ECDH-ECDSA-AES256-GCM-SHA384:ECDH-ECDSA-AES128-GCM-SHA256:ECDH-ECDSA-AES256-SHA384:ECDH-ECDSA-AES128-SHA256
compressed = false
connectionTTL = 0
connectionTimeout = 20
connectionsPerTarget = 0
defaultGroup = proxy_indexers
disabled = false
dnsResolutionInterval = 300
dropClonedEventsOnQueueFull = 5
dropEventsOnQueueFull = -1
ecdhCurves = prime256v1, secp384r1, secp521r1
forceTimebasedAutoLB = false
forwardedindex.0.whitelist = .*
forwardedindex.1.blacklist = _.*
forwardedindex.2.whitelist = (_audit|_internal|_introspection|_telemetry|_metrics|_metrics_rollup|_configtracker)
forwardedindex.filter.disable = false
heartbeatFrequency = 30
indexAndForward = false
maxConnectionsPerIndexer = 2
maxFailuresPerInterval = 2
maxQueueSize = auto
negotiateNewProtocol = true
polling_interval = 5
readTimeout = 300
secsInFailureInterval = 1
sendCookedData = true
socksResolveDNS = false
sslPassword = $7$Tb/iioBRZq72fj4tWQYjPXWK2NbNXQu503TG8B9t879bR8ihr1pDwNQFY+r0MlGEqxcozldWxXAkEfmEwcqYRKXF7esksFdHzwDJPLf8eJrDffdaTuBA7+IhHPr5kyxH
sslQuietShutdown = false
sslVersions = tls1.2
tcpSendBufSz = 0
useACK = false
useClientSSLCompression = true
writeTimeout = 300
[tcpout:proxy_indexers]
server = inputs1.cubicfrankfurt.splunkcloud.com:9997, inputs2.cubicfrankfurt.splunkcloud.com:9997, inputs3.cubicfrankfurt.splunkcloud.com:9997,inputs4.cubicfrankfurt.splunkcloud.com:9997, inputs5.cubicfrankfurt.splunkcloud.com:9997, inputs6.cubicfrankfurt.splunkcloud.com:9997, inputs1.cubicfrankfurt.splunkcloud.com:9997, inputs8.cubicfrankfurt.splunkcloud.com:9997, inputs9.cubicfrankfurt.splunkcloud.com:9997, inputs10.cubicfrankfurt.splunkcloud.com:9997, inputs11.cubicfrankfurt.splunkcloud.com:9997, inputs12.cubicfrankfurt.splunkcloud.com:9997, inputs13.cubicfrankfurt.splunkcloud.com:9997, inputs14.cubicfrankfurt.splunkcloud.com:9997, inputs15.cubicfrankfurt.splunkcloud.com:9997
socksResolveDNS = false
socksServer = 10.203.254.1:1080
[tcpout:scs]
autoLBFrequency = 120
clientCert = $SPLUNK_HOME/etc/apps/100_cubicfrankfurt_splunkcloud/default/cubicfrankfurt_server.pem
compressed = true
disabled = 1
server = cubicfrankfurt.forwarders.scs.splunk.com:9997
sslAltNameToCheck = *.forwarders.scs.splunk.com
sslVerifyServerCert = true
useClientSSLCompression = false
[tcpout:splunkcloud_20220204_9aaa4b04216cd9a0a4dc1eb274307fd1]
autoLBFrequency = 120
clientCert = $SPLUNK_HOME/etc/apps/100_cubicfrankfurt_splunkcloud/default/cubicfrankfurt_server.pem
compressed = false
server = inputs1.cubicfrankfurt.splunkcloud.com:9997, inputs2.cubicfrankfurt.splunkcloud.com:9997, inputs3.cubicfrankfurt.splunkcloud.com:9997, inputs4.cubicfrankfurt.splunkcloud.com:9997, inputs5.cubicfrankfurt.splunkcloud.com:9997, inputs6.cubicfrankfurt.splunkcloud.com:9997, inputs7.cubicfrankfurt.splunkcloud.com:9997, inputs8.cubicfrankfurt.splunkcloud.com:9997, inputs9.cubicfrankfurt.splunkcloud.com:9997, inputs10.cubicfrankfurt.splunkcloud.com:9997, inputs11.cubicfrankfurt.splunkcloud.com:9997, inputs12.cubicfrankfurt.splunkcloud.com:9997, inputs13.cubicfrankfurt.splunkcloud.com:9997, inputs14.cubicfrankfurt.splunkcloud.com:9997, inputs15.cubicfrankfurt.splunkcloud.com:9997
sslCommonNameToCheck = *.cubicfrankfurt.splunkcloud.com
sslVerifyServerCert = true
useClientSSLCompression = true

[PickleRick]

Hmm... Theoreticaly - it does indeed look good. (I'd run "splunk btool check" just to be sure that there are no mistyped parameters anywhere).

A stupid question - you did restart this instance after setting this config?

[andrewparkes]

output of the btool check below?

Invalid key in stanza?

[PickleRick]

This particular invalid key message is probably due to some legacy stuff left after upgrades. I think I've seen it somewhere already.

And it definitely doesn't have anything to do with your SOCKS proxy.

[andrewparkes]

Upgrades as in product updates in the installers? As this was built from new, hasn't been upgraded from any previous version as its a new deployment 

[PickleRick]

Interesting. I'd expect it as something that lingered on from some earlier Splunk version or something like that. But anyway, it's a completely different story and should not have anything to do with your problem at hand. Try to verify whether there is any attempt of contacting your SOCKS proxy at all (tcpdump, firewall logs, whatever).

As I said, SOCKS functionality isn't used very often so I suppose you might hit some bug. But it's good to verify first before creating a support case.

[andrewparkes]

I'm not sure whether that's a good or bad thing then, now literally nowhere to look. I was hoping id added a parameter wrong, or sausage fingered something... **bleep**!

Yup been rebooted numerous times, all without any joy

[PickleRick]

I'd try troubleshooting from the ground up.

Just set up a config with one of your servers instead of the whole bunch, provide its IP instead of DNS name and check if it works.

Use tcpdump/wireshark to see if the logs aren't misleading (if the Splunk instance itself is trying to resolve DNS name and it's not just some confusing message that in reality comes from the SOCKS server failing to resolve DNS). Verify if the connection to SOCKS server is happening at all. If so, check your SOCKS server's log.

SOCKS is not a very often used feature so maybe you hit some bug or maybe the logging of this module is confusing.