Installation

How to achieve access outbound via proxy?

andrewparkes
Loves-to-Learn

Hi,

We have deployed splunk on-prem components, heavy forwarder, syslog-ng and deployment server. 

Configured it correctly, we think, we can install the universal forwarder on an endpoint and see the endpoint in the on-prem console.

 

Now to get his information flowing out to splunk cloud, we have to go via a proxy server as it is the only way out of the environment. We've configured the servers, and proxy, to allow it to communicate with the yum repositories, so we know it can get out and connect. The issue we are now having is we don't seem able to get the data flowing out to the cloud, I've followed the below article, correctly I believe:

https://docs.splunk.com/Documentation/Splunk/9.0.2/Forwarding/ConfigureaforwardertouseaSOCKSproxy

This config should be on the heavy forwarder, is that correct?

The proxy server is configured correctly we also believe, but that could also be wrong. Anyone have any ideas/pointers/advice on why this doesn't work and how we resolve it?

Labels (1)
0 Karma

andrewparkes
Loves-to-Learn

Yes, granted its a bit vague, still trying to understand splunk and its components. Yes, aware its about SOCKS proxy, we have configured the service in the proxy server as per the symantec article and documentation. The use of the HTTP was to verify we could actually get this talking externally to something. We know its a different port, we have also configured the SOCKS requirements as per symantec and the article referenced above.

Not ruling out misconfiguration to be honest, its becoming a bit of a minefield! If i log into the onprem deployment server, i can see the endpoints we have installed the UF on. So we are getting from the endpoint to the onprem servers fine. I dont see any further errors in the deployment server. I do see them in the heavy forwarder console, as such:

andrewparkes_1-1672838906931.png

 

 

0 Karma

PickleRick
SplunkTrust
SplunkTrust

Your warnings tell you that indeed the forwarding is not happening. To find out why you need to look into /opt/splunk/var/log/splunk/splunkd.log

The Deployment Server<->Deployment Clients (UFs) communication is something completely different than forwarding events to outputs so in case of diagnosing the outputs it won't help.

Of course you should also check logs of your proxy for any connection attempts from your Splunk components.

0 Karma

PickleRick
SplunkTrust
SplunkTrust

You are very vague about how your environment is configured an what is actually happening.

You say your proxy is configured correctly but we have no way to know that or even why you believe that. Did you verify it in any way?

We don't have your forwarder's configuration and we don't know what's reported in logs.

So short of responding "something's wrong, most probably misconfigured", we're not able to say anything.

Most importantly - you are aware that the article talks about SOCKS proxy, not a HTTP proxy, right?

0 Karma

andrewparkes
Loves-to-Learn

andrewparkes_0-1672930537520.png

This is what i'm seeing in the splunkd log. any ideas how to resolve it would be appreciated. 

0 Karma

PickleRick
SplunkTrust
SplunkTrust

This log shows that the server doesn't know the destination server - can't resolve its DNS name.

Do

splunk btool ouputs list

replace hostnames/IPs with some placeholders if that's sensitive data for you and post it here.

0 Karma

andrewparkes
Loves-to-Learn

Output as below:

[root@ts-test-splnkhvf local]# /opt/splunk/bin/splunk btool outputs list
[rfs]
batchSizeThresholdKB = 2048
batchTimeout = 30
compression = zstd
compressionLevel = 3
dropEventsOnUploadError = false
partitionBy = legacy
[syslog]
maxEventSize = 1024
priority = <13>
type = udp
[tcpout]
ackTimeoutOnShutdown = 30
autoLBFrequency = 30
autoLBFrequencyIntervalOnGroupFailure = -1
autoLBVolume = 0
blockOnCloning = true
blockWarnThreshold = 100
channelReapInterval = 60000
channelReapLowater = 10
channelTTL = 300000
cipherSuite = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:AES256-GCM-SHA384:AES128-GCM-SHA256:AES128-SHA256:ECDH-ECDSA-AES256-GCM-SHA384:ECDH-ECDSA-AES128-GCM-SHA256:ECDH-ECDSA-AES256-SHA384:ECDH-ECDSA-AES128-SHA256
compressed = false
connectionTTL = 0
connectionTimeout = 20
connectionsPerTarget = 0
defaultGroup = proxy_indexers
disabled = false
dnsResolutionInterval = 300
dropClonedEventsOnQueueFull = 5
dropEventsOnQueueFull = -1
ecdhCurves = prime256v1, secp384r1, secp521r1
forceTimebasedAutoLB = false
forwardedindex.0.whitelist = .*
forwardedindex.1.blacklist = _.*
forwardedindex.2.whitelist = (_audit|_internal|_introspection|_telemetry|_metrics|_metrics_rollup|_configtracker)
forwardedindex.filter.disable = false
heartbeatFrequency = 30
indexAndForward = false
maxConnectionsPerIndexer = 2
maxFailuresPerInterval = 2
maxQueueSize = auto
negotiateNewProtocol = true
polling_interval = 5
readTimeout = 300
secsInFailureInterval = 1
sendCookedData = true
socksResolveDNS = false
sslPassword = $7$Tb/iioBRZq72fj4tWQYjPXWK2NbNXQu503TG8B9t879bR8ihr1pDwNQFY+r0MlGEqxcozldWxXAkEfmEwcqYRKXF7esksFdHzwDJPLf8eJrDffdaTuBA7+IhHPr5kyxH
sslQuietShutdown = false
sslVersions = tls1.2
tcpSendBufSz = 0
useACK = false
useClientSSLCompression = true
writeTimeout = 300
[tcpout:proxy_indexers]
server = inputs1.cubicfrankfurt.splunkcloud.com:9997, inputs2.cubicfrankfurt.splunkcloud.com:9997, inputs3.cubicfrankfurt.splunkcloud.com:9997,inputs4.cubicfrankfurt.splunkcloud.com:9997, inputs5.cubicfrankfurt.splunkcloud.com:9997, inputs6.cubicfrankfurt.splunkcloud.com:9997, inputs1.cubicfrankfurt.splunkcloud.com:9997, inputs8.cubicfrankfurt.splunkcloud.com:9997, inputs9.cubicfrankfurt.splunkcloud.com:9997, inputs10.cubicfrankfurt.splunkcloud.com:9997, inputs11.cubicfrankfurt.splunkcloud.com:9997, inputs12.cubicfrankfurt.splunkcloud.com:9997, inputs13.cubicfrankfurt.splunkcloud.com:9997, inputs14.cubicfrankfurt.splunkcloud.com:9997, inputs15.cubicfrankfurt.splunkcloud.com:9997
socksResolveDNS = false
socksServer = 10.203.254.1:1080
[tcpout:scs]
autoLBFrequency = 120
clientCert = $SPLUNK_HOME/etc/apps/100_cubicfrankfurt_splunkcloud/default/cubicfrankfurt_server.pem
compressed = true
disabled = 1
server = cubicfrankfurt.forwarders.scs.splunk.com:9997
sslAltNameToCheck = *.forwarders.scs.splunk.com
sslVerifyServerCert = true
useClientSSLCompression = false
[tcpout:splunkcloud_20220204_9aaa4b04216cd9a0a4dc1eb274307fd1]
autoLBFrequency = 120
clientCert = $SPLUNK_HOME/etc/apps/100_cubicfrankfurt_splunkcloud/default/cubicfrankfurt_server.pem
compressed = false
server = inputs1.cubicfrankfurt.splunkcloud.com:9997, inputs2.cubicfrankfurt.splunkcloud.com:9997, inputs3.cubicfrankfurt.splunkcloud.com:9997, inputs4.cubicfrankfurt.splunkcloud.com:9997, inputs5.cubicfrankfurt.splunkcloud.com:9997, inputs6.cubicfrankfurt.splunkcloud.com:9997, inputs7.cubicfrankfurt.splunkcloud.com:9997, inputs8.cubicfrankfurt.splunkcloud.com:9997, inputs9.cubicfrankfurt.splunkcloud.com:9997, inputs10.cubicfrankfurt.splunkcloud.com:9997, inputs11.cubicfrankfurt.splunkcloud.com:9997, inputs12.cubicfrankfurt.splunkcloud.com:9997, inputs13.cubicfrankfurt.splunkcloud.com:9997, inputs14.cubicfrankfurt.splunkcloud.com:9997, inputs15.cubicfrankfurt.splunkcloud.com:9997
sslCommonNameToCheck = *.cubicfrankfurt.splunkcloud.com
sslVerifyServerCert = true
useClientSSLCompression = true

0 Karma

PickleRick
SplunkTrust
SplunkTrust

Hmm... Theoreticaly - it does indeed look good. (I'd run "splunk btool check" just to be sure that there are no mistyped parameters anywhere).

A stupid question - you did restart this instance after setting this config?

 

0 Karma

andrewparkes
Loves-to-Learn

output of the btool check below?

Invalid key in stanza?

andrewparkes_0-1673282721853.png

 

0 Karma

PickleRick
SplunkTrust
SplunkTrust

This particular invalid key message is probably due to some legacy stuff left after upgrades. I think I've seen it somewhere already.

And it definitely doesn't have anything to do with your SOCKS proxy.

 

0 Karma

andrewparkes
Loves-to-Learn

Upgrades as in product updates in the installers? As this was built from new, hasn't been upgraded from any previous version as its a new deployment 

0 Karma

PickleRick
SplunkTrust
SplunkTrust

Interesting. I'd expect it as something that lingered on from some earlier Splunk version or something like that. But anyway, it's a completely different story and should not have anything to do with your problem at hand. Try to verify whether there is any attempt of contacting your SOCKS proxy at all (tcpdump, firewall logs, whatever).

As I said, SOCKS functionality isn't used very often so I suppose you might hit some bug. But it's good to verify first before creating a support case.

0 Karma

andrewparkes
Loves-to-Learn

I'm not sure whether that's a good or bad thing then, now literally nowhere to look. I was hoping id added a parameter wrong, or sausage fingered something... **bleep**!

 

Yup been rebooted numerous times, all without any joy

0 Karma

PickleRick
SplunkTrust
SplunkTrust

I'd try troubleshooting from the ground up.

Just set up a config with one of your servers instead of the whole bunch, provide its IP instead of DNS name and check if it works.

Use tcpdump/wireshark to see if the logs aren't misleading (if the Splunk instance itself is trying to resolve DNS name and it's not just some confusing message that in reality comes from the SOCKS server failing to resolve DNS). Verify if the connection to SOCKS server is happening at all. If so, check your SOCKS server's log.

SOCKS is not a very often used feature so maybe you hit some bug or maybe the logging of this module is confusing.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...