splunk tcp token - how to manage and mixed setup possibilities?

[vessev]

Hi fellow splunkers,

maybe my question was not good enough.
It would be a sufficient answer if someone could provide me a few links to read about splunktcp tokens.

At the moment i only have:
https://docs.splunk.com/Documentation/Forwarder/7.3.5/Forwarder/Controlforwarderaccess

Thanks and Best regards,

vess

---

Hi all,

i need authentication enabled for my forwarders/indexers on the listening tcp 9997 port.
This is important for us cause we want to open this port on a DMZ intermediate forwarder (universal forwarder).
The DMZ Intermediate Forwarder sends the data through a firewall to my indexer in the intranet.
If searched the splunk doku and found only one document:
https://docs.splunk.com/Documentation/Forwarder/7.3.5/Forwarder/Controlforwarderaccess

(In this doc is a typo "Enable a token" -> in the command change 'tok1' to 'my_token' )

Here i have a few questions:

1. How can i see all existing tokens on my indexer/forwarder? From the documentation i can only (create, enable, disable and delete)
2. Can i manage tokens on my clients (forwarders) via a deployment server?
3. On my indexer: After i create a token (which is directly enabled) all other incoming splunktcp traffic is blocked. Can i activate tokens only for a specific input? Like a separate input on [splunktcp://9998] - traffic on tcp 9997 should work without tokens.

This is what i get after creating a token (which is directly active by the way):

`04-17-2020 14:59:52.871 +0200 ERROR TcpInputProc - Error encountered for connection from src=10.x.x.x:51116. Local side shutting down
host = testforwarder source = /opt/splunk/var/log/splunk/splunkd.log sourcetype = splunkd

`

Thanks all,

best regards Michele

[vessev · Accepted solution]

Hello fellow splunkers,

now i will share you all my research - and my own working answers.

1. Problem: How to view existing splunktcptokens
1. Solution:

You can add all settings via curl - like explained from the support site.
To see which tokens are active simply use the command below (In my case it was necessary to use this command from a different linux system):

curl -v -k -u <user>:<password> curl -v -k -u <user>:<password> https://<host>:<management_port>/services/data/inputs/tcp/splunktcptoken

For a more GUI View open internet explorer and browse this side (user and password like above):

https://<host>:<management_port>/services/data/inputs/tcp/splunktcptoken

You cannot change anything within the internet explorer - it's just a graphical overview

User:password? -> for indexer its like the web "admin:changeme" for a universal forwarder its questioned at the installation!
management_port? -> per default 8089
Example:

curl -v -k -u admin:changeme https://splunk:8089/services/data/inputs/tcp/splunktcptoken

Also possibile at a universal forwarder which is in my case a "intermediate forwarder from our DMZ (demilitarized zone)"

curl -v -k -u admin:supersecurepassword https://splunkforwarder:8089/services/data/inputs/tcp/splunktcptoken

This is also an answer i will later comment on ...

2. Problem: Can i manage tokens on my clients (forwarders) via a deployment server?
2. Solution:
I would say not to try this but - I didn't test that cause in my setup i implemented it on a universal forwarder and did not had a deployment server.
Why do i say that.. it has something to do with the following mechanism.

2. SplunkTcpToken Explanation:
A little bit was documented in the splunk docs ... but it was not a lot of information. Let me clear some things here.

1. You'll have to create a splunktcptoken on the server which should work with splunktcp://9997 input + TCP_AUTH we name it "ServerA"
2. You'll need to modify the inputs.conf on ServerA as well

3. You'll need to modify the outputs.conf of every server who wants to send traffic to SeverA's "splunktcp:9997" Splunk Port.

Explanation (1):
Go to a linux server and open a bash shell and type the command below. Check that you have curl installed. (Normally it is.)

curl -v -k -u admin:changeme https://splunk:8089/services/data/inputs/tcp/splunktcptoken

It will create a token AND enables the token. Anything that is sending to the splunktcp://9997 at this time will directly be blocked.
The output of the command above shows something similar like that (i've shortet the output [...]):

<entry>
    <title>splunktcptoken://my_token</title>
    <id>https://splunkforwarder:8089/servicesNS/nobody/search/data/inputs/tcp/splunktcptoken/splunktcptoken%3A%252F%252Fmy_token</id>
   [...]
        <s:key name="_rcvbuf">1572864</s:key>
        <s:key name="disabled">0</s:key>
   [...]
        <s:key name="host">splunkforwarder</s:key>
        <s:key name="index">default</s:key>
        <s:key name="token">70C70ABF-5280-4G47-A298-551GH151A564</s:key>
   [...]

Necessary to see in the output:

<title>splunktcptoken://my_token</title>
<s:key name="token">70C70ABF-5280-4G47-A298-551GH151A564</s:key>

As i mentioned before you can also lookup those tokens via the internet explorer!
Any enabled token will directly activates a necessary authentication on ALL splunktcp input. Regardless if its SSL or not.
When i say Splunk TCP Input i mean only Splunk TCP input. Any other defined TCP input is not blocked. If testet that!
You can have a look on your splunkd.log (If your splunk runs on a linux see command below):

tail -f -n 10 /opt/splunkforwarder/var/log/splunk/splunkd.log

Look up for the following error which would show you blocked splunktcp connections:

02-25-1999 21:06:31.740 +0200 ERROR TcpInputProc - Invalid S2S token=Token not sent by forwarder received from src=10.0.0.200:57993.

Explanation (2):
Open the inputs config on ServerA (your server on which you want to setup splunktcp inputs tcp authentication) and setup your created token

vi /opt/splunkforwarder/etc/system/local/inputs.conf

Add the following stanza anywhere in your inputs config.

[splunktcptoken://my_token]
token = 5B191D53-46E8-49D0-9CBC-A44CB5097DF9

See the docs (https://docs.splunk.com/Documentation/Splunk/8.0.3/Admin/Inputsconf)
Details from the config:

 Access control settings.
[splunktcptoken://<token name>]
* Use this stanza to specify forwarders from which to accept data.
* You must configure a token on the receiver, then configure the same
  token on forwarders.
* The receiver discards data from forwarders that do not have the
  token configured.
* This setting is enabled for all receiving ports.
* This setting is optional.

token = <string>
* Value of token.

Explanation (3): Tricky part regarding "deployment config !"
Go to any of your clients which has a "Universal Forwarder" running.
Go to you "C:\Program Files\SplunkUniversalForwarder\etc\system\local\outputs.conf" and change the following:
From:

[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
server = splunkforwarder.contoso.com:9997

[tcpout-server://splunkforwarder.contoso.com:9997]

To (add the token "value" from Explanation (1)):

[tcpout]
defaultGroup = default-autolb-group
token = 5B191D53-46E8-49D0-9CBC-A44CB5097DF9

[tcpout:default-autolb-group]
server = splunkforwarder.contoso.com:9997

[tcpout-server://splunkforwarder.contoso.com:9997]

Until this part everything could be managed via deployment server BUT:
If you restart the server the universal forwarder will automatically "encrypt" your token.
Your C:\Program Files\SplunkUniversalForwarder\etc\system\local\outputs.conf looks now like that:

[tcpout]
defaultGroup = default-autolb-group
token = $7$JXINNYdakI+dlFjT6Zl63gk91s8/trLTxTFzaGMc3KA5RHldOCJFt0ZF+ZaliPW8HaKt5cxUqkoSNVrpScZyF+Jrc0Q=
channelReapInterval = 60000
channelReapLowater = 10
channelTTL = 300000
dnsResolutionInterval = 300
negotiateNewProtocol = true
socksResolveDNS = false
useClientSSLCompression = true

[tcpout:default-autolb-group]
server = splunkforwarder.contoso.com:9997

[tcpout-server://splunkforwarder.contoso.com:9997]

Cause of the change of the outputs.conf i'm not sure if you're able to use a deployment server to setup that. If someone knows better please correct me cause i did not test that. I dont use a deployment server in the dmz cause of security reasons. Maybe i will change my mind later.

3. Problem: Can i activate tokens only for a specific input?
3. Answer: Yes and No

3. Explanation: **
Splunk Auth is working only for "splunktcp" per default [splunktcp://9997] there is also the possibility for [splunktcp-ssl://<port>].
Therefore you don't need to worry about other [tcp://] inputs.
The only one downside of this Splunk TCP Auth is that you cannot create different splunktcp inputs. At least i have no idea of how to do that maybe one of you knows how to do that.
**Final Words:
If you want to use TCP Authentication fpr Splunk Traffic .. or as Splunk Docs describe it .. "Control Forwarder Access" you can only go full or go home.
There is no split input.
Some of you will say why don't use SSL? The built in SSL is just a little bit secure ... and the SSL Certs from your own CA is "in my humble opinion" a lot of work for the accomplishment.
Why do i want Splunk TCP Auth in the first place? -> We open the port 9997 for every DMZ Network therefore its a good idea to prevent others from messing with this port. Even with SSL Enabled ... i don't want someone messing with this port.

Best regards,

Michele Evermann

[ekost]

Thank you for your feedback on the forwarder access token documentation. I updated the docs to address some of the issues you discussed.