topic Re: Splunk Deployment Server Listening on HTTP 8088 in Getting Data In

Splunk Deployment Server Listening on HTTP 8088

TheColorBlack — Mon, 08 Feb 2021 22:47:10 GMT

Hey all, quick question, and I apologize in advance if this isn't the proper sub-forum for this question.

In a scaled multi-site cluster deployment is it normal for your Deployment Servers to be listening on the default HEC Collection Port, TCP 8088? I ask because my Deployment Server is, even though there is no explicit configuration on the server exposing port 8088.

To my knowledge I do not have any configurations under "$splunkHome/etc/system/local/" or "$splunkHome/etc/apps/" that would expose TCP 8088 to the world.

There are only two lines in my "$splunkHome/etc/apps/splunk_httpinput/local/inputs.conf" file, they are:

[http] useDeploymentServer = 1

As far as I know this stanza tells the Splunk Software to place any HTTP Token / configurations made via the Deployment Server UI straight into "$splunkHome/etc/deployment-apps/splunk_httpinput/local/inputs.conf" so they can be pushed out to Indexers / Heavy Forwarders, but does not serve to enable HTTP event collection via TCP 8088 on the deployment server itself.

As a newbie Splunk Admin, where else could I check to see what's causing my deployment server to listen on TCP port 8088?

Re: Splunk Deployment Server Listening on HTTP 8088

scelikok — Tue, 09 Feb 2021 03:47:21 GMT

Hi @TheColorBlack,

Is it possible that your DeploymentServers are in the same serverclass with Heavy Forwarders which uses HEC inputs? You can use btool to find the configuration about http input on Deployment server.

$SPLUNK_HOME/bin/splunk btool inputs list http --debug | grep local

Re: Splunk Deployment Server Listening on HTTP 8088

isoutamo — Tue, 09 Feb 2021 09:05:18 GMT

Deployment server is running inside splunkd process and it is using the same port than normal splunk management traffic. Usually this is 8089. You could check this e.g. from any DS client's configuration from any working DS client.

IN deploymentclient.conf [target-broker:deploymentServer] targetUri = 1.2.3.4:8089

When you have multisite cluster, it is managed by cluster master never ever by deployment server!

To check what is your splunkd's management port and where it has configured, you could use this:

splunk btool web list --debug |egrep mgmtHostPort /opt/splunk/etc/system/default/web.conf mgmtHostPort = 127.0.0.1:8089

r. Ismo

Re: Splunk Deployment Server Listening on HTTP 8088

TheColorBlack — Wed, 10 Feb 2021 22:27:36 GMT

@scelikok @isoutamo

Thank you both for the insight into the usefulness of btool. Unfortunately I'm still not able to find what's causing my deployment server to listen on 8088. Here's some command output executed from my deployment server.

For what it's worth I looked at the server classes defined on this deployment server and my deployment server is not part of any server class as far as I can tell.

root@aws-11-spds-01:~# netstat -anoltp |grep -i 8088 tcp 0 0 0.0.0.0:8088 0.0.0.0:* LISTEN 3010/splunkd off (0.00/0/0)

splunk btool inputs list http ------ [http] _rcvbuf = 1572864 ackIdleCleanup = true allowSslCompression = true allowSslRenegotiation = true dedicatedIoThreads = 2 disabled = 1 enableSSL = 1 host = aws-11-spds-01 index = default maxSockets = 0 maxThreads = 0 port = 8088 sslVersions = *,-ssl2 useDeploymentServer = 1

> splunk btool inputs list http --debug |grep -i 8088 /opt/splunk/etc/apps/splunk_httpinput/default/inputs.conf port = 8088 Contents of /opt/splunk/etc/apps/splunk_httpinput/default/inputs.conf ------ [http] disabled=1 port=8088 enableSSL=1 dedicatedIoThreads=2 maxThreads = 0 maxSockets = 0 useDeploymentServer=0 # ssl settings are similar to mgmt server sslVersions=*,-ssl2 allowSslCompression=true allowSslRenegotiation=true ackIdleCleanup=true Contents of /opt/splunk/etc/apps/splunk_httpinput/local/inputs.conf ------ [http] useDeploymentServer = 1

Re: Splunk Deployment Server Listening on HTTP 8088

isoutamo — Thu, 11 Feb 2021 05:54:31 GMT

this shows that your HEC input is disabled. Can you try

egrep -R 8088 /opt/splunk/etc| egrep \.conf

so we could see where this value is defined.

Re: Splunk Deployment Server Listening on HTTP 8088

TheColorBlack — Mon, 22 Feb 2021 17:37:02 GMT

Hey @isoutamo sorry for the late follow up to your question. Things got crazy in Texas last week lol. Thank you for your continued assistance.

Here's the output of grepping for 8088 across Splunk's /etc/ directory. The results show references to the AWS Addon as well as one or two default configuration files.

> egrep -R 8088 /opt/splunk/etc| egrep \.conf /opt/splunk/etc/system/README/inputs.conf.spec:* Default: 8088. /opt/splunk/etc/deployment-apps/Splunk_TA_aws/README/inputs.conf.spec:use_raw_hec = scheme://netloc/token, for instance, https://192.168.1.1:8088/550E8400-E29B-41D4-A716-446655440000. /opt/splunk/etc/deployment-apps/Splunk_TA_aws/README/aws_inspector.conf.spec:hec_port = 8088, Http Event Collector port /opt/splunk/etc/deployment-apps/Splunk_TA_aws/README/aws_config_rule.conf.spec:hec_port = 8088, Http Event Collector port /opt/splunk/etc/deployment-apps/Splunk_TA_aws/README/aws_kinesis.conf.spec:hec_port = 8088, Http Event Collector port /opt/splunk/etc/deployment-apps/Splunk_TA_aws/default/aws_inspector.conf:hec_port = 8088 /opt/splunk/etc/deployment-apps/Splunk_TA_aws/default/aws_config_rule.conf:hec_port = 8088 /opt/splunk/etc/deployment-apps/Splunk_TA_aws/default/aws_kinesis.conf:hec_port = 8088 /opt/splunk/etc/deployment-apps/Splunk_TA_aws/bin/3rdparty/python3/splunktalib/hec_config.py: "port": 8088, /opt/splunk/etc/deployment-apps/Splunk_TA_aws/bin/3rdparty/python3/splunktalib/hec_config.py: port = config.get("hec_port", 8088) /opt/splunk/etc/deployment-apps/Splunk_TA_aws/bin/3rdparty/python3/solnlib/hec_config.py: 'port': 8088} /opt/splunk/etc/deployment-apps/Splunk_TA_aws/bin/3rdparty/python3/solnlib/hec_config.py: 'port': 8088} /opt/splunk/etc/deployment-apps/splunk_httpinput/local/inputs.conf:port = 8088 /opt/splunk/etc/deployment-apps/splunk_httpinput/default/inputs.conf:port=8088 /opt/splunk/etc/deployment-apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py3/solnlib/hec_config.py: 'port': 8088} /opt/splunk/etc/deployment-apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py3/solnlib/hec_config.py: 'port': 8088} /opt/splunk/etc/deployment-apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py2/solnlib/hec_config.py: 'port': 8088} /opt/splunk/etc/deployment-apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py2/solnlib/hec_config.py: 'port': 8088} /opt/splunk/etc/apps/splunk_httpinput/default/inputs.conf:port=8088

Re: Splunk Deployment Server Listening on HTTP 8088

TheColorBlack — Mon, 01 Mar 2021 19:38:48 GMT

Alright, I've figured this one out. Posting the solution for anyone else who may run into the same "issue". This behavior is not explicitly stated in the Scale HTTP Event Collector with distributed deployments documentation. I've opened a case with Splunk Support to clarify weather or not this is the intended behavior and will update this thread accordingly with their answer.

The following is called out within the documentation under the "Place and distribute the HEC on heavy forwarders" heading within the aforementioned documentation.

If you plan to distribute HEC configurations through the deployment server, set the useDeploymentServer option in the [http] stanza of inputs.conf on the deployment server to 1. When this option is set to 1 and you make UI-based HEC changes on the deployment server, those changes are placed directly in the $SPLUNK_HOME/etc/deployment-apps/splunk_httpinput/ folder, rather than in $SPLUNK_HOME/etc/apps/splunk_httpinput/. See the inputs.conf spec file for further information.

It should also be noted that enabling this setting causes your deployment server to expose TCP Port 8088.

#$SplunkHome/etc/apps/splunk_httpinput/local/inputs.conf (This is the full config file) [http] useDeploymentServer = 0 allowSslCompression = false

root@test-deploymentServer:~# netstat -anoltp |grep -i 8088 NO RESULTS.

Versus

#$SplunkHome/etc/apps/splunk_httpinput/local/inputs.conf (This is the full config file) [http] useDeploymentServer = 1 allowSslCompression = false

root@test-DeploymentServer:~# netstat -anoltp |grep -i 8088 tcp 0 0 0.0.0.0:8088 0.0.0.0:* LISTEN 10639/splunkd off (0.00/0/0) root@test-DeploymentServer:~# telnet 127.0.0.1 8088 Trying 127.0.0.1... Connected to 127.0.0.1. Escape character is '^]'. ^]

E: Here's the official response from Splunk Support regarding this issue

According to the splunk attribute specifications: useDeploymentServer = <boolean> * Whether or not the HTTP event collector input should write its configuration to a deployment server repository. * When you enable this setting, the input writes its configuration to the directory that you specify with the 'repositoryLocation' setting in the serverclass.conf file. * You must copy the full contents of the splunk_httpinput app directory to this directory for the configuration to work. * When enabled, only the tokens defined in the splunk_httpinput app in this repository are viewable and editable through the API and Splunk Web. * When disabled, the input writes its configuration to $SPLUNK_HOME/etc/apps by default. * Default: 0 (disabled) https://docs.splunk.com/Documentation/Splunk/8.1.2/Admin/Inputsconf Maybe should be more explicit as you mention, if you feel that this is something should be improved or more clear thru our documentation we have Splunk Ideas, please check: https://docs.splunk.com/Documentation/Community/1.0/community/SplunkIdeas