Splunk Deployment Server Listening on HTTP 8088

[TheColorBlack]

Hey all, quick question, and I apologize in advance if this isn't the proper sub-forum for this question.

In a scaled multi-site cluster deployment is it normal for your Deployment Servers to be listening on the default HEC Collection Port, TCP 8088? I ask because my Deployment Server is, even though there is no explicit configuration on the server exposing port 8088.

To my knowledge I do not have any configurations under "$splunkHome/etc/system/local/" or "$splunkHome/etc/apps/" that would expose TCP 8088 to the world.

There are only two lines in my "$splunkHome/etc/apps/splunk_httpinput/local/inputs.conf" file, they are:

[http]
useDeploymentServer = 1

As far as I know this stanza tells the Splunk Software to place any HTTP Token / configurations made via the Deployment Server UI straight into "$splunkHome/etc/deployment-apps/splunk_httpinput/local/inputs.conf" so they can be pushed out to Indexers / Heavy Forwarders, but does not serve to enable HTTP event collection via TCP 8088 on the deployment server itself.

As a newbie Splunk Admin, where else could I check to see what's causing my deployment server to listen on TCP port 8088?

[TheColorBlack · Accepted solution]

Alright, I've figured this one out. Posting the solution for anyone else who may run into the same "issue". This behavior is not explicitly stated in the Scale HTTP Event Collector with distributed deployments documentation. I've opened a case with Splunk Support to clarify weather or not this is the intended behavior and will update this thread accordingly with their answer.

The following is called out within the documentation under the "Place and distribute the HEC on heavy forwarders" heading within the aforementioned documentation.

If you plan to distribute HEC configurations through the deployment server, set the useDeploymentServer option in the [http] stanza of inputs.conf on the deployment server to 1. When this option is set to 1 and you make UI-based HEC changes on the deployment server, those changes are placed directly in the $SPLUNK_HOME/etc/deployment-apps/splunk_httpinput/ folder, rather than in $SPLUNK_HOME/etc/apps/splunk_httpinput/. See the inputs.conf spec file for further information.

It should also be noted that enabling this setting causes your deployment server to expose TCP Port 8088.

#$SplunkHome/etc/apps/splunk_httpinput/local/inputs.conf (This is the full config file)

[http]
useDeploymentServer = 0
allowSslCompression = false

root@test-deploymentServer:~# netstat -anoltp |grep -i 8088

NO RESULTS.

Versus

#$SplunkHome/etc/apps/splunk_httpinput/local/inputs.conf (This is the full config file)

[http]
useDeploymentServer = 1
allowSslCompression = false

root@test-DeploymentServer:~# netstat -anoltp |grep -i 8088
tcp        0      0 0.0.0.0:8088            0.0.0.0:*               LISTEN      10639/splunkd        off (0.00/0/0)

root@test-DeploymentServer:~# telnet 127.0.0.1 8088
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
^]

E: Here's the official response from Splunk Support regarding this issue

According to the splunk attribute specifications:

useDeploymentServer = <boolean>
* Whether or not the HTTP event collector input should write its
  configuration to a deployment server repository.
* When you enable this setting, the input writes its
  configuration to the directory that you specify with the
  'repositoryLocation' setting in the serverclass.conf file.
* You must copy the full contents of the splunk_httpinput app directory
  to this directory for the configuration to work.
* When enabled, only the tokens defined in the splunk_httpinput app in this
  repository are viewable and editable through the API and Splunk Web.
* When disabled, the input writes its configuration to
  $SPLUNK_HOME/etc/apps by default.
* Default: 0 (disabled)

https://docs.splunk.com/Documentation/Splunk/8.1.2/Admin/Inputsconf 

Maybe should be more explicit as you mention, if you feel that this is something should be improved or more clear thru our documentation we have Splunk Ideas, please check:  https://docs.splunk.com/Documentation/Community/1.0/community/SplunkIdeas

[isoutamo]

Hi

Deployment server is running inside splunkd process and it is using the same port than normal splunk management traffic. Usually this is 8089.  You could check this e.g. from any DS client's configuration from any working DS client.

IN deploymentclient.conf
[target-broker:deploymentServer]
targetUri = 1.2.3.4:8089

 When you have multisite cluster, it is managed by cluster master never ever by deployment server!

To check what is your splunkd's management port and where it has configured, you could use this:

splunk btool web list --debug |egrep mgmtHostPort
/opt/splunk/etc/system/default/web.conf    mgmtHostPort = 127.0.0.1:8089

 r. Ismo

[scelikok]

Hi @TheColorBlack,

Is it possible that your DeploymentServers are in the same serverclass with Heavy Forwarders which uses HEC inputs? You can use btool to find the configuration about http input on Deployment server.

$SPLUNK_HOME/bin/splunk btool inputs list http --debug | grep local

[TheColorBlack]

@scelikok @isoutamo 

Thank you both for the insight into the usefulness of btool. Unfortunately I'm still not able to find what's causing my deployment server to listen on 8088. Here's some command output executed from my deployment server.

For what it's worth I looked at the server classes defined on this deployment server and my deployment server is not part of any server class as far as I can tell.

root@aws-11-spds-01:~# netstat -anoltp |grep -i 8088

tcp        0      0 0.0.0.0:8088            0.0.0.0:*               LISTEN      3010/splunkd         off (0.00/0/0)

splunk btool inputs list http 

------
[http]
_rcvbuf = 1572864
ackIdleCleanup = true
allowSslCompression = true
allowSslRenegotiation = true
dedicatedIoThreads = 2
disabled = 1
enableSSL = 1
host = aws-11-spds-01
index = default
maxSockets = 0
maxThreads = 0
port = 8088
sslVersions = *,-ssl2
useDeploymentServer = 1

> splunk btool inputs list http --debug |grep -i 8088

/opt/splunk/etc/apps/splunk_httpinput/default/inputs.conf port = 8088

Contents of /opt/splunk/etc/apps/splunk_httpinput/default/inputs.conf
------

[http]
disabled=1
port=8088
enableSSL=1
dedicatedIoThreads=2
maxThreads = 0
maxSockets = 0
useDeploymentServer=0
# ssl settings are similar to mgmt server
sslVersions=*,-ssl2
allowSslCompression=true
allowSslRenegotiation=true
ackIdleCleanup=true

Contents of /opt/splunk/etc/apps/splunk_httpinput/local/inputs.conf
------

[http]
useDeploymentServer = 1

[isoutamo]

Hi

this shows that your HEC input is disabled. Can you try 

egrep -R 8088 /opt/splunk/etc| egrep \.conf

so we could see where this value is defined.  

[TheColorBlack]

Hey @isoutamo sorry for the late follow up to your question. Things got crazy in Texas last week lol. Thank you for your continued assistance.

Here's the output of grepping for 8088 across Splunk's /etc/ directory. The results show references to the AWS Addon as well as one or two default configuration files.

> egrep -R 8088 /opt/splunk/etc| egrep \.conf

/opt/splunk/etc/system/README/inputs.conf.spec:* Default: 8088.
/opt/splunk/etc/deployment-apps/Splunk_TA_aws/README/inputs.conf.spec:use_raw_hec = scheme://netloc/token, for instance, https://192.168.1.1:8088/550E8400-E29B-41D4-A716-446655440000.
/opt/splunk/etc/deployment-apps/Splunk_TA_aws/README/aws_inspector.conf.spec:hec_port = 8088, Http Event Collector port
/opt/splunk/etc/deployment-apps/Splunk_TA_aws/README/aws_config_rule.conf.spec:hec_port = 8088, Http Event Collector port
/opt/splunk/etc/deployment-apps/Splunk_TA_aws/README/aws_kinesis.conf.spec:hec_port = 8088, Http Event Collector port
/opt/splunk/etc/deployment-apps/Splunk_TA_aws/default/aws_inspector.conf:hec_port = 8088
/opt/splunk/etc/deployment-apps/Splunk_TA_aws/default/aws_config_rule.conf:hec_port = 8088
/opt/splunk/etc/deployment-apps/Splunk_TA_aws/default/aws_kinesis.conf:hec_port = 8088
/opt/splunk/etc/deployment-apps/Splunk_TA_aws/bin/3rdparty/python3/splunktalib/hec_config.py:        "port": 8088,
/opt/splunk/etc/deployment-apps/Splunk_TA_aws/bin/3rdparty/python3/splunktalib/hec_config.py:    port = config.get("hec_port", 8088)
/opt/splunk/etc/deployment-apps/Splunk_TA_aws/bin/3rdparty/python3/solnlib/hec_config.py:            'port': 8088}
/opt/splunk/etc/deployment-apps/Splunk_TA_aws/bin/3rdparty/python3/solnlib/hec_config.py:            'port': 8088}
/opt/splunk/etc/deployment-apps/splunk_httpinput/local/inputs.conf:port = 8088
/opt/splunk/etc/deployment-apps/splunk_httpinput/default/inputs.conf:port=8088
/opt/splunk/etc/deployment-apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py3/solnlib/hec_config.py:            'port': 8088}
/opt/splunk/etc/deployment-apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py3/solnlib/hec_config.py:            'port': 8088}
/opt/splunk/etc/deployment-apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py2/solnlib/hec_config.py:            'port': 8088}
/opt/splunk/etc/deployment-apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py2/solnlib/hec_config.py:            'port': 8088}
/opt/splunk/etc/apps/splunk_httpinput/default/inputs.conf:port=8088