Splunk does not listen on specified port

[jameshfisher]

I have defined the following stanza in my inputs.conf:

[root@splunkenterprise etc]# cat /opt/splunk/etc/system/local/inputs.conf
[splunktcp:9997]
disabled = 0  # Yup, this crazily defaults to 1 (true)

I have restarted the service:

[root@splunkenterprise etc]# sudo service splunk stop
Stopping Splunk...
Stopping splunkweb...
                                                           [  OK  ]
Stopping splunkd...
Shutting down.  Please wait, as this may take a few minutes.
.                                                          [  OK  ]
Stopping splunk helpers...
                                                           [  OK  ]
Done.
[root@splunkenterprise etc]# sudo service splunk start
Starting Splunk...

Splunk> The Notorious B.I.G. D.A.T.A.

Checking prerequisites...
    Checking http port [8000]: open
    Checking mgmt port [8089]: open
    Checking configuration...  Done.
    Checking critical directories...    Done
    Checking indexes...
        Validated: _audit _blocksignature _internal _introspection _thefishbucket history main summary
    Done
    Checking filesystem compatibility...  Done
    Checking conf files for problems...
    Done
All preliminary checks passed.

Starting splunk server daemon (splunkd)...  
Done
                                                           [  OK  ]
                                                           [  OK  ]
Starting splunkweb...  Done

If you get stuck, we're here to help.  
Look for answers here: http the idiotic rules on this site say that this is a link to an external site which I am not allowed to post

The Splunk web interface is at http foo. bar baz :8000 the idiotic rules on this site say that this is a link to an external site which I am not allowed to post

I then expect port 9997 to be open. But it isn't:

[root@splunkenterprise etc]# netstat -tulpn | grep 9997
... nothing here ...

It is also not listed if I visit http foo bar baz :8000/en-GB/manager/launcher/datainputstats (the idiotic rules on this site say that this is a link to an external site which I am not allowed to post).

Why is TCP port 9997 not open? It seems pretty clear to me that I have requested for it to be open.

Consulting the documentation (see http docs dot splunk dot com/Documentation/Splunk/6.1.2/admin/inputsconf, I apparently don't have enough karma to link to what is supposedly an external site):

[splunktcp:<port>]
* This input stanza is same as [splunktcpthe idiotic rules on this site say that this is a link to an external site which I am not allowed to post:<port>] but without any remote server restriction

Alright, so my inputs.conf is equivalent to

[splunktcpthe idiotic rules on this site say that this is a link to an external site which I am not allowed to post]
disabled = 0

Consulting the documentation on that:

[splunktcpthe idiotic rules on this site say that this is a link to an external site which I am not allowed to post]
...
* This is the same as TCP, except the remote server is assumed to be a Splunk instance, most likely a forwarder. 

Alright, so my inputs.conf is mostly equivalent to:

[tcpthe idiotic rules on this site say that this is a link to an external site which I am not allowed to post]
disabled = 0

Consulting the documentation:

[tcpthe idiotic rules on this site say that this is a link to an external site which I am not allowed to post]
* Configure Splunk to listen on a specific port. 
...

So the documentation says that my inputs.conf has configured Splunk to listen on the specific port 9997. And when restarting, Splunk says it's "Checking conf files for problems", but finds none. Which implies to me that port 9997 should be open. But it isn't.

[martin_mueller · Accepted solution]

Not seeing splunktcp://9997 in Settings -> Data Inputs -> TCP is correct. You should see it in Settings -> Forwarding and Receiving -> Configure receiving.
The connection from Forwarders is too special to be listed under regular data inputs... despite appearing in inputs.conf 🙂

[KpiBuff]

Aha! After reading docs, tutorials, and having a heck of a time getting my Universal Forwarder connected, this was the answer. Thanks Martin!

[piebob]

hi @jameshfisher: the rule regarding posting links is to prevent spammers from posting spam links. after you've spent some time here and have contributed more than just questions, you will accrue enough karma points to post links. i agree that links to the docs should be exempt, though--i'll look into fixing that. thanks.

[jameshfisher]

If someone could turn off the idiotic rules on this site say that I am not allowed to post links to external sites, it would make your site better, i.e. usable.