Why does the forwarder service try to use a non-9997 port?

[patterc]

We have a Splunk Enterprise installed in a DMZ with strict firewall rules about how to communicate with our index array. When I set up forwarding on the outputs.conf, I designated our Indexer IPs and port 9997.

[tcpout:default-autolb-group]
disabled = false
server = IP1:9997, IP2:9997, IP3:9997, IP4:9997

Forwarding isn't working, though. When I check the ports with the "lsof -i -P -n" command, I see that the Heavy Forwarder tries to talk to IP1 over random ports

splunkd 31931 root   61u  IPv4 570678      0t0  TCP [Heavy Forwarder]:41464->[IP2]:9997 (SYN_SENT)

Can I force the outbound SYN to go over port 9997?

[PavelP · Accepted solution]

Hello @patterc ,

TCP [Heavy Forwarder]:41464->[IP2]:9997 (SYN_SENT) means the HF use an ephemeral source port (in this case 41464) to connect to the dest port 9997. There is no error so far. In the outputs.conf you define the destination port only.

The SYN_SENT means the HF try to establish a TCP connection, the first step is to send a SYN packet but haven't got an answer yet. This mostly means there is a firewall or ACL in-between, which blocking the connection.

So, to answer your question directly - the HF uses a pseudo-random ephemeral source port (https://en.wikipedia.org/wiki/Ephemeral_port) to establish a TCP connection to a server port (9997).

The second question "Can I force the outbound SYN to go over port 9997?" - theoretically yes, but then HF will be able to establish only ONE connection to the indexer at the time IF this source port not used yet. This is a wrong path 🙂

Instead try to find what is blocking the connection. This can be (in order of desc probability) firewall in-between, firewall on the indexer or firewall on HF (unusual).

[patterc]

Thanks Pavel for the explanation on an Ephemeral port. I haven't heard that yet. My organization is fairly strict on the firewall rules, but this makes sense. I supposed you can get some backed up connections if you're trying to send data and receive it on the same port.

There is definitely a firewall blocking the packets in this case because the firewall rule is expecting all comms to go over port 9997.

[richgalloway]

That is normal behavior for Splunk forwarders (and most other processes). You specify the destination port of a connection, not the source port. The source server's OS will choose the source port. For example, many users may run ssh to connect from one server to another, but they can't all go out on port 22. As it happens, none of them go out port 22 as 22 is the receiving port for ssh connections.
Change your firewall rules to allow connections from DMZ forwarders on any port to your indexers' port 9997.

[gcusello]

Hi @patterc,
Using the outputs.con as you did, you are forcing Forwarder to use the 9997 port, and Splunk doesn't try to use a different port, it uses only the configured ports; but you have to check if the routes between Forwarder and Indexers on 9997 port are open.

Another stupid question: did you enabled Indexers to receive on 9997 port?

Ciao.
Giuseppe

[patterc]

Yes, the Indexers receive data on port 9997. They receive data from other forwarders successfully.

The problem is that when the Forwarder sends the data out, it leaves the box over a non-9997 port. That can be seen in the line ending with (SYN_SENT) where it tried to use port 41464. If I run the lsof -i -P -n command again, the port will be different.