ports used between splunk instances

[nwieseler]

I have two splunk instances on either side of a firewall. I use the deployment server and a shared license. Each instance is an indexer and I use distributed search if that matters.

I'm seeing inbound firewall traffic from 8089 to a random high port on the inside of the firewall.

What ports do I need to permit between these two servers? Right now I'm just permitting any tcp between the two but over the last 12 hours, they've made connections on 18 different ports.

How are you guys watching traffic if you have a firewall between the two (in this case it's a firewall between a couple of vlans).

Thanks!

Nick

Snip of ports I'm seeing:

50280
51473
52118
52843
56289
63700
64082
51649
51836
52449
53372
53716
57232
57330
57634
58366
59676
62753

[Ayn]

The only port used would be 8089. It's also common for forwarding Splunk instances to send their data on port 9997, but from your description of your scenario that doesn't apply here. The random high ports you're seeing are source ports, not destination ports - the only traffic you should need to allow is to port 8089 on your Splunk instances.

[jonuwz]

On the splunk server - port 63700 ( for example ) will be the destination port for communication back to a client.

But the client would have initiated the connection to splunk_server:8089.

You'll have a firewall rule saying 'allow traffic from client ip to server ip on 8089'.

The source port will be > 1024.

The firewall then keeps track of that connection by inspecting sequence numbers etc etc to allow traffic back to the client ip:port.

The server will not be able to initiate a connection to the client on a random high port ( unless you configured it that way )

[Ayn]

If you have devices that don't keep track of state (like routers) and look at the traffic logs there you'll see something like what you've pasted - while it IS true that data is sent from random high ports on 10.128.10.10 towards 10.1.10.10, but the connection itself is initiated from the latter. A stateless device will not able to tell the difference.

[Ayn]

I'm referring to the devices you have in your network that inspect the traffic and then pass it on. Mainly firewalls and routers. With stateful I mean that they can keep track of which state a connection is on, so that they know if a connection is initiated from one host towards another or the other way around.

[nwieseler]

That seems like a more likely possibility although I'm not sure what you mean by "stateful inspecting devices"?

Thanks!

[Ayn]

Sounds like some of your stateful inspecting devices are confused in your environment - I can guarantee you that Splunk will not initiate connections on those high ports.

[nwieseler]

Typically I would agree here, however, see this snip from my logs... you'll see the source port is 8089 and the destination port is a high port which is why I'm entirely baffled by this... I def. could be missing something here though...

%ASA-5-106100: access-list NET_ACCESS_IN permitted tcp outside/10.128.10.10(8089) -> inside/10.1.10.10(52118)