Getting Data In

TCP listener answered not all connections

azanoli
Explorer

I have the problem, that the TCP listener on indexer xxpu031 answered not all connections. In the TCP dump below, the connection requests from ixpw021 are not answered, the connection requests from ixpw031 are answered.
In the input.conf are no restrictions. The TCP listener listen on all interfaces. There are no errors in the splunkd.log.

# tcpdump -i eth0 host ixpw021 and port 9997
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
09:48:12.644614 IP ixpw021.inventx.ch.61885 > xxpu031.inventx.ch.9997: S 176421035:176421035(0) win 8192 <mss 1460,nop,wscale 8,nop,nop,sackOK>
09:48:18.694549 IP ixpw021.inventx.ch.61885 > xxpu031.inventx.ch.9997: S 176421035:176421035(0) win 8192 <mss 1460,nop,nop,sackOK>
09:48:39.648615 IP ixpw021.inventx.ch.61890 > xxpu031.inventx.ch.9997: S 773549642:773549642(0) win 8192 <mss 1460,nop,wscale 8,nop,nop,sackOK>
09:48:42.640417 IP ixpw021.inventx.ch.61890 > xxpu031.inventx.ch.9997: S 773549642:773549642(0) win 8192 <mss 1460,nop,wscale 8,nop,nop,sackOK>
09:48:48.646348 IP ixpw021.inventx.ch.61890 > xxpu031.inventx.ch.9997: S 773549642:773549642(0) win 8192 <mss 1460,nop,nop,sackOK>

# tcpdump -i eth0 host ixpw031 and port 9997
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
09:49:32.459160 IP ixpw031.inventx.ch.61019 > xxpu031.inventx.ch.9997: S 3801163340:3801163340(0) win 8192 <mss 1460,nop,wscale 8,nop,nop,sackOK>
09:49:32.459175 IP xxpu031.inventx.ch.9997 > ixpw031.inventx.ch.61019: S 3807617395:3807617395(0) ack 3801163341 win 5840 <mss 1460,nop,nop,sackOK,nop,wscale 7>
09:49:32.459604 IP ixpw031.inventx.ch.61019 > xxpu031.inventx.ch.9997: . ack 1 win 256
09:50:02.464550 IP ixpw031.inventx.ch.61019 > xxpu031.inventx.ch.9997: F 1:1(0) ack 1 win 256
09:50:02.464588 IP xxpu031.inventx.ch.9997 > ixpw031.inventx.ch.61019: . ack 2 win 46
09:50:02.464662 IP xxpu031.inventx.ch.9997 > ixpw031.inventx.ch.61019: F 1:1(0) ack 2 win 46
09:50:02.464974 IP ixpw031.inventx.ch.61019 > xxpu031.inventx.ch.9997: . ack 2 win 256
09:50:02.466847 IP ixpw031.inventx.ch.51026 > xxpu031.inventx.ch.9997: P 429:858(429) ack 1 win 256
09:50:02.466855 IP xxpu031.inventx.ch.9997 > ixpw031.inventx.ch.51026: . ack 858 win 501
09:50:02.467182 IP ixpw031.inventx.ch.61021 > xxpu031.inventx.ch.9997: S 2401879704:2401879704(0) win 8192 <mss 1460,nop,wscale 8,nop,nop,sackOK>
09:50:02.467195 IP xxpu031.inventx.ch.9997 > ixpw031.inventx.ch.61021: S 3841597957:3841597957(0) ack 2401879705 win 5840 <mss 1460,nop,nop,sackOK,nop,wscale 7>
09:50:02.467550 IP ixpw031.inventx.ch.61021 > xxpu031.inventx.ch.9997: . ack 1 win 256

# more inputs.conf
[default]
host = xxpu031s.inventx.ch

[splunktcp:9997]

[udp://514]
disabled = false
connection_host = dns
sourcetype = syslog

# netstat -an | grep LIST
tcp        0      0 0.0.0.0:9997                0.0.0.0:*                   LISTEN      
Tags (2)
1 Solution

azanoli
Explorer

The routing table was wrong. We have 2 interfaces, a userLAN and a serviceLAN, on the server. The incoming packets received at the userLAN, the outgoing packets were sent to the serviceLAN. After correction of the routing table, all works correctly.

View solution in original post

azanoli
Explorer

The routing table was wrong. We have 2 interfaces, a userLAN and a serviceLAN, on the server. The incoming packets received at the userLAN, the outgoing packets were sent to the serviceLAN. After correction of the routing table, all works correctly.

azanoli
Explorer

Probably we found the problem. In discussion with our security and network team, I looked at the routing table on the server. We have 2 interfaces, a userLAN and a serviceLAN, on the server. The problem is, we receive the incoming packet on the userLAN, but the routing table points the network to the serviceLAN.

0 Karma

Ayn
Legend

Yes I understand that, you showed that it in your initial question. What I'm asking is if a connection from these 2 non-working servers towards another open port on your Splunk indexer works? Because this really really looks like some kind of firewall problem to me, there's no reason for Splunk itself to not even establish the connection.

azanoli
Explorer

In our splunk environment are about 50 Microsoft server with the splunk universal forwarder. They work all correctly with the exception of 2 servers. Both of them are configured like the other ones.

The TCP SYN for the 3-way-handshake arrives at the indexer but the TCP listener did not answer with a TCP ACK as you can see in the TCP dump. Thus the TCP connection will not established.

0 Karma

Ayn
Legend

So a connection from this host to some other open port will work properly?

azanoli
Explorer

iptables is not installed, TCP wrapper is not activated.

0 Karma

Ayn
Legend

iptables rules?

Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...