Stream capture issue with persistent connection(s)

[bwheelock]

I'm new to Stream and not particularly experienced with wire data, but we have a test box receiving a span port (for now) to capture traffic to a certain server, and have been successful in setting up tcp flow captures. We recently reached out to another team to inspect some of their data coming to our Stream test server, however, and noticed that we are seeing hardly any traffic from their server. We only have a single Splunk event per port/protocol their server, so far, and it came in a long while after we started capturing.

We have no filters in place at the moment, and are capturing only tcp traffic.

To troubleshoot, we ran a tcpdump for traffic from this team's source, where we do see all of their raw traffic as expected, and then did a direct import of that capture file. The result was that we had two Splunk events total, one for each port being used by their application. We ran three separate captures, each limited to a different number of packets: 100, 200, 1000. This produced six events in Splunk, where we assumed there would be perhaps dozens given the traffic they are generating.

The issue appears to be with certain(?) persistent connections, but perhaps this is expected behavior? To my knowledge the application sending this traffic is your typical aggregate of clients talking to a database through an application/service. IE: Client hits app, app speaks to db, we capture app to db traffic.

This Answers post appears to be reporting the same or similar, but the marked answer is not an actual solution.

[vshcherbakov_sp]

bwheelock, what do you mean by "events" here? TCP connections? What streams are enabled? By default, Stream captures only TCP and UDP flow events, i.e. it generates a single event for a TCP connection or UDP flow. If you want to capture specific protocol (HTTP, DNS, etc.) events, you will need to enable/configure the corresponding streams in the App for Stream config UI.

[bwheelock]

I apologize, I think I need to read more first. We are capturing TCP flows.

"Events" are meant to only refer to Splunk events / individual results. I accidentally also referred to capture results as events when I should have said raw packets.

I think the problem here is/was my understanding of Stream architecture. I had assumed each Splunk event was a single packet initially, having not yet inspected much of the data, but I see now that each Splunk event is a grouping of reassembled packets. I'm still a little lost on how/when reassembly takes place, so I think I need to read more documentation. For example we tested some SSH file transfers and after transferring four files, one at a time, we received one Splunk event that appears to have grouped together all of that wire data.