When using the Splunk Stream app, why does Splunk suddenly stop indexing NetFlow data every 2 hours?

[fedecastiglio]

Hi community,

I've configured Splunk Stream to ingest NetFlow data (stream collector and Splunk indexer running on the same box), and it's actually working. But exactly every 2 hours, there is a 10 minute gap of data. Packet captures show normal traffic during that gap, so it looks like Splunk is not indexing that data.

Any idea of what could be the reason?

Thanks!

[fedecastiglio]

Looking at the streamfwd.log this seems to be the cause (several errors with diverse id numbers):

stream.NetflowReceiver - NetFlowDecoder::decodeFlow Unable to decode flow set data. No template with id <#>

So I guess Splunk Stream is not processing some template records?

[fedecastiglio]

Some updates:
After tweaking kernel parameters mentioned in Stream documentation I noticed some improvement.

We still have gaps every 2 hours, but those gaps last for 2 minutes.

This are the parameters:

(Before - Default)
net.core.rmem_default = 212992
net.core.rmem_max = 212992
net.core.netdev_max_backlog = 1000

(After)
net.core.rmem_default = 33554432
net.core.rmem_max = 33554432
net.core.netdev_max_backlog = 10000

Documentation suggests those parameters for high-volume packet capture. I'm not entirely sure of what's going on, as I don't think we have that amount of traffic.