I got an other question(s) regarding Splunk App for Stream
I am playing around with the Netflow feature of Stream
I convinced our network guys to send us some netflows.
Even that this is only around 10% of our current netflow data, I estimate around 20 Mbps but it can be a bit bursty, I constantly get the following error
2016-12-05 11:59:59 ERROR [140212496193280] (SplunkSenderModularInput.cpp:429) stream.SplunkSenderModularInput - Event queue overflow; dropping 10001 events
{"timestamp":"2016-12-05T10:59:58.901309Z","agentMode":0,"level":"ERROR","message":"Event queue overflow; dropping 10001 events"}
My best guess is, that the sender queue is "full".
Yes, I have seen the 14 month old post https://answers.splunk.com/answers/311059/splunk-app-for-stream-if-indexing-queue-blocked-or.html
But I am not sure if this applies and/or if some things changed, since some stuff seriously changed since 2014 where streamfwd was still configured in XML.
I am currently running Streamfwd on a Splunk Enterprise Heavy Forwarder with 4 Cores and 4GB on our Openstack Environement
I currently use the following streamfwd.conf
[streamfwd]
processingThreads = 4
netflowReceiver.0.port = 18001
netflowReceiver.0.protocol = udp
netflowReceiver.0.ip = 10.0.102.240
netflowReceiver.0.decoder = netflow
netflowReceiver.0.decodingTreads = 8
outputs.conf
# Turn off indexing on the forwarder
[indexAndForward]
index = false
# TCP output global
[tcpout]
defaultGroup = cluster
forwardedindex.filter.disable = true
indexAndForward = false
# TCP output cluster group
[tcpout:cert-cluster]
indexerDiscovery = cluster-master
forceTimebasedAutoLB = true
useACK = true
maxQueueSize = 500MB
# indexer discovery group
[indexer_discovery:cert-cluster-master]
master_uri = https://:8089
pass4SymmKey =
I am not sure if the maxQueueSize helps on a Heavy Forwarder
The data is forwarded to an indexer cluster, 3 peers in the same environment, replicated to an other 3 backup peers in a 2nd location
This created about 500 Kbps to 2.5 Mbps Indexing rate
What do I have to tweak where?
Fowarder: ouputs.conf or streamfwd.conf and which parameters?
Indexers: resources seem to be not the problem, Maximum load average less than 0.3, etc.
PS: @Splunkers the streamfwd.conf documentation is quite incomplete
I find variables which are not explained, i.e. ipAddr, is this now the interface address it should connect to? can I connect to multiple, i.e. connect to multiple? or does this overwrite the IP .. or whatever...
Default Values are not documented, also /default/ does not reveal it
I keep finding variables in the examples that are not documented at all, i.e. netflowReceiver.0.decodingThreads http://docs.splunk.com/Documentation/StreamApp/7.0.0/DeployStreamApp/Performancetestresults#Flow_collector_test_environment
... View more