Forwarder to splunkstorm is timing out?

[SeanKilleen]

Hi all,

I did the following:

• Set up a splunk forwarder
• Obtained my SplunkStorm Credentials
• Installed splunk credentials SPL credentials (though I'm not sure that I did this 100% correctly)
• Edited inputs.conf to add windows system log events
• Started the forwarder.

In the log, I see several lines of:

Line 272: 10-11-2013 11:53:30.478 -0400 WARN  TcpOutputProc - Raw connection to ip=107.20.29.58:9997 timed out
Line 276: 10-11-2013 11:54:30.479 -0400 WARN  TcpOutputProc - Cooked connection to ip=54.224.46.188:9997 timed out

I ran the command splunk cmd btool outputs list --debug and got the result (sslpassword and project id has been changed from its value to [redacted]:

C:\Program Files\SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\default\outputs.conf                        [tcpout]
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               autoLBFrequency = 30
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               blockOnCloning = true
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               blockWarnThreshold = 100
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               compressed = false
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               connectionTimeout = 20
C:\Program Files\SplunkUniversalForwarder\etc\apps\stormforwarder_[redacted]\default\outputs.conf defaultGroup = storm_indexers
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               disabled = false
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               dropClonedEventsOnQueueFull = 5
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               dropEventsOnQueueFull = -1
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               forceTimebasedAutoLB = false
C:\Program Files\SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\default\outputs.conf                        forwardedindex.0.whitelist = .*
C:\Program Files\SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\default\outputs.conf                        forwardedindex.1.blacklist = _.*
C:\Program Files\SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\default\outputs.conf                        forwardedindex.2.whitelist = _audit
C:\Program Files\SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\default\outputs.conf                        forwardedindex.filter.disable = false
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               heartbeatFrequency = 30
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               indexAndForward = false
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               maxConnectionsPerIndexer = 2
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               maxFailuresPerInterval = 2
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               maxQueueSize = auto
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               readTimeout = 300
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               secsInFailureInterval = 1
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               sendCookedData = true
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               useACK = false
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               writeTimeout = 300
C:\Program Files\SplunkUniversalForwarder\etc\apps\stormforwarder_[redacted]\local\outputs.conf   [tcpout:storm_indexers]
C:\Program Files\SplunkUniversalForwarder\etc\apps\stormforwarder_[redacted]\default\outputs.conf autoLB = true
C:\Program Files\SplunkUniversalForwarder\etc\apps\stormforwarder_[redacted]\default\outputs.conf server = forwarder.d9bw-e6eh.data.splunkstorm.com:9997
C:\Program Files\SplunkUniversalForwarder\etc\apps\stormforwarder_[redacted]\default\outputs.conf sslCertPath = $SPLUNK_HOME/etc/auth/server.pem
C:\Program Files\SplunkUniversalForwarder\etc\apps\stormforwarder_[redacted]\local\outputs.conf   sslPassword = [redacted]
C:\Program Files\SplunkUniversalForwarder\etc\apps\stormforwarder_[redacted]\default\outputs.conf sslRootCAPath = $SPLUNK_HOME/etc/apps/stormforwarder_[redacted]/ssl/star.splunkstorm.com.chain
C:\Program Files\SplunkUniversalForwarder\etc\apps\stormforwarder_[redacted]\default\outputs.conf useACK = true
C:\Program Files\SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\default\outputs.conf                        [tcpout]
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               autoLBFrequency = 30
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               blockOnCloning = true
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               blockWarnThreshold = 100
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               compressed = false
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               connectionTimeout = 20
C:\Program Files\SplunkUniversalForwarder\etc\apps\stormforwarder_[redacted]\default\outputs.conf defaultGroup = storm_indexers
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               disabled = false
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               dropClonedEventsOnQueueFull = 5
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               dropEventsOnQueueFull = -1
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               forceTimebasedAutoLB = false
C:\Program Files\SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\default\outputs.conf                        forwardedindex.0.whitelist = .*
C:\Program Files\SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\default\outputs.conf                        forwardedindex.1.blacklist = _.*
C:\Program Files\SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\default\outputs.conf                        forwardedindex.2.whitelist = _audit
C:\Program Files\SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\default\outputs.conf                        forwardedindex.filter.disable = false
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               heartbeatFrequency = 30
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               indexAndForward = false
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               maxConnectionsPerIndexer = 2
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               maxFailuresPerInterval = 2
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               maxQueueSize = auto
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               readTimeout = 300
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               secsInFailureInterval = 1
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               sendCookedData = true
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               useACK = false
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               writeTimeout = 300
C:\Program Files\SplunkUniversalForwarder\etc\apps\stormforwarder_[redacted]\local\outputs.conf   [tcpout:storm_indexers]
C:\Program Files\SplunkUniversalForwarder\etc\apps\stormforwarder_[redacted]\default\outputs.conf autoLB = true
C:\Program Files\SplunkUniversalForwarder\etc\apps\stormforwarder_[redacted]\default\outputs.conf server = forwarder.d9bw-e6eh.data.splunkstorm.com:9997
C:\Program Files\SplunkUniversalForwarder\etc\apps\stormforwarder_[redacted]\default\outputs.conf sslCertPath = $SPLUNK_HOME/etc/auth/server.pem
C:\Program Files\SplunkUniversalForwarder\etc\apps\stormforwarder_[redacted]\local\outputs.conf   sslPassword = [redacted]
C:\Program Files\SplunkUniversalForwarder\etc\apps\stormforwarder_[redacted]\default\outputs.conf sslRootCAPath = $SPLUNK_HOME/etc/apps/stormforwarder_[redacted]/ssl/star.splunkstorm.com.chain
C:\Program Files\SplunkUniversalForwarder\etc\apps\stormforwarder_[redacted]\default\outputs.conf useACK = true

[akoeplinger]

I'm seeing exactly the same behavior, my Windows event logs and Perfmon data isn't showing up in Splunk Storm but a normal text-based logfile does. I've already opened a ticket a couple days ago, but got no response yet.

[akoeplinger]

@pbradfordkc: Windows Event Logs don't work yet when using Universal Forwarder 6 with Storm, see http://answers.splunk.com/answers/123027/splunk-storm-universal-forwarder

[pbradfordkc]

I am running into the same issue. I am seeing text based logs but no windows event logs?

Does splunk storm support windows event logs. I thought it was because i was running server 2012 r2 but 2012 r2 is now supported with 6.1. Is there any trouble shooting steps out there?

[yannK · Accepted solution]

The timeout is simply caused when the forwarder rotating across the ip of the dns load balancer.
Or if a maintenance if happening.

[yannK]

the backlog effect does not produce the same messages, it does like "cannot sent dagta to the output queue, parsing queue full".

[yannK]

yes, please open a ticket from the storm portal (help page), and authorize the support team to check your project.

[SeanKilleen]

Related: I guess I'm also asking, Should I start another question based on the fact that I still can't seem to receive any log entries into SplunkStorm from splunk despite it not showing any errors in the logs besides the TcpOutputProc messages?

[SeanKilleen]

Thanks. So I shouldn't be worried that I see hundreds of those entries in splunkd.log? And one last follow-up: I also see no entries being uploaded to splunk, but could this be because it is processing the first giant backlog of windows event log events?