Getting Data In

Forwarder to splunkstorm is timing out?

Engager

Hi all,

I did the following:

  • Set up a splunk forwarder
  • Obtained my SplunkStorm Credentials
  • Installed splunk credentials SPL credentials (though I'm not sure that I did this 100% correctly)
  • Edited inputs.conf to add windows system log events
  • Started the forwarder.

In the log, I see several lines of:

Line 272: 10-11-2013 11:53:30.478 -0400 WARN  TcpOutputProc - Raw connection to ip=107.20.29.58:9997 timed out
Line 276: 10-11-2013 11:54:30.479 -0400 WARN  TcpOutputProc - Cooked connection to ip=54.224.46.188:9997 timed out

I ran the command splunk cmd btool outputs list --debug and got the result (sslpassword and project id has been changed from its value to [redacted]:

C:\Program Files\SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\default\outputs.conf                        [tcpout]
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               autoLBFrequency = 30
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               blockOnCloning = true
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               blockWarnThreshold = 100
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               compressed = false
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               connectionTimeout = 20
C:\Program Files\SplunkUniversalForwarder\etc\apps\stormforwarder_[redacted]\default\outputs.conf defaultGroup = storm_indexers
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               disabled = false
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               dropClonedEventsOnQueueFull = 5
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               dropEventsOnQueueFull = -1
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               forceTimebasedAutoLB = false
C:\Program Files\SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\default\outputs.conf                        forwardedindex.0.whitelist = .*
C:\Program Files\SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\default\outputs.conf                        forwardedindex.1.blacklist = _.*
C:\Program Files\SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\default\outputs.conf                        forwardedindex.2.whitelist = _audit
C:\Program Files\SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\default\outputs.conf                        forwardedindex.filter.disable = false
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               heartbeatFrequency = 30
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               indexAndForward = false
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               maxConnectionsPerIndexer = 2
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               maxFailuresPerInterval = 2
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               maxQueueSize = auto
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               readTimeout = 300
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               secsInFailureInterval = 1
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               sendCookedData = true
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               useACK = false
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               writeTimeout = 300
C:\Program Files\SplunkUniversalForwarder\etc\apps\stormforwarder_[redacted]\local\outputs.conf   [tcpout:storm_indexers]
C:\Program Files\SplunkUniversalForwarder\etc\apps\stormforwarder_[redacted]\default\outputs.conf autoLB = true
C:\Program Files\SplunkUniversalForwarder\etc\apps\stormforwarder_[redacted]\default\outputs.conf server = forwarder.d9bw-e6eh.data.splunkstorm.com:9997
C:\Program Files\SplunkUniversalForwarder\etc\apps\stormforwarder_[redacted]\default\outputs.conf sslCertPath = $SPLUNK_HOME/etc/auth/server.pem
C:\Program Files\SplunkUniversalForwarder\etc\apps\stormforwarder_[redacted]\local\outputs.conf   sslPassword = [redacted]
C:\Program Files\SplunkUniversalForwarder\etc\apps\stormforwarder_[redacted]\default\outputs.conf sslRootCAPath = $SPLUNK_HOME/etc/apps/stormforwarder_[redacted]/ssl/star.splunkstorm.com.chain
C:\Program Files\SplunkUniversalForwarder\etc\apps\stormforwarder_[redacted]\default\outputs.conf useACK = true
C:\Program Files\SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\default\outputs.conf                        [tcpout]
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               autoLBFrequency = 30
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               blockOnCloning = true
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               blockWarnThreshold = 100
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               compressed = false
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               connectionTimeout = 20
C:\Program Files\SplunkUniversalForwarder\etc\apps\stormforwarder_[redacted]\default\outputs.conf defaultGroup = storm_indexers
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               disabled = false
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               dropClonedEventsOnQueueFull = 5
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               dropEventsOnQueueFull = -1
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               forceTimebasedAutoLB = false
C:\Program Files\SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\default\outputs.conf                        forwardedindex.0.whitelist = .*
C:\Program Files\SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\default\outputs.conf                        forwardedindex.1.blacklist = _.*
C:\Program Files\SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\default\outputs.conf                        forwardedindex.2.whitelist = _audit
C:\Program Files\SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\default\outputs.conf                        forwardedindex.filter.disable = false
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               heartbeatFrequency = 30
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               indexAndForward = false
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               maxConnectionsPerIndexer = 2
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               maxFailuresPerInterval = 2
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               maxQueueSize = auto
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               readTimeout = 300
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               secsInFailureInterval = 1
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               sendCookedData = true
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               useACK = false
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               writeTimeout = 300
C:\Program Files\SplunkUniversalForwarder\etc\apps\stormforwarder_[redacted]\local\outputs.conf   [tcpout:storm_indexers]
C:\Program Files\SplunkUniversalForwarder\etc\apps\stormforwarder_[redacted]\default\outputs.conf autoLB = true
C:\Program Files\SplunkUniversalForwarder\etc\apps\stormforwarder_[redacted]\default\outputs.conf server = forwarder.d9bw-e6eh.data.splunkstorm.com:9997
C:\Program Files\SplunkUniversalForwarder\etc\apps\stormforwarder_[redacted]\default\outputs.conf sslCertPath = $SPLUNK_HOME/etc/auth/server.pem
C:\Program Files\SplunkUniversalForwarder\etc\apps\stormforwarder_[redacted]\local\outputs.conf   sslPassword = [redacted]
C:\Program Files\SplunkUniversalForwarder\etc\apps\stormforwarder_[redacted]\default\outputs.conf sslRootCAPath = $SPLUNK_HOME/etc/apps/stormforwarder_[redacted]/ssl/star.splunkstorm.com.chain
C:\Program Files\SplunkUniversalForwarder\etc\apps\stormforwarder_[redacted]\default\outputs.conf useACK = true
0 Karma
1 Solution

Splunk Employee
Splunk Employee

The timeout is simply caused when the forwarder rotating across the ip of the dns load balancer.
Or if a maintenance if happening.

View solution in original post

New Member

I'm seeing exactly the same behavior, my Windows event logs and Perfmon data isn't showing up in Splunk Storm but a normal text-based logfile does. I've already opened a ticket a couple days ago, but got no response yet.

0 Karma

New Member

@pbradfordkc: Windows Event Logs don't work yet when using Universal Forwarder 6 with Storm, see http://answers.splunk.com/answers/123027/splunk-storm-universal-forwarder

0 Karma

Explorer

I am running into the same issue. I am seeing text based logs but no windows event logs?

Does splunk storm support windows event logs. I thought it was because i was running server 2012 r2 but 2012 r2 is now supported with 6.1. Is there any trouble shooting steps out there?

0 Karma

Splunk Employee
Splunk Employee

The timeout is simply caused when the forwarder rotating across the ip of the dns load balancer.
Or if a maintenance if happening.

View solution in original post

Splunk Employee
Splunk Employee

the backlog effect does not produce the same messages, it does like "cannot sent dagta to the output queue, parsing queue full".

0 Karma

Splunk Employee
Splunk Employee

yes, please open a ticket from the storm portal (help page), and authorize the support team to check your project.

0 Karma

Engager

Related: I guess I'm also asking, Should I start another question based on the fact that I still can't seem to receive any log entries into SplunkStorm from splunk despite it not showing any errors in the logs besides the TcpOutputProc messages?

0 Karma

Engager

Thanks. So I shouldn't be worried that I see hundreds of those entries in splunkd.log? And one last follow-up: I also see no entries being uploaded to splunk, but could this be because it is processing the first giant backlog of windows event log events?

0 Karma