Splunk Universal Forwarder stops forwarding after one successful forward

[aelliott]

I have a windows domain controller with a universal forwarder.
I have Splunk_TA_Windows deployed out to it using the universal forwarder(this is the only app deployed),
I have an outputs.conf file pointing to my indexer port

[tcpout:DomainControllers]
server=myserver.mycompany.com:6666

I have my indexer with a splunk 2 splunk looking watching on port 6666

I have this in my inputs.conf on the Universal forwarder:
[WinEventLog://Security]
disabled = 0
start_from = oldest
current_only = 1
evt_resolve_ad_obj = 0
checkpointInterval = 5
index = dclogs

The data is forwarded once to the indexer successfully, then does not send anything more, the logs simply say that is is phoning home.

I send a small update to the splunk_ta_windows (such as adding a space) and it then sends the data to my indexer Once and only once.

Here are the only possible errors that i see in the logs:

TcpOutputFd - Read error. Either the application has not called WSAStartup, or WSAStartup failed.
05-06-2014 08:21:22.632 -0500 INFO  TcpOutputProc - Connection to 44.44.44.44:6666 closed. Read error. Either the application has not called WSAStartup, or WSAStartup failed.
ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-regmon.exe""  splunk-regmon - SysmonMigrator::read - 'sysmon.conf' was not found, no migration is required.

More Logs:
05-07-2014 16:28:49.717 -0500 DEBUG TcpOutputProc - channel registered
05-07-2014 16:28:49.717 -0500 DEBUG TcpOutputProc - Pushed eventId=86331 on chanID=33 to back of tcp client (tcp output) queue
05-07-2014 16:28:49.717 -0500 DEBUG TcpOutputProc - channel registered
05-07-2014 16:28:49.717 -0500 DEBUG TcpOutputProc - Pushed eventId=86332 on chanID=33 to back of tcp client (tcp output) queue
05-07-2014 16:28:49.717 -0500 DEBUG TcpOutputProc - channel registered
05-07-2014 16:28:49.717 -0500 DEBUG TcpOutputProc - Pushed eventId=86333 on chanID=33 to back of tcp client (tcp output) queue
05-07-2014 16:28:49.732 -0500 DEBUG TcpOutputProc - channel registered
05-07-2014 16:28:49.732 -0500 DEBUG TcpOutputProc - Pushed eventId=86334 on chanID=33 to back of tcp client (tcp output) queue
05-07-2014 16:28:49.732 -0500 DEBUG TcpOutputProc - channel registered
05-07-2014 16:28:49.732 -0500 DEBUG TcpOutputProc - Unregistering Channel for : source::C:\Program Files\SplunkUniversalForwarder\var\log\splunk\splunkd.log|host::mydc|splunkd|3744.44.44.444:6666, oneTimeClient=0, _events.size()=1, _refCount=2, _waitingAckQ.size()=0, _supportsACK=0
05-07-2014 16:28:49.732 -0500 DEBUG TcpOutputProc - Pushed eventId=86335 on chanID=33 to back of tcp client (tcp output) queue
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - numchannels = 0
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - ---- existing clients - start ----
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - Client 444.444.44.444:6666, oneTimeClient=0, _events.size()=0, _refCount=1, _waitingAckQ.size()=0, _supportsACK=0
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - ---- existing clients - end ----
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - AutoLB timer started to select new connection
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - Removing quarantine for idx=444.444.44.444:6666
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - Pinging idx=444.444.44.444:6666
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - Destryong AutoLBWrappedPollableDescriptor for 444.444.44.444:6666
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - BEGIN - randomizeConnectionsList
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - Indexer uri 444.444.44.444:6666, client refCount=1, client=non-NULL
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - BEGIN - After sorting
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - Indexer uri 444.444.44.444:6666, client refCount=1, client=non-NULL
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - Found currently active indexer 444.444.44.444:6666, client refCount=1, client=non-NULL
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - getting connected clients
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - Sending HB to 444.444.44.444:6666, oneTimeClient=0, _events.size()=0, _refCount=1, _waitingAckQ.size()=0, _supportsACK=0
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - Sending hb from TcpOutputClient for 444.444.44.444:6666, oneTimeClient=0, _events.size()=0, _refCount=1, _waitingAckQ.size()=0, _supportsACK=0
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - Connector::runRawStateMachine in state=eRawInit
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - tcpConnect to 444.444.44.444:6666
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - ConnectionSuccessful. _rawConnectionState=eRawTcpConnectInProgress
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - Connector::runRawStateMachine in state=eRawTcpConnectDone
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - Destryong AutoLBWrappedPollableDescriptor for 444.444.44.444:6666
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - channel not registered yet
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - Registering Channel for : source::C:\Program Files\SplunkUniversalForwarder\var\log\splunk\splunkd.log|host::mydc|splunkd|37444.444.44.444:6666, oneTimeClient=0, _events.size()=0, _refCount=1, _waitingAckQ.size()=0, _supportsACK=0
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - Pushed eventId=86336 on chanID=33 to back of tcp client (tcp output) queue
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - channel registered
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - Pushed eventId=86337 on chanID=33 to back of tcp client (tcp output) queue
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - channel registered
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - Pushed eventId=86338 on chanID=33 to back of tcp client (tcp output) queue
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - channel registered

[aelliott · Accepted solution]

Just learned that our server is on Windows 2012 R2, Will be getting the splunk 6.1 forwarder on there sometime to verify that is our issue.

[aelliott]

This was indeed the issue, updated to 6.1 and it is now successfully forwarding.

[aelliott]

This is specifically my Windows Security Event Logs, the splunk logs get forwarded just fine.

[aelliott]

Raised my maxkbs to 0 (unlimited) and it ran for 4 minutes.. instead of the ~45-60 minutes, changed it to 56kbps.. stopped after 22 minutes.

[aelliott]

Now it went for about 45 minutes, then stopped

metrics.log states: 05-06-2014 10:50:42.121 -0500 INFO StatusMgr - destPort=6666, eventType=connect_done, sourceHost=44.44.44.44, sourceIp=44.44.44.44, sourcePort=50138, statusee=TcpInputProcessor
05-06-2014 10:50:42.121 -0500 INFO StatusMgr - sourcePort=6666, ssl=false, statusee=TcpInputProcessor
05-06-2014 10:50:42.340 -0500 INFO StatusMgr - destPort=6666, eventType=connect_close, sourceHost=44.44.44.44, sourceIp=44.44.44.44, sourcePort=50138, statusee=TcpInputProcessor

so my domain controller 44.44.44.44 is still connecting to the indexer

[aelliott]

i think i've figured this out
http://answers.splunk.com/answers/64554/starting-point-of-index