Getting Data In
Highlighted

Splunk Universal Forwarder stops forwarding after one successful forward

Motivator

I have a windows domain controller with a universal forwarder.
I have SplunkTAWindows deployed out to it using the universal forwarder(this is the only app deployed),
I have an outputs.conf file pointing to my indexer port

[tcpout:DomainControllers]
server=myserver.mycompany.com:6666

I have my indexer with a splunk 2 splunk looking watching on port 6666

I have this in my inputs.conf on the Universal forwarder:
[WinEventLog://Security]
disabled = 0
startfrom = oldest
current
only = 1
evtresolvead_obj = 0
checkpointInterval = 5
index = dclogs

The data is forwarded once to the indexer successfully, then does not send anything more, the logs simply say that is is phoning home.

I send a small update to the splunktawindows (such as adding a space) and it then sends the data to my indexer Once and only once.

Here are the only possible errors that i see in the logs:

TcpOutputFd - Read error. Either the application has not called WSAStartup, or WSAStartup failed.
05-06-2014 08:21:22.632 -0500 INFO  TcpOutputProc - Connection to 44.44.44.44:6666 closed. Read error. Either the application has not called WSAStartup, or WSAStartup failed.
ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-regmon.exe""  splunk-regmon - SysmonMigrator::read - 'sysmon.conf' was not found, no migration is required.

More Logs:
05-07-2014 16:28:49.717 -0500 DEBUG TcpOutputProc - channel registered
05-07-2014 16:28:49.717 -0500 DEBUG TcpOutputProc - Pushed eventId=86331 on chanID=33 to back of tcp client (tcp output) queue
05-07-2014 16:28:49.717 -0500 DEBUG TcpOutputProc - channel registered
05-07-2014 16:28:49.717 -0500 DEBUG TcpOutputProc - Pushed eventId=86332 on chanID=33 to back of tcp client (tcp output) queue
05-07-2014 16:28:49.717 -0500 DEBUG TcpOutputProc - channel registered
05-07-2014 16:28:49.717 -0500 DEBUG TcpOutputProc - Pushed eventId=86333 on chanID=33 to back of tcp client (tcp output) queue
05-07-2014 16:28:49.732 -0500 DEBUG TcpOutputProc - channel registered
05-07-2014 16:28:49.732 -0500 DEBUG TcpOutputProc - Pushed eventId=86334 on chanID=33 to back of tcp client (tcp output) queue
05-07-2014 16:28:49.732 -0500 DEBUG TcpOutputProc - channel registered
05-07-2014 16:28:49.732 -0500 DEBUG TcpOutputProc - Unregistering Channel for : source::C:\Program Files\SplunkUniversalForwarder\var\log\splunk\splunkd.log|host::mydc|splunkd|3744.44.44.444:6666, oneTimeClient=0, _events.size()=1, _refCount=2, _waitingAckQ.size()=0, _supportsACK=0
05-07-2014 16:28:49.732 -0500 DEBUG TcpOutputProc - Pushed eventId=86335 on chanID=33 to back of tcp client (tcp output) queue
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - numchannels = 0
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - ---- existing clients - start ----
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - Client 444.444.44.444:6666, oneTimeClient=0, _events.size()=0, _refCount=1, _waitingAckQ.size()=0, _supportsACK=0
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - ---- existing clients - end ----
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - AutoLB timer started to select new connection
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - Removing quarantine for idx=444.444.44.444:6666
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - Pinging idx=444.444.44.444:6666
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - Destryong AutoLBWrappedPollableDescriptor for 444.444.44.444:6666
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - BEGIN - randomizeConnectionsList
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - Indexer uri 444.444.44.444:6666, client refCount=1, client=non-NULL
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - BEGIN - After sorting
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - Indexer uri 444.444.44.444:6666, client refCount=1, client=non-NULL
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - Found currently active indexer 444.444.44.444:6666, client refCount=1, client=non-NULL
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - getting connected clients
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - Sending HB to 444.444.44.444:6666, oneTimeClient=0, _events.size()=0, _refCount=1, _waitingAckQ.size()=0, _supportsACK=0
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - Sending hb from TcpOutputClient for 444.444.44.444:6666, oneTimeClient=0, _events.size()=0, _refCount=1, _waitingAckQ.size()=0, _supportsACK=0
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - Connector::runRawStateMachine in state=eRawInit
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - tcpConnect to 444.444.44.444:6666
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - ConnectionSuccessful. _rawConnectionState=eRawTcpConnectInProgress
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - Connector::runRawStateMachine in state=eRawTcpConnectDone
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - Destryong AutoLBWrappedPollableDescriptor for 444.444.44.444:6666
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - channel not registered yet
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - Registering Channel for : source::C:\Program Files\SplunkUniversalForwarder\var\log\splunk\splunkd.log|host::mydc|splunkd|37444.444.44.444:6666, oneTimeClient=0, _events.size()=0, _refCount=1, _waitingAckQ.size()=0, _supportsACK=0
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - Pushed eventId=86336 on chanID=33 to back of tcp client (tcp output) queue
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - channel registered
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - Pushed eventId=86337 on chanID=33 to back of tcp client (tcp output) queue
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - channel registered
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - Pushed eventId=86338 on chanID=33 to back of tcp client (tcp output) queue
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - channel registered

0 Karma
Highlighted

Re: Splunk Universal Forwarder stops forwarding after one successful forward

Motivator
0 Karma
Highlighted

Re: Splunk Universal Forwarder stops forwarding after one successful forward

Motivator

Now it went for about 45 minutes, then stopped

metrics.log states: 05-06-2014 10:50:42.121 -0500 INFO StatusMgr - destPort=6666, eventType=connectdone, sourceHost=44.44.44.44, sourceIp=44.44.44.44, sourcePort=50138, statusee=TcpInputProcessor
05-06-2014 10:50:42.121 -0500 INFO StatusMgr - sourcePort=6666, ssl=false, statusee=TcpInputProcessor
05-06-2014 10:50:42.340 -0500 INFO StatusMgr - destPort=6666, eventType=connect
close, sourceHost=44.44.44.44, sourceIp=44.44.44.44, sourcePort=50138, statusee=TcpInputProcessor

so my domain controller 44.44.44.44 is still connecting to the indexer

0 Karma
Highlighted

Re: Splunk Universal Forwarder stops forwarding after one successful forward

Motivator

Raised my maxkbs to 0 (unlimited) and it ran for 4 minutes.. instead of the ~45-60 minutes, changed it to 56kbps.. stopped after 22 minutes.

0 Karma
Highlighted

Re: Splunk Universal Forwarder stops forwarding after one successful forward

Motivator

This is specifically my Windows Security Event Logs, the splunk logs get forwarded just fine.

0 Karma
Highlighted

Re: Splunk Universal Forwarder stops forwarding after one successful forward

Motivator

Just learned that our server is on Windows 2012 R2, Will be getting the splunk 6.1 forwarder on there sometime to verify that is our issue.

View solution in original post

0 Karma
Highlighted

Re: Splunk Universal Forwarder stops forwarding after one successful forward

Motivator

This was indeed the issue, updated to 6.1 and it is now successfully forwarding.

0 Karma