Getting Data In

Splunk Universal Forwarder stops forwarding after one successful forward

aelliott
Motivator

I have a windows domain controller with a universal forwarder.
I have Splunk_TA_Windows deployed out to it using the universal forwarder(this is the only app deployed),
I have an outputs.conf file pointing to my indexer port

[tcpout:DomainControllers]
server=myserver.mycompany.com:6666

I have my indexer with a splunk 2 splunk looking watching on port 6666

I have this in my inputs.conf on the Universal forwarder:
[WinEventLog://Security]
disabled = 0
start_from = oldest
current_only = 1
evt_resolve_ad_obj = 0
checkpointInterval = 5
index = dclogs

The data is forwarded once to the indexer successfully, then does not send anything more, the logs simply say that is is phoning home.

I send a small update to the splunk_ta_windows (such as adding a space) and it then sends the data to my indexer Once and only once.

Here are the only possible errors that i see in the logs:

TcpOutputFd - Read error. Either the application has not called WSAStartup, or WSAStartup failed.
05-06-2014 08:21:22.632 -0500 INFO  TcpOutputProc - Connection to 44.44.44.44:6666 closed. Read error. Either the application has not called WSAStartup, or WSAStartup failed.
ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-regmon.exe""  splunk-regmon - SysmonMigrator::read - 'sysmon.conf' was not found, no migration is required.

More Logs:
05-07-2014 16:28:49.717 -0500 DEBUG TcpOutputProc - channel registered
05-07-2014 16:28:49.717 -0500 DEBUG TcpOutputProc - Pushed eventId=86331 on chanID=33 to back of tcp client (tcp output) queue
05-07-2014 16:28:49.717 -0500 DEBUG TcpOutputProc - channel registered
05-07-2014 16:28:49.717 -0500 DEBUG TcpOutputProc - Pushed eventId=86332 on chanID=33 to back of tcp client (tcp output) queue
05-07-2014 16:28:49.717 -0500 DEBUG TcpOutputProc - channel registered
05-07-2014 16:28:49.717 -0500 DEBUG TcpOutputProc - Pushed eventId=86333 on chanID=33 to back of tcp client (tcp output) queue
05-07-2014 16:28:49.732 -0500 DEBUG TcpOutputProc - channel registered
05-07-2014 16:28:49.732 -0500 DEBUG TcpOutputProc - Pushed eventId=86334 on chanID=33 to back of tcp client (tcp output) queue
05-07-2014 16:28:49.732 -0500 DEBUG TcpOutputProc - channel registered
05-07-2014 16:28:49.732 -0500 DEBUG TcpOutputProc - Unregistering Channel for : source::C:\Program Files\SplunkUniversalForwarder\var\log\splunk\splunkd.log|host::mydc|splunkd|3744.44.44.444:6666, oneTimeClient=0, _events.size()=1, _refCount=2, _waitingAckQ.size()=0, _supportsACK=0
05-07-2014 16:28:49.732 -0500 DEBUG TcpOutputProc - Pushed eventId=86335 on chanID=33 to back of tcp client (tcp output) queue
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - numchannels = 0
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - ---- existing clients - start ----
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - Client 444.444.44.444:6666, oneTimeClient=0, _events.size()=0, _refCount=1, _waitingAckQ.size()=0, _supportsACK=0
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - ---- existing clients - end ----
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - AutoLB timer started to select new connection
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - Removing quarantine for idx=444.444.44.444:6666
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - Pinging idx=444.444.44.444:6666
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - Destryong AutoLBWrappedPollableDescriptor for 444.444.44.444:6666
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - BEGIN - randomizeConnectionsList
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - Indexer uri 444.444.44.444:6666, client refCount=1, client=non-NULL
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - BEGIN - After sorting
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - Indexer uri 444.444.44.444:6666, client refCount=1, client=non-NULL
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - Found currently active indexer 444.444.44.444:6666, client refCount=1, client=non-NULL
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - getting connected clients
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - Sending HB to 444.444.44.444:6666, oneTimeClient=0, _events.size()=0, _refCount=1, _waitingAckQ.size()=0, _supportsACK=0
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - Sending hb from TcpOutputClient for 444.444.44.444:6666, oneTimeClient=0, _events.size()=0, _refCount=1, _waitingAckQ.size()=0, _supportsACK=0
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - Connector::runRawStateMachine in state=eRawInit
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - tcpConnect to 444.444.44.444:6666
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - ConnectionSuccessful. _rawConnectionState=eRawTcpConnectInProgress
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - Connector::runRawStateMachine in state=eRawTcpConnectDone
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - Destryong AutoLBWrappedPollableDescriptor for 444.444.44.444:6666
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - channel not registered yet
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - Registering Channel for : source::C:\Program Files\SplunkUniversalForwarder\var\log\splunk\splunkd.log|host::mydc|splunkd|37444.444.44.444:6666, oneTimeClient=0, _events.size()=0, _refCount=1, _waitingAckQ.size()=0, _supportsACK=0
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - Pushed eventId=86336 on chanID=33 to back of tcp client (tcp output) queue
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - channel registered
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - Pushed eventId=86337 on chanID=33 to back of tcp client (tcp output) queue
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - channel registered
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - Pushed eventId=86338 on chanID=33 to back of tcp client (tcp output) queue
05-07-2014 16:28:52.748 -0500 DEBUG TcpOutputProc - channel registered

0 Karma
1 Solution

aelliott
Motivator

Just learned that our server is on Windows 2012 R2, Will be getting the splunk 6.1 forwarder on there sometime to verify that is our issue.

View solution in original post

0 Karma

aelliott
Motivator

Just learned that our server is on Windows 2012 R2, Will be getting the splunk 6.1 forwarder on there sometime to verify that is our issue.

0 Karma

aelliott
Motivator

This was indeed the issue, updated to 6.1 and it is now successfully forwarding.

0 Karma

aelliott
Motivator

This is specifically my Windows Security Event Logs, the splunk logs get forwarded just fine.

0 Karma

aelliott
Motivator

Raised my maxkbs to 0 (unlimited) and it ran for 4 minutes.. instead of the ~45-60 minutes, changed it to 56kbps.. stopped after 22 minutes.

0 Karma

aelliott
Motivator

Now it went for about 45 minutes, then stopped

metrics.log states: 05-06-2014 10:50:42.121 -0500 INFO StatusMgr - destPort=6666, eventType=connect_done, sourceHost=44.44.44.44, sourceIp=44.44.44.44, sourcePort=50138, statusee=TcpInputProcessor
05-06-2014 10:50:42.121 -0500 INFO StatusMgr - sourcePort=6666, ssl=false, statusee=TcpInputProcessor
05-06-2014 10:50:42.340 -0500 INFO StatusMgr - destPort=6666, eventType=connect_close, sourceHost=44.44.44.44, sourceIp=44.44.44.44, sourcePort=50138, statusee=TcpInputProcessor

so my domain controller 44.44.44.44 is still connecting to the indexer

0 Karma

aelliott
Motivator
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...