Windows Server 2008R2 Splunk server not receiving Windows Event Logs from a Windows 7 PC with Universal Forwarder installed

seeia — Mon, 21 Nov 2016 14:27:46 GMT

I initially tested the Splunk Server on a Windows 7 machine and installed the Universal Forwarder on another WIndows 7 machine.
This worked with no issues other than having to run sfc /scannow to get the Forwarder installed.

Now, I want to set up a permanent server on a Windows Server 2008R2 machine and am having issues. I am setting up the forwarder on the same Windows 7 machine I set up forwarding on before.

As soon as I installed the forwarder, I see data appear under 'Search' on the server. However, the data never updates and stays stale at the time of install. I noticed an error on the PC about a registry leak and thought this was the cause. I fixed the corrupted user profile on the PC but still the issue occurred.

I then looked at the splunkd.conf logs and noticed several TcpOut error informing the connection was refused. So, I added

[splunktcp://9997] 
connection_host = none

to the inputs.conf.

The "connection refused" errors are resolved but I still don't see updated data on the server.
I then looked more here and saw that adding this into inputs.conf could help:

[script://$SPLUNK_HOME\bin\scripts\splunk-wmi.path]
disabled = 0

[WinEventLog://Application]
disabled = 0

[WinEventLog://Security]
disabled = 0

[WinEventLog://System]
disabled = 0

[WinEventLog://ForwardedEvents]
checkpointInterval = 5
current_only = 0
disabled = 0
start_from = oldest

[WinEventLog://Setup]
checkpointInterval = 5
current_only = 0
disabled = 0
start_from = oldest

and also tried adding "wineventlog" to the Indexes on the server.
Still, no good.

I can telnet to the server so the connection seems to be working.
TcpOutputProc is reporting this:

11-21-2016 08:42:58.109 -0500 INFO  TcpOutputProc - Initializing connection for non-ssl forwarding to splunkserver:9997
11-21-2016 08:42:58.125 -0500 INFO  TcpOutputProc - tcpout group default-autolb-group using Auto load balanced forwarding
11-21-2016 08:42:58.125 -0500 INFO  TcpOutputProc - Group default-autolb-group initialized with maxQueueSize=512000 in bytes.
11-21-2016 08:42:58.921 -0500 INFO  TcpOutputProc - Connected to idx=10.5.1.181:9997

I see these warnings but not sure if they are relevant:

11-21-2016 08:41:54.532 -0500 WARN  ProcessRunner - Process with pid 2264 did not exit within a given grace period after being signaled to exit. Will have to forcibly terminate.
11-21-2016 08:42:55.348 -0500 WARN  AuditTrailManager - Private key file does not exist but is defined in audit.conf - no local event signing will take place. You can create auditTrail keys if necessary by running splunk createssl audit-keys
11-21-2016 08:42:55.847 -0500 INFO  LMTracker - Setting feature=ResetWarnings state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
11-21-2016 08:42:58.063 -0500 WARN  PubSubMgr - Can't parse deployment server address from conf: ""
11-21-2016 08:42:58.281 -0500 WARN  UserManagerPro - Can't find [distributedSearch] stanza in distsearch.conf, using default authtoken HTTP timeouts
11-21-2016 08:42:58.655 -0500 WARN  X509Verify - X509 certificate (O=SplunkUser,CN=SplunkServerDefaultCert) should not be used, as it is issued by Splunk's own default Certificate Authority (CA). This puts your Splunk instance at very high-risk of the MITM attack. Either commercial-CA-signed or self-CA-signed certificates must be used; see: <http://docs.splunk.com/Documentation/Splunk/latest/Security/Howtoself-signcertificates>

The one that stands out to me the most is 11-21-2016 08:42:58.063 -0500 WARN PubSubMgr - Can't parse deployment server address from conf: ""
but would addressing this fix the problem?

Is there something else that I have missed?

Re: Windows Server 2008R2 Splunk server not receiving Windows Event Logs from a Windows 7 PC with Universal Forwarder installed

seeia — Mon, 21 Nov 2016 18:38:41 GMT

Nevermind, I see what I did wrong.

I had set up the server to search an index I created called "test" for these logs.
The forwarder was configured to use "wineventlog" (per the above).

So the data is being forwarded now, I just can't see it displayed on the main Search UI by default.
If I run a manual search, I can see the data in index "wineventlog".

Now, I'm just wondering how to get this to appear in the UI. I'll open another question if I don't see how to do it.

topic Windows Server 2008R2 Splunk server not receiving Windows Event Logs from a Windows 7 PC with Universal Forwarder installed in Getting Data In

Windows Server 2008R2 Splunk server not receiving Windows Event Logs from a Windows 7 PC with Universal Forwarder installed

Re: Windows Server 2008R2 Splunk server not receiving Windows Event Logs from a Windows 7 PC with Universal Forwarder installed