I initially tested the Splunk Server on a Windows 7 machine and installed the Universal Forwarder on another WIndows 7 machine.
This worked with no issues other than having to run sfc /scannow to get the Forwarder installed.
Now, I want to set up a permanent server on a Windows Server 2008R2 machine and am having issues. I am setting up the forwarder on the same Windows 7 machine I set up forwarding on before.
As soon as I installed the forwarder, I see data appear under 'Search' on the server. However, the data never updates and stays stale at the time of install. I noticed an error on the PC about a registry leak and thought this was the cause. I fixed the corrupted user profile on the PC but still the issue occurred.
I then looked at the splunkd.conf logs and noticed several TcpOut error informing the connection was refused. So, I added
[splunktcp://9997] connection_host = none
to the inputs.conf.
The "connection refused" errors are resolved but I still don't see updated data on the server.
I then looked more here and saw that adding this into inputs.conf could help:
[script://$SPLUNK_HOME\bin\scripts\splunk-wmi.path] disabled = 0 [WinEventLog://Application] disabled = 0 [WinEventLog://Security] disabled = 0 [WinEventLog://System] disabled = 0 [WinEventLog://ForwardedEvents] checkpointInterval = 5 current_only = 0 disabled = 0 start_from = oldest [WinEventLog://Setup] checkpointInterval = 5 current_only = 0 disabled = 0 start_from = oldest
and also tried adding "wineventlog" to the Indexes on the server.
Still, no good.
I can telnet to the server so the connection seems to be working.
TcpOutputProc is reporting this:
11-21-2016 08:42:58.109 -0500 INFO TcpOutputProc - Initializing connection for non-ssl forwarding to splunkserver:9997 11-21-2016 08:42:58.125 -0500 INFO TcpOutputProc - tcpout group default-autolb-group using Auto load balanced forwarding 11-21-2016 08:42:58.125 -0500 INFO TcpOutputProc - Group default-autolb-group initialized with maxQueueSize=512000 in bytes. 11-21-2016 08:42:58.921 -0500 INFO TcpOutputProc - Connected to idx=10.5.1.181:9997
I see these warnings but not sure if they are relevant:
11-21-2016 08:41:54.532 -0500 WARN ProcessRunner - Process with pid 2264 did not exit within a given grace period after being signaled to exit. Will have to forcibly terminate. 11-21-2016 08:42:55.348 -0500 WARN AuditTrailManager - Private key file does not exist but is defined in audit.conf - no local event signing will take place. You can create auditTrail keys if necessary by running splunk createssl audit-keys 11-21-2016 08:42:55.847 -0500 INFO LMTracker - Setting feature=ResetWarnings state=DISABLED_DUE_TO_LICENSE (featureStatus=2) 11-21-2016 08:42:58.063 -0500 WARN PubSubMgr - Can't parse deployment server address from conf: "" 11-21-2016 08:42:58.281 -0500 WARN UserManagerPro - Can't find [distributedSearch] stanza in distsearch.conf, using default authtoken HTTP timeouts 11-21-2016 08:42:58.655 -0500 WARN X509Verify - X509 certificate (O=SplunkUser,CN=SplunkServerDefaultCert) should not be used, as it is issued by Splunk's own default Certificate Authority (CA). This puts your Splunk instance at very high-risk of the MITM attack. Either commercial-CA-signed or self-CA-signed certificates must be used; see: <http://docs.splunk.com/Documentation/Splunk/latest/Security/Howtoself-signcertificates>
The one that stands out to me the most is 11-21-2016 08:42:58.063 -0500 WARN PubSubMgr - Can't parse deployment server address from conf: ""
but would addressing this fix the problem?
Is there something else that I have missed?
Nevermind, I see what I did wrong.
I had set up the server to search an index I created called "test" for these logs.
The forwarder was configured to use "wineventlog" (per the above).
So the data is being forwarded now, I just can't see it displayed on the main Search UI by default.
If I run a manual search, I can see the data in index "wineventlog".
Now, I'm just wondering how to get this to appear in the UI. I'll open another question if I don't see how to do it.