We are failing to get events indexed with the following topology: Splunk 4.2 receiving compressed events over the internet from a single primary Universal Forwarder, in turn receiving uncompressed events from local-area secondary Universal Forwarders. All installations are on Windows Server 2008R2 (64-bit).
In our test the secondary forwarders have a simple inputs.conf:
disabled = 0
Events from the primary forwarder show up in central Splunk, and in another test we sucessfully forwarded events from a Linux secondary forwarder through the primary forwarder to the central Splunk. The problem is thus windows-specific.
But, with a windows secondary forwarder, the application eventlog and splunk index::_internal events do not appear in the central splunk. This we ascertained from viewing the summary dashboard looking for the hosts and searching for the hosts. However the forwarders metrics.log reportis successful forwarding and there are no errors in splunkd.log:
I would like to get these twice-forwarded windows events to appear in the main splunk.
Interesting. It would be useful to have some more information about your configuration. First, what type of forwarders are each kind? I assume the "secondary" source forwarders are UF, but are the "primary" ones LWF, UF, or standard? It would also be helpful to see the outputs.conf from each tier, and the inputs.conf from the middle tier, and whether you are applying any TRANSFORMS to _TCP_ROUTING.
All the forwarders Universal Forwarder 4.2 on Windows Server 2008R2. Only the central destination is a full-fledged Splunk instance.
We are not applying any transforms to TCPROUTING
Monitored file inputs forward fine from source UF->intermediate UF->Splunk
Only Windows Event Log and WMI events fail to appear in Splunk when passed on by an intermediate UF.
Therefore, we have worked around the problem by uninstalling the second-tier forwarders and using WMI on the old "intermediate" UF to pull WMI events from other machines and forward direct to Central Splunk.
Did anyone ever come to a conclusion on the underlying problem here?
I'm seeing similar behavior under 4.2.2, with a light-weight forwarder as the middle-tier system and universal forwarders as the bottom tier (those two tiers are all Windows; Splunk indexing cluster is Linux).
The middle-tier's inputs.conf:
[udp://514] sourcetype = syslog connection_host = dns [tcp://514] sourcetype = syslog connection_host = dns [splunktcp-ssl://9997] disabled = 0 connection_host = dns _TCP_ROUTING = indexCluster [SSL] serverCert=$SPLUNK_HOME\etc\auth\server.pem password=password rootCA=$SPLUNK_HOME\etc\auth\ca.pem requireClientCert=false
[tcpout] maxQueueSize = 1000 forwardedindex.0.whitelist = .* forwardedindex.1.whitelist = _.* forwardedindex.filter.disable = false disabled=false defaultGroup=indexCluster [tcpout:indexCluster] server=10.128.81.31:9997,10.128.81.32:9997 autoLB = true [tcpout-server://10.128.81.31:9997] sslCertPath=$SPLUNK_HOME/etc/auth/server.pem sslPassword=password sslRootCAPath=$SPLUNK_HOME/etc/auth/ca.pem sslVerifyServerCert=false [et cetera]
[tcpout] maxQueueSize = 1000 forwardedindex.0.whitelist = .* forwardedindex.1.whitelist = _.* forwardedindex.filter.disable = false disabled=false defaultGroup=Collector [tcpout:Collector] server=middle-tier:9997 autoLB = true [tcpout-server://middle-tier:9997] sslCertPath=$SPLUNK_HOME/etc/auth/server.pem sslPassword=password sslRootCAPath=$SPLUNK_HOME/etc/auth/ca.pem sslVerifyServerCert=false
With this configuration, I do see regular file-monitor logs coming through from the second-tier systems (for example, I see metrics.log come to my central _internal index; note that I explicitly whitelisted the _ indexes in my outputs.conf above), but I never see any WinEventLog: events get passed through the forwarder (except those generated by the forwarder itself).
WMI isn't really a viable workaround here, as I expect it wouldn't be for many customers.
Any thoughts, Gerald?
Possibly OP was encountering SPL-39592 (see http://docs.splunk.com/Documentation/Splunk/latest/ReleaseNotes/WorkaroundtoaddWindowseventloginputs... )?
I don't have that problem, I do have WinEventLog:Application and :Security stanzas enabled on the systems running the Universal Forwarder (and, for systems with a Universal Forwarder sending data directly to my indexing cluster, I do receive WinEventLog events).
We have worked around the issue by having the primary Universal Forwarder remotely collect WMI logs (wmi.conf) from the other machines on the local network. This does away with second-tier forwarders entirely.
We do however encounter occasional missing events or missing "Message" fields with this setup.