Why are Windows Event Logs not forwarded after installing a new Windows server?

[krusty]

Hi there,

I have the following issue detected in our environment and I'm not sure where the problem comes from.
We have several Windows Server monitored with a heavy forwarder. The Event logs are grabbed remotely by WMI.
So far everything works as expected.

Now we have done a new installation of one Windows Server.
The Server has the same name and IP address. Only the OS has changed from Windows Server 2008 R2 to Windows Server 2012 R2.
If I do a wbemtest with the user on the Splunk heavy forwarder, the Splunk service is running, and I can see the events from the fresh installed server. So there are no permission or firewall issues between the forwarder and the Windows Server.
But I can't see any events from this server on the indexer.

Does someone has an idea what is going wrong or how I can figure out the problem?

For your information. I removed the configuration of the Windows Server on the forwarder, restarted the forwarder and add the Windows Server again and restarted the forwarder. Nothing happens.
I removed the index of the Windows Server from the indexer, restart the indexer and added the index again. Nothing happens.

Could it be possible that the Splunk forwarder stores Information of grabbed events in another file?

For any ideas I'll be very thankful.

[edit: it's a heavy forwarder not a universal one]

[dvwijk · Accepted solution]

Hi. Splunk keeps track of which eventid it has indexed for each server. You should search for fish buckets en resetting these.

So it will start working when it reaches the number it last indexed on the old server 🙂

[gcusello]

Hi krusty,
to better understand you situation, you should make some test, to understand if your Indexers receive logs from your new Forwarder or not:

On Indexers:
- run the following search index=_internal host=your_hostname and index=_internal host=your_hostname*

On forwarder:
- verify indexers name (or IP address) in $SPLUNK_HOME\etc\system\local\outputs.conf
- verify hostname in $SPLUNK_HOME\etc\system\local\server.conf
- verify hostname in $SPLUNK_HOME\etc\system\local\inputs.conf
- by BLI telnet indexers_IP 9997
If eventually for Indexers you use username, try with IP address.

If all the checks are OK run on forwarder (from CLI) $SPLUNK_HOME\bin\splunk cmd btool props list --debug > ppp.txt and verify in this file the correct addressing to Indexers.

If you don't receive logs in _internal there is a problem in communication between Forwarder and Indexers.
If you have logs in _internal but not in wineventlog (or a different index you use), modify on forwarder inputs.conf in $SPLUNK_HOME\etc\apps\TA_Windows inserting in one stanza crcSalt = <SOUCE> (literally, including the angle brackets).
In this way you reload all events (see https://docs.splunk.com/Documentation/Splunk/6.5.2/Admin/Inputsconf).

Let me know.

Bye.
Giuseppe

[krusty]

Hi Giuseppe,

thanks for your answer.
I have to clarify something first. I monitor the remote eventlog with wmi, so I have on the forwarder all Windows Servers configured in the wmi.conf file.
Example:
[WMI:win2k12r2-vm]
disabled = 0
event_log_file = Application, Security, System, HardwareEvents, Internet Explorer, Key Management Service, Windows PowerShell
index = win2k12r2-vm
interval = 5
server = win2k12r2-vm

You assume that the forwarder is the new Installation, but this isn't true. The forwarder runs over more than 2 years without any problem. It is only the monitored Windows Server which was new installed and now I cannot get any events into the indexer.

For me it is importand to find out if any events will be send to the indexer or if the forwarder has the issue with receiving the data via wmi from the new server.

Kind regards,
Thomas

[woodcock]

Do you have a really, REALLY good reason for going with WMI? There is a reason that Splunk created the WinEventLog facility (actually probably hundreds). I would NEVER use WMI direct to get data into Splunk.

[krusty]

Hmm, a really good reason to do so? I think not really.
We start with our splunk environment many years ago. If I'm right at the beginning there were no other choice as monitor the Windows events by using WMI or by installing a light forwarder on each server.
We didn't like to install another agent on the server so we decided to go with a heavy forwarder and the WMI (remote event log grabbing) solution.

I do my very best to change the situation now, because I read a lot in the forum about the disadvantage of using WMI and the advantage of using UF instead.

Do you have on each Windows server a UF installed where you want to grab the Event logs?

[gcusello]

OK it's really different!
I never used WMI for remote monitoring, always Forwarders.
Sorry I cannot help you!
Bye.
Giuseppe